Better load balancing when on-prem cluster gets overutilized and note in the UI which tells the problem

Our on-premises virtualization clusters are stretched across two data centers. We aim to keep resource utilization in the clusters below 50% so that in the event of a data center failure, there are still enough resources available for all VMs. To achieve this, we have created a policy in Turbonomic that sets utilization constraints for all hosts: 50% CPU utilization and 50% memory utilization. To better maintain the 50% utilization, we have set the desired state center to 95 and a diameter to 40 to allow more flexibility around the 95% mark and avoid constant VM move actions.
In one cluster, resource utilization steadily increased, causing all hosts to approach or exceed 50%. On one host, the resource usage of its VMs increased significantly, resulting in sustained CPU load close to 100% for days, yet Turbonomic showed the host as green without risks and did not generate move actions to relocate VMs to other hosts in the cluster.

It would have been sensible to move some VMs to other hosts, even if this exceeded the 50% utilization constraint. It's better to have 60 or 70% load on all hosts than one host consistently at 100% while many others remain at 50%. One solution could be to use soft and hard utilization constraints. The target could be a soft maximum load of 50%, which can be exceeded if needed to achieve a more balanced cluster load, while also indicating actions to provision more hosts. A hard constraint, such as 75%, must not be breached by relocating VMs to hosts already exceeding it.

Furthermore, a completely overloaded host should not be indicated as green without risks in Turbonomic; instead, it should be marked as red with a risk, highlighting that no load balancing is possible due to lack of available resources.
The problem at the moment is that Turbonomic behaves in such a way that a cluster slowly reaches the set limit of 50%, the load increases faster on one host, but load balancing no longer takes place because other hosts are close to 50%.