Undocumented HA Advanced Option - das.restartVmsWithoutResourceChecks

Some time ago, a colleague of mine (@stan_jurena) was challenged by one VMware customer who experienced APD (All Path Down) storage situation in the whole HA Cluster and he expected that VMs will be killed by VMware Hypervisor (ESXi) because of HA Cluster APD response setting "Power off and restart VMs - Aggressive restart policy". To be honest, I had the same expectation. However, after the discussion with VMware engineering, we have been told, that the primary role of HA Cluster is to keep VMs up and running, so "Aggressive restart policy" will restart VM only in certain conditions which are much better described in vSphere Client 7 UI. See the screenshot below.

APD Aggressive restart policy
A VM will be powered off, If HA determines the VM can be restarted on a different host, or if HA cannot detect the resources on other hosts because of network connectivity loss (network partition).

So, what it means? Aggressive restart policy is the same as Conservative but extended for the situation when there is network partitioning. This can be helpful in situations when you have IP storage and experience IP network issues but it does not help in a situation when you have dedicated Fibre Channel SAN and the storage is not available for the whole vSphere Cluster.

We explained to VMware engineering, that there are situations when it is much better to kill all VMs than keep compute (VMs) running without available storage. Based on these discussions, there was created a Feature Request, which was internally named as "super aggressive option" APD. I'm happy to see, that it was implemented and released in vSphere 7 as vSphere advanced option

das.restartVmsWithoutResourceChecks = false (default) / true (super aggressive)

I think this advanced option will be very useful for infrastructure architects / technical designers who will have a good justification to use this advanced option. Here are my typical justifications

• When the storage subsystem is unavailable for some time, Linux operating system switch file system to Read-Only mode which has a negative impact on running applications. Such a situation typically leads to server restart anyway.
• When you have an OS/Application clustering solution (for example MSCS) on top of vSphere clustering, having one Application node on one vSphere cluster and another Application node on different vSphere cluster, you prefer to kill VM (App Node) on the problematic cluster (without available storage) to fail-over to App Node (VM) running on the healthy cluster.

Hope this makes sense.

Please leave the comment if you will find this advanced option useful. VMware Engineering might consider adding this option into GUI, based vSphere architects / technical designers' feedback.

References

• Duncan Epping wrote the blog post about it here.
• For other "Advanced configuration options for VMware High Availability in vSphere 5.x and 6.x" check VMware KB 2033250.

CPU capacity planning and sizing

During infrastructure capacity planning and sizing, the technical designer has to calculate CPU, RAM, Storage, and Network resource requirements. Recently, I had an interesting discussion with my colleagues on how to estimate CPU requirements for application workload.

Each computer application requires some CPU resources for computational tasks and additional resources for I/O tasks. It is obvious that the computational tasks require CPU cycles, however, it is not so obvious that there are CPU cycles associated also with I/O. In other words, each I/O requires some CPU resources. It does not matter if it is memory, storage, or network I/O.

For example, a generally accepted rule of thumb in the networking is that

1 Hertz of CPU processing is required to send or receive 1 bit/s of TCP/IP.

[Source: VMware vSphere 6.5 Host Resources Deep Dive]

This would mean 2.5Gb/s would require ~ 2.5Ghz CPU, thus ~ 100% of one CPU Core @ 2.5 GHz.

It would be nice to have a similar rule of thumb for storage I/O. I did quick research (googling) but was not able to find any information about CPU requirements for storage I/O.  I did a quick test in my home lab and start the synthetic random workload (4KB I/O) on 4vCPU VM on ESXi host having CPU at 2 GHz, where I was able to see 5,000 IOPS with CPU utilization 8.5%. This would mean one 4KB I/O requires 136 Hz.

4KB I/O on 4x vCPU VM with pCPU @ 2 GHz 

I did another test with 512 B I/O where 1 IOPS requires 114 Hz.
And for 64KB I/O size, 1 IOPS requires 161 Hz.

0.5 KB I/O => 512 Bytes I/O (4,096 bits) = 114 Hz
4 KB I/O => 4,096 Bytes I/O (32,768 bits) = 136 Hz
64 KB I/O => 65,536 Bytes I/O (52,4288 bits) = 161 Hz

Based on my observations, it is difficult to define the rule of thumb for 1 Bit/s or Byte/s but rather I would define CPU (Hz) requirements for 1 storage I/O.

Based on multiple assessments of real datacenter environments, I would say that typical average storage I/O size is around 40-50 KB, therefore here is my rule of thumb

1 Storage I/O requires ~ 150 Hz of CPU processing

This would mean 10,000 IOPS would require ~ 1.5 GHz, thus 60% of one CPU Core @ 2.5 GHz.

Please, be aware that this is a very simplified calculation but clearly shows that storage workload is always associated with CPU requirements and it can help with capacity planning and infrastructure sizing.

What do you think about this calculation?
Do you observe different numbers?
Would you calculate it differently?
You can leave the comment below the article.

vSphere 7 - Storage Requirements for the vCenter Server Appliance

I have upgraded vSphere in my home lab and realized that VCSA 7.0 storage requirements increased significantly.

Here are the requirements of vCenter Server Appliance 6.7

Here are the requirements of vCenter Server Appliance 7.0

You can see the difference by yourself. VCSA 7.0 requires roughly 30%-60% more storage than VCSA 6.7. It is good to know it especially for home labs where hardware resources are limited or during logical designs of new environments where you do some math calculation to plan hardware requirements.