I just returned from a one week vacation to a warm sunny beach on a small island not too far from Singapore. Even on my vacations my conversations often migrate to technology and my travel mate is an old friend and current employee at VMware, Dave Korsunsky. Sitting by a pool with a cocktail in hand at a fantastic hotel I asked my friend, “what is the right number of hosts per DRS/HA cluster?” Great conversation for a vacation, right?
I started thinking about this topic at Sydney’s vForum a month ago. VMware’s Dan Anderson suggested that designs that implemented maximum cluster sizes (32 hosts per cluster) were the result of misguided reasoning. Dan insisted that clusters need never be larger than eight hosts per cluster. And on this subject we bantered for a few minutes. Dan convinced me that there are few compelling reasons to implement large clusters. And we could think of many reasons to avoid them. I do not think it easy to assign one number as the “right” cluster size. But there are many principles that suggest small to medium sized clusters being choices.
First, the argument for the largest clusters: DRS efficiency. This was my primary claim in favor of 32-host clusters. My reasoning is simple: with more hosts in the cluster there are more CPU and memory resource holes into which DRS can place running virtual machines to optimize the cluster’s performance. The more hosts, the more options to the scheduler.
But on retrospect I think this is a weak argument. Its not backed by data and in practice I cannot imagine a 16 host cluster being much more efficient than an eight host cluster. Once vCenter is managing hundreds or more virtual machines per cluster, it has an astronomical number of combinations for VM placement. So, doubling the host (and the virtual machine count) should have little impact to cluster efficiency.
More importantly, with respect to the efficiency argument, maximum CPU and memory utilization will be bound either by the failover capacity or the target utilization, which is usually about 80%. With 20% reserved for resource spikes, the failover capacity is equal to the reserved resources at a 4+1 HA cluster. Any any cluster larger than this, the failover capacity is less than 20%. This means that only target utilization bounds resource efficiency.
The efficiency calculation is a little more tricky if you want to size your cluster for target resource utilization after a host failure. In this case each additional host provides some incremental value to the cluster’s utilization. To size a 4+1 cluster to 80% utilization after host failure, you will want to restrict CPU usage in the five hosts to 64%. Going to a 5+1 cluster results in a pre-failure CPU utilization target of 66%. The increases slowly approach 80% as the clusters get larger and larger. But, you can see that the incremental resource utilization improvement is never more than 2%. So, growing a cluster slightly provides very little value in terms of resource utilization.
Now why might you want to keep a cluster small? I can think of a few reasons.
It is generally wise to avoid mixing different classes of servers in a single pool. DRS does not make scheduling decisions based on the performance characteristics of the server so a new, powerful server in a cluster is just as likely to receive a mission-critical virtual machine as older, slower host. This would be unfortunate if a cluster contained servers with radically different–although EVC compatible–CPUs like the Intel Xeon 5400 and Xeon 5500 series. In the former case ESX would be using its software memory management unit which could perform as much as 40% worse than the hardware MMU in the Xeon 5500.
(I will momentarily digress to answer a question I often get in my performance talks: what is the impact of Enhanced vMotion Compatibility (EVC) on virtual machine performance? Briefly: very little to none. The instructions that are disabled on newer processors only benefit applications that were compiled to use those new instructions. Those applications are rare in the enterprise space.)
Given my recommendation that servers in a cluster should be of a similar class of performance, you will soon find that your purchasing patterns will influence your cluster size. If you are one of the few people lucky enough to work at a company that is buying servers by the truckload, you can size your clusters however you want. But the vast majority of VMware’s customers make smaller purchases of anywhere from four to 16 servers at a time. These will make nice, homogenous clusters of moderate size.
One more argument Dave offered for keeping clusters small is to use clusters for logical separation of applications of different class. By putting your mission-critical applications in a cluster of their own your “server huggers” will sleep better at night. They will be able to keep one eye on the iron that can make or break their job. In my opinion, using physical separation in a virtual world is resisting the complete cloud and hardware independent virtualization that we are all striving for. But I cannot begrudge an administrator that wants to hold onto some semblance of physical hardware best practices while traveling the multi-year journey to the private cloud.
Another of Dan’s arguments against large customers is the cumbersome nature of their change control. Clusters have to be managed to a consistent state and the complexity of this process is dependent on the number of items being managed. A very large cluster will present unique challenges when managing change.
So, have I given a recommendation? I am not sure. If anything I feel that Dave, Dan and I believe that a minimum cluster size needs should be set to guarantee that the CPU utilization target, and not the HA failover capacity, is the defining the number of wasted resources. This means a minimum cluster of something like four or five hosts. While neither of us claims a specific problem that will occur with very large clusters, we cannot imagine the value of a 32-host cluster. So, we think the right cluster size is somewhere shy of 10.
I am quite interested to hear your thoughts on this. Perhaps the best guidance will grow out of the crucible of debate.