ESX must expend CPU cycles to maintain running virtual CPUs whether they are being used by an application or not. This means that host efficiency drops as more vCPUs are put on the server. But applications that scale well with CPUs will deliver greater performance when their virtual machines have been given more CPUs. The administrator must therefore balance the desires of an individual application’s owner with the needs of the entire cluster’s of applications.

There are several resources that VI administrators can use to inform their decisions in virtual machine sizing. I have listed some of them below.

Bruce Herndon’s Cost-of-SMP Article

Last summer the VMmark team’s Bruce Herndon published an article on the cost of SMP. I summarized his findings in a vPivot article I wrote on the ESX 4 scheduler. There are two key messages that you can take away from these posts to inform your decisions on virtual machine sizing:

• Over-sized virtual machines only hurt system performance when the server’s CPUs are saturated.  When utilization is low, unneeded vCPUs only penalize the system’s CPU utilization, not the applications’ performance.
• Unneeded 2-way virtual machines are not very harmful to the environment.  But administrators should be very careful with 4-way virtual machines and larger.

Co-stop and Ready Time

Ready time indicates a vCPU waiting for an available core when it has work to perform. Co-scheduling stop time (or co-stop time) indicates a vCPU being paused by the scheduler to allow its sibling vCPUs to catch up. These two counters can help administrators recognize a certain kind of stress due to limited CPU resources.

Ready time is generally a sign of the unavailability of CPU. Correction usually requires the administrator reducing work on the host (migrating virtual machines, decreasing vCPU count, etc.) or increasing CPU capacity (more hosts or faster CPUs). Co-stop time is a sign that the scheduler is allowing vCPUs to develop skew while it runs portions of virtual machines on available cores. Considerable numbers for these counters are 10% ready time and 3% co-stop time. There is no guarantee that application performance is suffering if these thresholds are crossed, but a problem may be present.

The important thing about ready time and co-stop time is that they are signs that you are using all of the CPU you have available to you. This could be a Good Thing. But it could also be a surprise to you. When these counters get high it is a good time to start asking yourself if you capacity usage meets your expectations. If not, you should inspect your virtual machines to be sure that the applications are using the vCPUs you have given them. If your guest tools show poor in-guest utilization then decrease those VM sizes. That will free up resources in the cluster for more virtual machines.

Application Scalability Information

I wish we lived in a world where every ISV published data showing their applications’ abilities to scale with cores. Unfortunately for us, many software vendors have for years allowed their customers to assume that each doubling of cores would double the performance of the application. VMware has chosen to provide some scalability information so our customers know how well or how poorly applications scale. But every customer of a software company deserves to have the vendor provide guidance on sizing the server. And those vendors deserve the right to put these results out on their own products. Go talk to your ISV to get the information you need to size your virtual machines.

CPU Usage Calculations and CapacityIQ

I am belatedly updating this post with a fourth way of identifying oversized virtual machines: mathematical calculation or Capacity IQ.

When a virtual machine consistently uses only a fraction of its vCPU resources it is possible that the virtual machine can be downsized and still deliver the same application performance. The calculation to determine this is simple: multiply the vCPU count by utilization and round up. Set the virtual machine’s vCPU count to the result of that calculation.

If you own CapacityIQ it will make this calculation for you for every virtual machine in your data center. Here is an screenshot of its recommendations based on virtual machine CPU and memory utilization. Click for a clearer picture.

CapacityIQ monitors CPU and memory utilization to recommend VM downsizing.

Among the many results presented by VRC, the configurations that most perplexed us were the two and four virtual machine configurations, each with four vCPUs per virtual machine.  The configuration with two virtual machines looked good and matched our internal numbers.  In this configuration there are a total of eight vCPUs on the host which maps each to its own physical core on the Xeon 5500 series processor.  The problem arose when the virtual machine count was increased to four, resulting in 16 total vCPUs.  In this configuration each vCPU is paired with one logical, Hyper-Threaded core.  Project VRC showed this configuration supporting no more desktops than the two-VM configuration, which suggests no value to Hyper-Threading on this configuration.

It took us some time to understand the reason for these results, but we eventually identified a very specific condition where ESX’s scheduler enforces fairness in scheduling vCPUs at at cost of throughput.  ESX’s scheduler has long be subject of the intensive scrutiny of a large number of VMware engineers to guarantee fair access to the processor for each virtual machine.  It is because of this fairness that VMware’s customers can rely on CPU resource controls.  But, when fairness goes too far, throughput may be sub-optimal.

Hyper-Threading presents particular problems to fairness because of the non-linear performance it delivers.  A thread will run at one speed when it has full access to a physical core, at another speed when it is sharing a core, and at third speed when sharing a core with a different thread.  As a result, ESX’s scheduler will sometimes pause a thread to enforce fairness.  These pauses are more common when Hyper-Threading is present to account for its lack of uniformity in thread performance.  If the host lacks vCPUs that are ready to run, the result is CPU utilization below saturation, leaving CPU cycles unused.

There are three specific conditions that can excite this condition:

1. A Xeon 5500 series processor is present with Hyper-Threading enabled,
2. CPU utilization is near saturation, and
3. A roughly one-to-one mapping between vCPUs and logical processors.

In this scenario, VMware vSphere favors fairness over throughput and sometimes pauses one vCPU to dedicate a whole core to another vCPU, eliminating gains provided by Hyper-Threading.  In cases outside of these three conditions, the performance of VMware vSphere 4 meets the high expectations of VMware’s R&D team and its customers.  Of course production environments rarely (never?) have a one-to-one ratio of vCPUs to logical processors.  This occurs when there are only four 4-way virtual machines on a Xeon 5500 system, for example.

But environments such as Project VRC’s are simplifications of production environments meant to understand the capabilities of virtual platforms.  VMware has provided a patch to Project VRC that will allow them to improve throughput in their environment.  We are going to release this patch and its documentation to the general public within a couple of weeks.  I do not expect that any of VMware’s customers will benefit from the changes is allows, but I will later document the patch and its usage for anyone that cares to experiment.

On VI3, many of VMware’s customers disabled Hyper-Threading on their older, Netburst architecture Intel processors. Intel has vaguely described the new Hyper-Threading as more efficient than the previous generation and I believe this to be due to a shorter pipeline and an improved ability to context switch pipeline stage data. Long pipelines–such as the Netburst era Xeons of model numbers x1xx and x2xx–are more likely to suffer bubbles during context switches and are therefore penalized versus shorter pipeline products, such as the Core i7. Furthermore, by pushing and restoring pipeline stage data during a hardware context switch, the new HT can reduce pipeline bubbles.

But the gains vSphere users experience as a result of the new Hyper-Threading also comes from changes in ESX. ESX’s scheduler must make decisions as to when to co-locate two worlds on a physical core to take advantage of Hyper-Threading. In some conditions the scheduler will perform this co-location and in others it will allow a world to run on the core by itself. The decision to execute worlds concurrently instead of serially on a physical core can be informally called the scheduler’s trust of Hyper-Threading. The vSphere scheduler trusts Hyper-Threading more than the VI3 scheduler did. This amplifies the effect of HT.

I am now going to bore you with a disclaimer before I give you any data showing the effect of Hyper-Threading. The value of HT will vary from workload to workload and the ultimate authority of HT’s value is the end-user. The following numbers are the result of informal analysis and VMware that should only be used as a guide in your own analysis. Please do not make purchasing decisions on this information, which is devoid of the detail we would normally commit to a white paper.

In addition to the gains we informally cite here, I can say that we have not yet seen a workload where the new Hyper-Threading slows down consolidated performance. As far as we can tell, the new Hyper-Threading should be left enabled in 100% of virtualized environments.

Usually this issue arises inside a new VMware customer or an established customer that has added new staff to the virtualization team. In both cases the administrators are familiar with existing tools and require good reason to retool their thinking and environment around a new measurement system.

I was discussing the response to these concerns with my friend and colleague Kaushik Banerjee, the head of VMware’s outbound engineering group.  Kaushik and I spend a lot of time thinking about communicating technical details to our customers and in this case we chose different approaches to answer the question.  Both responses are complementary, so choose the one that suits your needs.

Kaushik’s Approach: The Killer Examples

Kaushik suggested that if we show cases where the guest OS’s counters were obviously wrong that a naturally suspicious VI admin would never trust the guest counters again. To that end, I offer the following screen shots to make our point.

Perfmon's counters show utilization higher in the guest than the host reports.

This screen shot shows two counters available in Perfmon inside a Windows guest with the vmStatsProvider installed (available by default since vSphere). The darker, red line is the CPU utilization as reported by the guest. The lighter, greenish (?) line is CPU utilization of the virtual machine, from the host’s perspective. This is the real CPU utilization passed up to the host by vmStatsProvider.  Notice how the host is always reporting higher utilization than the guest.  This is due to one of the reasons why guest counters cannot be trusted: they are unaware of hypervisor overheads.

This second screen shot shows a different case where the host utilization is lower than that reported by the guest.  Again, the dark red line represents the guest OS’s report of CPU utilization and the lighter line shows the real CPU utilization as reported by ESX.

Perfmon's counters report a higher CPU utilization than ESX's.

The reason the host shows lower utilization than the guest is because the guest is unaware that it is only getting a fraction of the host’s CPU, time-sliced by ESX’s scheduler.  In this case the virtual machine was contending for CPU with other active virtual machines but this just the same principle would apply had a CPU limit been set.

Scott’s Approach: A Detailed Explanation

My approach to convincing VI admins to avoid guest tools is based on bottomless thirst for information that is common to technophiles. If I can provide an explanation for the underlying system of resource scheduling and manipulation, then our admins will be able to deal with the guest counter issue and maybe solve other issues with their newfound knowledge.

There are four reasons why guest counters cannot be trusted:

1. The guest is unaware of virtualization overheads.  As screen shot one showed above, the hypervisor will increase the CPU load as it virtualizes the hardware for the guest operating system.  That additional CPU work is not seeing by guest tools.
2. The guest is unaware that it is only seeing the portion of CPU that ESX’s scheduler is allowing it to see.  Because of contention or resource restrictions, virtual machines only get a slice of the CPU’s time.  When a guest thinks it is getting 100% of the CPU it may not know that the processor is being shared by eight other virtual machines. See the second screen shot above.
3. Time skew in virtual machines can change the sample window for time-based counters.  This means that the guest may have measured 10 milliseconds of time passage during a read command when 12 milliseconds have elapsed.  This is more common on older versions of ESX and when the host CPU is saturated.  More on this below.
4. The virtual machines are unaware that they are being de-scheduled when idle, which means that they appear to be working more of the time than they are.  Consider a case where a virtual machine is idle 90% of the time.  If ESX does not schedule the VM during its idle time then the guest will think that its processor queues are full 100% of the time that it is being executed.

The time drift explanation (item three) was historically the most problematic for VMware.  On older versions of our products time drift was common.  As ESX has matured we have reduced the amount of drift which has improved the accuracy of guest counters.  But the timer hardware is still being virtualized in software running on the host CPU.  This means that if host processor is fully utilized, the timer may not be scheduled on time, resulting in a delay in some ticks and a resultant skew in guest time.

References

Performance Best Practices and Benchmarking Guidelines.  This white paper was the last version that we printed that included benchmarking best practices, which contains some discussion on the need to measure performance from outside the host-under-test.

Timekeeping in Virtual Machines.  This document–not updated since VI3 but still accurate in its theory–will give the background on VMware-based time keeping and provide an explanation as to how skew can occur.

VMware vSphere™ 4: The CPU Scheduler in VMware® ESX™ 4.  This white paper provides great detail how the scheduler works which will fully explains the notion of time slicing.

I spend a great deal of time answering customers’ questions about the scheduler. Never have so many questions been asked about such an abstruse component for which so little user influence is possible. But CPU scheduling is central to system performance, so VMware strives to provide as much information on the subject as possible. In this blog entry, I want to point out a few nuggets of information on the CPU scheduler. These four bullets answer 95% of the questions I get asked.

Item 1: ESX 4’s Scheduler Better Uses Caches Across Sockets

On UMA systems at low load levels, virtual machine performance improves when each virtual CPU (vCPU) is placed on its own socket. This is because providing each vCPU its own socket also gives it the entire cache on that CPU. On page 18 of a recent paper on the scheduler written by Seongbeom Kim, a graph highlights the case where vCPU spreading improves performance.

The X-axis represents different combinations of VM and vCPU counts. SPECjbb is memory intensive and shows great gains with increases in CPU cache. The few cases that show dramatic benefit due to the ESX 4.0 scheduler are benefiting from the distribution of vCPUs across sockets. Very large gains are possible in this somewhat uncommon case.

Item 2: Overuse of SMP Only Slows Consolidated Environments At Saturation

For years customers have asked me how many vCPUs they should give to their VMs. The best guidance, “as few as possible”, seems too vague to satisfy. It remains the only correct answer, unfortunately. But a recent experiment performed by Bruce Herndon’s team sheds some light on this VM sizing question.

In this experiment we ran VMmark against VMs that were configured outside of VMmark specifications. In one case some of the virtual machines were given too few vCPUs and in another they were given too many. Because VMmark’s workload is fixed, increasing the VMs’ sizes does not increase the work performed by the VMs. In other words, the system’s score does not depend on the VMs’ vCPU count. Until CPU saturation, that is.

Notice that the scores are similar between the undersized, right-sized, and over-sized VMs. Up until tile 10 (60 VMs) they are nearly identical. There is a slight difference in processor utilization that begins to impact throughput (score) as the system runs out of CPU. At that point the additional vCPUs waste cycles which degrades system performance. Two points I will call out from this work:

• Sloppy VI admins that provide too many vCPUs need not worry about performance when their servers are under low load. But performance will suffer when CPU utilization spikes.
• The penalty of over-sizing VMs gets worse as VMs get larger. Using a 2-way VM is not that bad, but unneeded use of 4-way VMs when one or two processors suffice can cost up to 15% of your system throughput. I presume that unnecessarily eight vCPUs would be criminal.

Item 3: ESX Has Not Strictly Co-scheduled Since ESX 2.5

I have documented ESX’s relaxation of co-scheduling previously (Co-scheduling SMP VMs in VMware ESX Server). But this statement cannot be repeated too frequently: ESX has not strictly co-scheduled virtual machines since version 2.5. This means that ESX can place vCPUs from SMP VMs individually. It is not necessary to wait for physical cores to be available for every vCPU before starting the VM. However, as Item 3 pointed out, this does not give you free license to over-size your VMs. Be frugal with your SMP VMs and assign vCPUs only when you need them.

Item 4: The Cell Construct Has Been Eliminated in ESX 4.0

In the performance best practices deck that I give at conferences I talk about the benefits of creating small virtual machines over large ones. In versions of ESX up to ESX 3.5, the scheduler used a construct called a cell that would contain and lock CPU cores. The vCPUs from a single VM could never span a cell. With a ESX 3.x’s cell size of four this meant that VMs never spanned multiple four-core sockets. Consider this figure:

What this figure shows is that a 4-way VM on ESX 3.5 can only be placed in two locations on this hypothetical two-socket configuration. There are 12 combinations for a 2-way VM and eight for a uniprocessor VM. The scheduler has more opportunities to optimize VM placement when you provide it with smaller VMs.

In ESX 4 we have eliminated the cell lock so VMs can span multiple sockets, as item one states. Continue to think of this placement problem as a challenge to the scheduler that you can alleviate. By choosing multiple, smaller VMs you free the scheduler to pursue opportunities to optimize performance in consolidated environments