It was during discussions with advanced VMware customers in Melbourne that a solution to this problem occurred to me. I have reasoned it through and I think it holds water. I have socialized it with more customers and my colleagues and we think it stands. So I want to introduce a system for implementing virtual machines with a better assurance of a performance SLA.

The key to this process is that minimum performance can be measured using limits and that performance can be assured using reservations. You can develop and document virtual machines with performance SLAs using the following procedure:

• First, as always, define a small number of strictly-sized virtual machines to be used by all applications in your environment. Often these look something like small VMs of 1 vCPU and 4 GB RAM, medium VMs of 2 vCPUs and 8 GB of RAM, and large VMs of 4 vCPUs and 16 GB of RAM. Tune these numbers for your environment, as needed.
• For any application, benchmark its maximum performance against each of these virtual machine configurations on an unloaded system. Chose an ISV-supplied benchmark or a well-known third party tool. This sets your high water mark for throughput for each application in its virtual machine.
• For each configuration, set a CPU limit at 50% of the available CPU and a memory limit of 50% of the available memory. Retest the application against this smaller, limited configuration.
• During the applications’ deployment, change the limits to reservations. That is, remove limits and set reservations equal to the limits’ previous values, in this case 50%.
• Your application now has a maximum performance defined in bullet two, and a “guaranteed” performance measured in bullet three.  This is your application’s performance SLA.

The concept is simple: limits can be used to measure the performance of an application in the presence of that degree of contention. Reservations ensure that those resource amounts are always present. Here are some notes on this process:

• This is not a true guarantee since network and storage throughput may drop. No tool can eliminate this risk entirely but SIOC and NetIOC can reduce the risk of a network- or storage-induced performance failure.
• The memory test is going to be highly dependent on the working set created by your load generation tool. Your mileage will vary depending on your application owners’ use of the virtual machine.
• vCenter will guarantee that the reservations are always available through a process called admission control, which checks the cluster to ensure that enough CPU or memory is available to run the virtual machine immediately and in the event of a server failure.

As I said above, this is not a true guarantee of application performance. But it is as close as we can get until AppSpeed or a replacement evolves into universal application latency measurement that is fed into vCenter. And this is another in a growing list of reasons why CPU and memory reservations should be part of all VMware deployments.

ESX must expend CPU cycles to maintain running virtual CPUs whether they are being used by an application or not. This means that host efficiency drops as more vCPUs are put on the server. But applications that scale well with CPUs will deliver greater performance when their virtual machines have been given more CPUs. The administrator must therefore balance the desires of an individual application’s owner with the needs of the entire cluster’s of applications.

There are several resources that VI administrators can use to inform their decisions in virtual machine sizing. I have listed some of them below.

Bruce Herndon’s Cost-of-SMP Article

Last summer the VMmark team’s Bruce Herndon published an article on the cost of SMP. I summarized his findings in a vPivot article I wrote on the ESX 4 scheduler. There are two key messages that you can take away from these posts to inform your decisions on virtual machine sizing:

• Over-sized virtual machines only hurt system performance when the server’s CPUs are saturated.  When utilization is low, unneeded vCPUs only penalize the system’s CPU utilization, not the applications’ performance.
• Unneeded 2-way virtual machines are not very harmful to the environment.  But administrators should be very careful with 4-way virtual machines and larger.

Co-stop and Ready Time

Ready time indicates a vCPU waiting for an available core when it has work to perform. Co-scheduling stop time (or co-stop time) indicates a vCPU being paused by the scheduler to allow its sibling vCPUs to catch up. These two counters can help administrators recognize a certain kind of stress due to limited CPU resources.

Ready time is generally a sign of the unavailability of CPU. Correction usually requires the administrator reducing work on the host (migrating virtual machines, decreasing vCPU count, etc.) or increasing CPU capacity (more hosts or faster CPUs). Co-stop time is a sign that the scheduler is allowing vCPUs to develop skew while it runs portions of virtual machines on available cores. Considerable numbers for these counters are 10% ready time and 3% co-stop time. There is no guarantee that application performance is suffering if these thresholds are crossed, but a problem may be present.

The important thing about ready time and co-stop time is that they are signs that you are using all of the CPU you have available to you. This could be a Good Thing. But it could also be a surprise to you. When these counters get high it is a good time to start asking yourself if you capacity usage meets your expectations. If not, you should inspect your virtual machines to be sure that the applications are using the vCPUs you have given them. If your guest tools show poor in-guest utilization then decrease those VM sizes. That will free up resources in the cluster for more virtual machines.

Application Scalability Information

I wish we lived in a world where every ISV published data showing their applications’ abilities to scale with cores. Unfortunately for us, many software vendors have for years allowed their customers to assume that each doubling of cores would double the performance of the application. VMware has chosen to provide some scalability information so our customers know how well or how poorly applications scale. But every customer of a software company deserves to have the vendor provide guidance on sizing the server. And those vendors deserve the right to put these results out on their own products. Go talk to your ISV to get the information you need to size your virtual machines.

CPU Usage Calculations and CapacityIQ

I am belatedly updating this post with a fourth way of identifying oversized virtual machines: mathematical calculation or Capacity IQ.

When a virtual machine consistently uses only a fraction of its vCPU resources it is possible that the virtual machine can be downsized and still deliver the same application performance. The calculation to determine this is simple: multiply the vCPU count by utilization and round up. Set the virtual machine’s vCPU count to the result of that calculation.

If you own CapacityIQ it will make this calculation for you for every virtual machine in your data center. Here is an screenshot of its recommendations based on virtual machine CPU and memory utilization. Click for a clearer picture.

CapacityIQ monitors CPU and memory utilization to recommend VM downsizing.

esxtop shows three processor utilization rows. What do they mean?

This capture shows utilization of each physical and logical processor core in the system.  The first row, PCPU USED, provides the percent of each physical core used by the logical core, multiplied by turbo mode, a processor feature that temporarily increases the core’s internal clock frequency.  This means two threads running at full, turbo speed might produce a number like 55% for each entry. It also means that one thread can drive its logical CPU to 100% only if the logical core’s sibling is unused. The second row is a straightforward calculation of the utilization of each logical core and the third row similarly shows utilization of physical cores.

Where things get really confusing is when these results are combined into three system-wide, aggregate utilization numbers, as seen by esxtop’s batch printout.  The three utilization types above generated different utilization numbers.  Unfortunately esxtop’s batch mode labels these counters slightly differently.  But this table includes both names:

The “Single Core Example” column provides an example calculation based on threads running as fast as possible on a single Hyper-Threaded physical core.  There are some interesting observations on these calculations:

• Even a great number of threads running full bore on an HT system will not produce a PCPU USED(%) number much over 50%.
• Two running threads will produce a lower PCPU USED(%) than one running thread.  This is because each thread’s utilization is calculated against the physical core.  With two running, each is averaging 50% of the core.  But a single thread that is not sharing the core can drive this to 100%.  In both cases the actual number could be a little higher if turbo mode is on.
• You must have at least two threads–one on each logical core–to drive PCPU UTIL(%) to 100%.
• CORE UTIL(%) can be driven to 100% with only one thread per physical core.

Look for this content to be rolled into our esxtop documentation soon.

While memory over-subscription is a critical feature for production environments, balancing the demands of heterogenous applications of varying demands in a resource starved environment is difficult.  Without guidance from administrators on the relative importance of the virtual machines running these applications, a hypervisor will be forced to make arbitrary decisions in assigning limited resources.  Effective use of over-commitment requires a sound resource control system.  The only product on the market that does this well is VMware vSphere.

Both Greg and my articles only talked of memory over-commitment, but the rules apply for CPU over-commitment, too.  Microsoft will realize how important resource controls are somewhere between year two and five of their product’s life.  I can only imagine where vSphere will be by then.

On VI3, many of VMware’s customers disabled Hyper-Threading on their older, Netburst architecture Intel processors. Intel has vaguely described the new Hyper-Threading as more efficient than the previous generation and I believe this to be due to a shorter pipeline and an improved ability to context switch pipeline stage data. Long pipelines–such as the Netburst era Xeons of model numbers x1xx and x2xx–are more likely to suffer bubbles during context switches and are therefore penalized versus shorter pipeline products, such as the Core i7. Furthermore, by pushing and restoring pipeline stage data during a hardware context switch, the new HT can reduce pipeline bubbles.

But the gains vSphere users experience as a result of the new Hyper-Threading also comes from changes in ESX. ESX’s scheduler must make decisions as to when to co-locate two worlds on a physical core to take advantage of Hyper-Threading. In some conditions the scheduler will perform this co-location and in others it will allow a world to run on the core by itself. The decision to execute worlds concurrently instead of serially on a physical core can be informally called the scheduler’s trust of Hyper-Threading. The vSphere scheduler trusts Hyper-Threading more than the VI3 scheduler did. This amplifies the effect of HT.

I am now going to bore you with a disclaimer before I give you any data showing the effect of Hyper-Threading. The value of HT will vary from workload to workload and the ultimate authority of HT’s value is the end-user. The following numbers are the result of informal analysis and VMware that should only be used as a guide in your own analysis. Please do not make purchasing decisions on this information, which is devoid of the detail we would normally commit to a white paper.

In addition to the gains we informally cite here, I can say that we have not yet seen a workload where the new Hyper-Threading slows down consolidated performance. As far as we can tell, the new Hyper-Threading should be left enabled in 100% of virtualized environments.

I spend a great deal of time answering customers’ questions about the scheduler. Never have so many questions been asked about such an abstruse component for which so little user influence is possible. But CPU scheduling is central to system performance, so VMware strives to provide as much information on the subject as possible. In this blog entry, I want to point out a few nuggets of information on the CPU scheduler. These four bullets answer 95% of the questions I get asked.

Item 1: ESX 4’s Scheduler Better Uses Caches Across Sockets

On UMA systems at low load levels, virtual machine performance improves when each virtual CPU (vCPU) is placed on its own socket. This is because providing each vCPU its own socket also gives it the entire cache on that CPU. On page 18 of a recent paper on the scheduler written by Seongbeom Kim, a graph highlights the case where vCPU spreading improves performance.

The X-axis represents different combinations of VM and vCPU counts. SPECjbb is memory intensive and shows great gains with increases in CPU cache. The few cases that show dramatic benefit due to the ESX 4.0 scheduler are benefiting from the distribution of vCPUs across sockets. Very large gains are possible in this somewhat uncommon case.

Item 2: Overuse of SMP Only Slows Consolidated Environments At Saturation

For years customers have asked me how many vCPUs they should give to their VMs. The best guidance, “as few as possible”, seems too vague to satisfy. It remains the only correct answer, unfortunately. But a recent experiment performed by Bruce Herndon’s team sheds some light on this VM sizing question.

In this experiment we ran VMmark against VMs that were configured outside of VMmark specifications. In one case some of the virtual machines were given too few vCPUs and in another they were given too many. Because VMmark’s workload is fixed, increasing the VMs’ sizes does not increase the work performed by the VMs. In other words, the system’s score does not depend on the VMs’ vCPU count. Until CPU saturation, that is.

Notice that the scores are similar between the undersized, right-sized, and over-sized VMs. Up until tile 10 (60 VMs) they are nearly identical. There is a slight difference in processor utilization that begins to impact throughput (score) as the system runs out of CPU. At that point the additional vCPUs waste cycles which degrades system performance. Two points I will call out from this work:

• Sloppy VI admins that provide too many vCPUs need not worry about performance when their servers are under low load. But performance will suffer when CPU utilization spikes.
• The penalty of over-sizing VMs gets worse as VMs get larger. Using a 2-way VM is not that bad, but unneeded use of 4-way VMs when one or two processors suffice can cost up to 15% of your system throughput. I presume that unnecessarily eight vCPUs would be criminal.

Item 3: ESX Has Not Strictly Co-scheduled Since ESX 2.5

I have documented ESX’s relaxation of co-scheduling previously (Co-scheduling SMP VMs in VMware ESX Server). But this statement cannot be repeated too frequently: ESX has not strictly co-scheduled virtual machines since version 2.5. This means that ESX can place vCPUs from SMP VMs individually. It is not necessary to wait for physical cores to be available for every vCPU before starting the VM. However, as Item 3 pointed out, this does not give you free license to over-size your VMs. Be frugal with your SMP VMs and assign vCPUs only when you need them.

Item 4: The Cell Construct Has Been Eliminated in ESX 4.0

In the performance best practices deck that I give at conferences I talk about the benefits of creating small virtual machines over large ones. In versions of ESX up to ESX 3.5, the scheduler used a construct called a cell that would contain and lock CPU cores. The vCPUs from a single VM could never span a cell. With a ESX 3.x’s cell size of four this meant that VMs never spanned multiple four-core sockets. Consider this figure:

What this figure shows is that a 4-way VM on ESX 3.5 can only be placed in two locations on this hypothetical two-socket configuration. There are 12 combinations for a 2-way VM and eight for a uniprocessor VM. The scheduler has more opportunities to optimize VM placement when you provide it with smaller VMs.

In ESX 4 we have eliminated the cell lock so VMs can span multiple sockets, as item one states. Continue to think of this placement problem as a challenge to the scheduler that you can alleviate. By choosing multiple, smaller VMs you free the scheduler to pursue opportunities to optimize performance in consolidated environments

Newer processors are much more important to virtualized environments than the non-virtualized counterpart. Generational improvements have not just increased the raw compute power, they have also reduced virtualization overheads. This blog entry will describe three key changes that have particularly impacted virtual performance.

Hardware Assist Is Faster

In 2008, with the launch of the Opteron 1300, 2300 and 8300 parts, AMD became the first CPU vendor to produce a hardware memory management unit equipped to support virtualization. They called this technology Rapid Virtualization Indexing (RVI). This year Intel did the same with Extended Page Tables (EPT) on its Xeon 5500 line. Both vendors have been providing the ability to virtualize privileged instructions since 2006, with continually improving results. Consider the following graph showing the latency of one key instruction from Intel:

This instruction, VMEXIT, is called each time the guest exits to the kernel. The graph shows its latency (delay) in completing this instruction, which represents a wait time incurred by the guest. Clearly Intel has made great strides in reducing VMEXIT’s wait time from its Netburst parts (Prescott and Cedar Mill) to its Core architecture (Merom and Penryn) and on to its current generation, Core i7 (Nehalem). AMD processors have shown commensurate gains with AMD-V.

In a recent white paper detailing SQL Server on vSphere, the following graph showed the gains derived by using AMD-V in the Opteron 8324 (Shanghai).

Binary translation, AMD-V, and AMV-V plus RVI are measured using SQL Server.

This graph shows the practical value of the great gains that CPU manufacturers have made with virtualization assist.  Hardware assist can now be regularly relied upon for great performance.

Pipelines Are Shorter

The longest pipelines in the x86 world were in Intel’s Netburst processors. These processor’s pipelines had twice as many stages at their counterparts at AMD and twice as many as the generation of Intel CPUs that followed. The increased pipeline length would have enabled support for 8 GHz silicon, had it arrived. Instead, silicon switching speeds hit a wall at 4 GHz and Intel (and its customers) were forced to suffer the drawbacks of large pipelines.

Large pipelines are not necessarily a problem for desktop environments, where single threaded applications used to dominate the market. But in the enterprise, application thread counts were larger. Furthermore, consolidation in virtual environments drove thread counts even higher. With more contexts in the processor, the number of pipeline stalls and flushes increased, and efficiency fell.

Because of decreased efficiency of consolidated workloads on processors with long pipelines, VMware has often recommended that performance-intensive VMs be run on processors no older than 2-3 years. This excludes Intel’s Netburst parts. VI3 and vSphere will do a fine job at virtualizing your less-demanding applications on any supported processors. But you should use newer parts for applications that hold your highest performance expectations.

Caches Are Larger

A cache is highly effective when it fully contains the software’s working set. The addition from the hypervisor of even a small about of code will change the working set and reduce cache hit rate. I’ve attempted to illustrate this concept with the following simplified view of the relationship between cache hit rates, application working set, and cache sizes:

Performance drops with small cache systems for even small increases to working set size.

This graph is based on a model that greatly simplifies working sets and the hypervisor’s impact on them. Assuming that ESX increases the working set by 256 KB, this graph shows the decrease cache hit rate due to the contributions of the hypervisor. Notice that with very small caches and very small application working sets, the cache hit rate suffers greatly due to the addition of even 256 KB of virtualization code. And even up to 2 MB, a 10% decrease in cache hit rate can be seen in some applications. With a 256 KB contribution by the kernel, cache hit rates do not change significantly with cache sizes of 4 MB and beyond.

In some cases a 10% improvement in cache hit rate can double application throughput. This means that a doubling of cache size can profoundly effect the performance of virtual applications as compared to native. Given ESX’s small contribution to the working set, you can see why we at VMware recommend that customers run their performance-intensive workloads on CPUs with 4 MB caches or larger.