Among the many results presented by VRC, the configurations that most perplexed us were the two and four virtual machine configurations, each with four vCPUs per virtual machine.  The configuration with two virtual machines looked good and matched our internal numbers.  In this configuration there are a total of eight vCPUs on the host which maps each to its own physical core on the Xeon 5500 series processor.  The problem arose when the virtual machine count was increased to four, resulting in 16 total vCPUs.  In this configuration each vCPU is paired with one logical, Hyper-Threaded core.  Project VRC showed this configuration supporting no more desktops than the two-VM configuration, which suggests no value to Hyper-Threading on this configuration.

It took us some time to understand the reason for these results, but we eventually identified a very specific condition where ESX’s scheduler enforces fairness in scheduling vCPUs at at cost of throughput.  ESX’s scheduler has long be subject of the intensive scrutiny of a large number of VMware engineers to guarantee fair access to the processor for each virtual machine.  It is because of this fairness that VMware’s customers can rely on CPU resource controls.  But, when fairness goes too far, throughput may be sub-optimal.

Hyper-Threading presents particular problems to fairness because of the non-linear performance it delivers.  A thread will run at one speed when it has full access to a physical core, at another speed when it is sharing a core, and at third speed when sharing a core with a different thread.  As a result, ESX’s scheduler will sometimes pause a thread to enforce fairness.  These pauses are more common when Hyper-Threading is present to account for its lack of uniformity in thread performance.  If the host lacks vCPUs that are ready to run, the result is CPU utilization below saturation, leaving CPU cycles unused.

There are three specific conditions that can excite this condition:

1. A Xeon 5500 series processor is present with Hyper-Threading enabled,
2. CPU utilization is near saturation, and
3. A roughly one-to-one mapping between vCPUs and logical processors.

In this scenario, VMware vSphere favors fairness over throughput and sometimes pauses one vCPU to dedicate a whole core to another vCPU, eliminating gains provided by Hyper-Threading.  In cases outside of these three conditions, the performance of VMware vSphere 4 meets the high expectations of VMware’s R&D team and its customers.  Of course production environments rarely (never?) have a one-to-one ratio of vCPUs to logical processors.  This occurs when there are only four 4-way virtual machines on a Xeon 5500 system, for example.

But environments such as Project VRC’s are simplifications of production environments meant to understand the capabilities of virtual platforms.  VMware has provided a patch to Project VRC that will allow them to improve throughput in their environment.  We are going to release this patch and its documentation to the general public within a couple of weeks.  I do not expect that any of VMware’s customers will benefit from the changes is allows, but I will later document the patch and its usage for anyone that cares to experiment.

On VI3, many of VMware’s customers disabled Hyper-Threading on their older, Netburst architecture Intel processors. Intel has vaguely described the new Hyper-Threading as more efficient than the previous generation and I believe this to be due to a shorter pipeline and an improved ability to context switch pipeline stage data. Long pipelines–such as the Netburst era Xeons of model numbers x1xx and x2xx–are more likely to suffer bubbles during context switches and are therefore penalized versus shorter pipeline products, such as the Core i7. Furthermore, by pushing and restoring pipeline stage data during a hardware context switch, the new HT can reduce pipeline bubbles.

But the gains vSphere users experience as a result of the new Hyper-Threading also comes from changes in ESX. ESX’s scheduler must make decisions as to when to co-locate two worlds on a physical core to take advantage of Hyper-Threading. In some conditions the scheduler will perform this co-location and in others it will allow a world to run on the core by itself. The decision to execute worlds concurrently instead of serially on a physical core can be informally called the scheduler’s trust of Hyper-Threading. The vSphere scheduler trusts Hyper-Threading more than the VI3 scheduler did. This amplifies the effect of HT.

I am now going to bore you with a disclaimer before I give you any data showing the effect of Hyper-Threading. The value of HT will vary from workload to workload and the ultimate authority of HT’s value is the end-user. The following numbers are the result of informal analysis and VMware that should only be used as a guide in your own analysis. Please do not make purchasing decisions on this information, which is devoid of the detail we would normally commit to a white paper.

In addition to the gains we informally cite here, I can say that we have not yet seen a workload where the new Hyper-Threading slows down consolidated performance. As far as we can tell, the new Hyper-Threading should be left enabled in 100% of virtualized environments.

VMware’s dedicated and curious customer base somehow obtained information on an undocumented parameter that limits the maximum number of concurrent VMotions.  Information on modifying this parameter is sprinkled around the internet tubes but my favorite comes from Jason Boche.  Jason and others have identified how you can change the current limit of two VMotions per host.  I want to give you an explanation of why you should not do this and show you what you can expect from future releases of vSphere.

The concurrent VMotion limit was set after careful analysis of the capabilities of existing hardware, VMotion implementation details, and a large number of enterprise applications.  It has been set at two to provide a near 100% guarantee of no downtime during the migrations.  On some workloads or on older hardware, it is possible that three or more concurrent migrations could saturate a system resource and result in downtime.  If your hardware is brand new and your application is not wildly touching memory, it is possible that you can somewhat safely increase the concurrent VMotion limit.  But, as Kit Colbert told me, “I think allowing four simultaneous VMotions is probably OK in most scenarios, but if the VMs are really large and/or have very big working sets, then I’d dissuade customers from bumping up the limits.”

And if Kit’s gentle reminder is not enough to dissuade you from making this change to your production environments, I will point out that problems that arise as a result of changing the concurrent VMotion limit are not supported by VMware.  We simply cannot promise the unfaltering quality of VMotion if end users increase this limit.

Now, back to the feature preview in my ongoing performance road show (today: Bellevue, WA!).  Using an in-house development version of ESX, we are running eight concurrent VMotions on a single host with the even better quality than that of vSphere 4.  A phenomenally dedicated group of engineers has drastically improved the throughput of VMotions and decreased the already tiny virtual machine stun time.  This means that we can maintain our zero downtime commitment while upping the number of concurrent VMotions by a factor of four.  Furthermore, total migration time has also decreased, so our development host can evacuate a large number of virtual machines almost order of magnitude faster than vSphere 4.

(*) Any time I talk about this or other unreleased features the standard VMware disclaimer applies.  We are not committing this feature to any specific product nor committing any product to a specific date.  Not my rules, guy

To get the most out of your servers’ memory you have to over-commit it.  Go too far and the host will swap.  This fact is immutable in enterprise memory management.  Because rotating disks have seek times that are six orders of magnitude larger than memory, when your host swaps your applications’ performance suffers catastrophically.  Solid state drives (SSD) can mitigate the performance cost by reducing swap latency by a couple orders of magnitude.  But SSDs still have delays tens of thousands of times worse than memory.

VMware engineers have been working on a technology that we unofficially call on-demand memory compression (ODMC) or compression cache, depending on who you talk to.  The idea of ODMC is to avoid swapping by compressing a set of target pages to a special region.  We have measured the additional latency–the compression time–to be over a hundred times better than rotating disk latencies.  And this will decrease at CPU performance increases.

SSD devices will continue to play an important part in solving the performance problems of extreme memory over-commit because of the great volume of data they can serve at a speed much faster than rotating disks.  But it will be a secondary solution to the faster ODMC, which uses a small dedicated area of system memory.  VMware’s long-term prioritization for managing the most aggressively over-committed memory looks like this:

1. Do not swap if possible.  We will continue to leverage transparent page sharing and ballooning to make swapping a last resort.
2. Use ODMC to a predefined cache to decrease memory utilization.*
3. Swap to persistent memory (SSD) installed locally in the server.**
4. Swap to the array, which may benefit from installed SSDs.

(*) Demonstrated in the lab and coming in a future product.

(**) Part of our vision and not yet demonstrated.

The end goal of this prioritized use of different technologies is the reduction of the performance penalty of swap.  If VMware can reduce the penalty of swap, the virtual machines in extremely over-committed environments will not slow much when they need more memory than the host has available.  When this performance cost of swapping is reduced, you can safely drive your consolidation ratios even higher.

I cannot wait for the general availability of ODMC.  It is another technological accomplishment in a long line of innovation out of our phenomenal engineering organization.  At times I wonder when the last time another operating system software company introduced something really awesome to the industry.  But of course, I am biased.

I think this rumor sprung from our somewhat limited support of paravirtualized drivers in FT-protected virtual machines, which will be improved in a subsequent release.  And while most of you probably know that PVSCSI and vmxnet3 run together, I thought it worth a brief comment on this blog.  Myths are like cockroaches.  For every one you see there are hundreds hiding behind the walls.

Last year was my first year captaining the performance troubleshooting lab.  I designed the content around the performance problems I see most frequently in customer deployments. On the philosophy that every attendee should get their money’s worth in the two hour slot, I packed in more experiments than anyone could finish in the allotted time.  That worked great for experienced VI admins, but some of the more novice administrators walked out with their heads spinning.

I do want to continue challenging even the strongest admins that come to the lab, but I must provide enough basics to make the lab enjoyable for everyone.  With that requirement in your mind, maybe you can offer ideas as to how the lab should change and what content you would like to see.  Here are a few primer questions:

• What comments do you have from last year’s lab?  Did I succeed in my struggle to make the lab interesting to VCPs and beginners alike?  If not, which side of that spectrum did I favor?
• What content do you want to see added?  A few ideas:
  • Swap management: managing swap through swap file placement, use of SSDs, and memory compression*
  • Storage IO Control*: using shares and limits to maintain performance of critical VMs in the presence of storage bottlenecks
  • NFS performance diagnosis*
  • Managing storage alignment
  • Guest defgragmentation impact
  • Anything else you are itching for?
• What content should we should repeat with updated examples?
• What do you want us to avoid repeating?  Have you learned all you need to know about vscsiStats, for instance?
• What would you think if we made this lab self-paced but promised performance experts in the room and on-call for the entire day?

(*) Coming soon to a hypervisor near you.

Put your comments here and they may influence the content of this lab.

When people ask me about file system fragmentation I explain that fragmentation can come from two sources: the guest file system or VMFS.  In 2009 we included experiments in our thin provisioning white paper that showed that both internal and external fragmentation in VMFS have no significant effect on performance.  As for guest fragmentation, VMware has avoided the business of optimizing native operating systems so there is no extant, official guidance.

More precisely, the large number of mappings from the guest file to the disk make it difficult to know how changes to each can impact the system’s performance as a whole.  But in talking with this partner I realized that there are two inescapable truths that suggest guest defragmentation is critical in a virtualized environment:

1. Defragmentation can decrease the number of disk commands and the resultant IOPS.
2. The fewer IOs, the more efficient the virtualization.

Guest defrag tools will order each file’s blocks sequentially in the guest file system.  This will enable the guest to make a few number of calls to larger, contiguous data than had the blocks been separated on the guest file system.  By making fewer calls to larger blocks, the following things happen:

• The array can leverage its faster sequential access capabilities to improve storage throughput.
• The hypervisor handles fewer SCSI messages from the guest resulting in lower overhead.
• The smaller number of commands results in fewer outstanding operations in the 32-element HBA queue, which allows more virtual machines to access the storage concurrently.

I have not found out how much consolidated workloads have to gain from guest defragmentation.  Nor have I quantified the impact to shared storage of a shift from a larger number of small commands to a smaller number of large commands.  But I am going to work with this partner and see if we can publish some numbers in 2010.

During my short cab ride from the Mandalay Bay to McCarran (LAS) the driver and I discussed routes and fares to and from the airport.  The route from the airport, favored by taxis that pick you up at LAS, follows the highway, starting on I-215 and then to I-15, at a cost of around $28.  This as opposed to the more direct, airport-bound trip on Paradise, Tropicana and the Boulevard, which takes less time and totals only $11.50.  I asked my driver, “Why do the taxis to the airport take the second, shorter, cheaper route when the hotel-bound taxis always take the highway?”  The driver answered simply: “Airport taxis are f****** crooks.”

I have spent a little time investigating these two routes on Google maps and they do not seem to be dramatically different in terms of total length.  But clearly the fares differ widely.  I am sure the taxis prefer a larger fare but might there be an innocuous reason for the highway route?  Perhaps the risk of traffic delay is decreased by taking the highway and avoiding the strip?

My driver today seemed certain that greed fuels the decision to take the longer route.  He said that no visitor should ever pay more than $12 for a ride from the airport.  He advised me to refuse payment to the next cabbie that took me for the long ride.  He also said a call to the taxi cab company, which gains from the overcharge, would have no effect.  But a call to the taxi cab authority is justified and would result in a $300 fine to the taxi company for the long haul.

Has anyone ever taken the short route from the airport?  Are there any problems with getting taxis to take that route instead of the highway?  Is there a good reason to take the highway instead of surface streets?

Usually this issue arises inside a new VMware customer or an established customer that has added new staff to the virtualization team. In both cases the administrators are familiar with existing tools and require good reason to retool their thinking and environment around a new measurement system.

I was discussing the response to these concerns with my friend and colleague Kaushik Banerjee, the head of VMware’s outbound engineering group.  Kaushik and I spend a lot of time thinking about communicating technical details to our customers and in this case we chose different approaches to answer the question.  Both responses are complementary, so choose the one that suits your needs.

Kaushik’s Approach: The Killer Examples

Kaushik suggested that if we show cases where the guest OS’s counters were obviously wrong that a naturally suspicious VI admin would never trust the guest counters again. To that end, I offer the following screen shots to make our point.

Perfmon's counters show utilization higher in the guest than the host reports.

This screen shot shows two counters available in Perfmon inside a Windows guest with the vmStatsProvider installed (available by default since vSphere). The darker, red line is the CPU utilization as reported by the guest. The lighter, greenish (?) line is CPU utilization of the virtual machine, from the host’s perspective. This is the real CPU utilization passed up to the host by vmStatsProvider.  Notice how the host is always reporting higher utilization than the guest.  This is due to one of the reasons why guest counters cannot be trusted: they are unaware of hypervisor overheads.

This second screen shot shows a different case where the host utilization is lower than that reported by the guest.  Again, the dark red line represents the guest OS’s report of CPU utilization and the lighter line shows the real CPU utilization as reported by ESX.

Perfmon's counters report a higher CPU utilization than ESX's.

The reason the host shows lower utilization than the guest is because the guest is unaware that it is only getting a fraction of the host’s CPU, time-sliced by ESX’s scheduler.  In this case the virtual machine was contending for CPU with other active virtual machines but this just the same principle would apply had a CPU limit been set.

Scott’s Approach: A Detailed Explanation

My approach to convincing VI admins to avoid guest tools is based on bottomless thirst for information that is common to technophiles. If I can provide an explanation for the underlying system of resource scheduling and manipulation, then our admins will be able to deal with the guest counter issue and maybe solve other issues with their newfound knowledge.

There are four reasons why guest counters cannot be trusted:

1. The guest is unaware of virtualization overheads.  As screen shot one showed above, the hypervisor will increase the CPU load as it virtualizes the hardware for the guest operating system.  That additional CPU work is not seeing by guest tools.
2. The guest is unaware that it is only seeing the portion of CPU that ESX’s scheduler is allowing it to see.  Because of contention or resource restrictions, virtual machines only get a slice of the CPU’s time.  When a guest thinks it is getting 100% of the CPU it may not know that the processor is being shared by eight other virtual machines. See the second screen shot above.
3. Time skew in virtual machines can change the sample window for time-based counters.  This means that the guest may have measured 10 milliseconds of time passage during a read command when 12 milliseconds have elapsed.  This is more common on older versions of ESX and when the host CPU is saturated.  More on this below.
4. The virtual machines are unaware that they are being de-scheduled when idle, which means that they appear to be working more of the time than they are.  Consider a case where a virtual machine is idle 90% of the time.  If ESX does not schedule the VM during its idle time then the guest will think that its processor queues are full 100% of the time that it is being executed.

The time drift explanation (item three) was historically the most problematic for VMware.  On older versions of our products time drift was common.  As ESX has matured we have reduced the amount of drift which has improved the accuracy of guest counters.  But the timer hardware is still being virtualized in software running on the host CPU.  This means that if host processor is fully utilized, the timer may not be scheduled on time, resulting in a delay in some ticks and a resultant skew in guest time.

References

Performance Best Practices and Benchmarking Guidelines.  This white paper was the last version that we printed that included benchmarking best practices, which contains some discussion on the need to measure performance from outside the host-under-test.

Timekeeping in Virtual Machines.  This document–not updated since VI3 but still accurate in its theory–will give the background on VMware-based time keeping and provide an explanation as to how skew can occur.

VMware vSphere™ 4: The CPU Scheduler in VMware® ESX™ 4.  This white paper provides great detail how the scheduler works which will fully explains the notion of time slicing.

One technique for storage driver efficiency improvements is interrupt coalescing.  Coalescing can be thought of as buffering: multiple events are queued for simultaneous processing.  For coalescing to improve efficiency, interrupts must stream in fast enough to create large batch requests. Otherwise the timeout window will pass with no additional interrupts arriving.  This means the single interrupt is handled as normal but after a useless delay.

An intelligent storage driver will therefore coalesce at high IO but not low IO.  In the years we have spent optimizing ESX’s LSI Logic virtual storage adapter, we have fine-tuned the coalescing behavior to give fantastic performance on all workloads.  This is done by tracking two key storage counters:

• Outstanding IOs (OIOs): Represents the virtual machine’s demand for IO.
• IOs per second (IOPS):  Represents the storage system’s supply of IO.

The robust LSI Logic driver increases coalescing as OIOs and IOPS increase.  No coalescing is used with few OIOs or low throughput.  This produces efficient IO at large throughput and low latency IO when throughput is small.

Currently the PVSCSI driver coalesces based on OIOs only, and not throughput.  This means that when the virtual machine is requesting a lot of IO but the storage is not delivering, the PVSCSI driver is coalescing interrupts.  But without the storage supplying a steady stream of IOs there are no interrupts to coalesce.  The result is a slightly increased latency with little or no efficiency gain for PVSCSI in low throughput environments.

LSI Logic is so efficient at low throughput levels that there is no need for a special device driver to improve efficiency.  The CPU utilization difference between LSI and PVSCSI at hundreds of IOPS is insignificant.  But at massive amounts of IO–where 10-50K IOPS are streaming over the virtual SCSI bus–PVSCSI can save a large number of CPU cycles.  Because of that, our first implementation of PVSCSI was built on the assumption that customers would only use the technology when they had backed their virtual machines by world-class storage.

But VMware’s marketing engine (me, really) started telling everyone about PVSCSI without the right caveat (“only for massive IO systems!”)  So, everyone started using it as a general solution.  This meant that in one condition–slow storage (low IOPS) with a demanding virtual machine (high OIOs)–PVSCSI has been inefficiently coalescing IOs resulting in performance slightly worse than LSI Logic.

But now VMware’s customers want PVSCSI as a general solution and not just for high IO workloads.  As a result we are including advanced coalescing behavior in PVSCSI for future versions of ESX.  More on that when the release vehicle is set.

PVSCSI In A Nutshell

If you plodded through the above technical explanation of interrupt coalescing and PVSCSI I applaud you. If you just want a summary of what to do, here it is:

• For existing products, only use PVSCSI against VMDKs that are backed by fast (greater than 2,000 IOPS) storage.
• If you have installed PVSCSI in low IO environments, do not worry about reconfiguring to LSI Logic.  The net loss of performance is very small.  And clearly these low IO virtual machines are not running your performance-critical applications.
• For future products*, PVSCSI will be as efficient as LSI Logic for all environments.

(*) Specific product versions not yet announced.

Update: February 16

The simple, almost austere KB on this rare occurrence raised more questions than answers. You may notice that the KB has been updated with text from this blog since the blog’s original publication. A white paper on PVSCSI that had been under construction for quite some time was also released with a VROOM! article we often pair with such a white paper.