VMmark is clearly the reigning king of consolidation benchmarks and anything that enters its arena must stand against its standard. VMmark pioneered a new method of benchmarking that resonates with virtualization experts. It tests system performance by adding fixed load virtual machines instead of scaling up a single application to system saturation.  Traditional benchmarks tune up their load generation against a single instance but VMmark piles on virtual machines until the system is capable of no more work.

VMmark is one of VMware’s many industry-leading initiatives and was started when VMware worked closely with server vendors that wanted to benchmark their servers’ ability to run virtual machines. VMmark was conceived many years ago, well before VMware had competition. It is because of this fact that I scratch my head at claims that VMmark is biased towards VMware. There was no commercial implementation of Xen when VMmark was specified and Microsoft was only dreaming of entering the market.

But even in an environment devoid of competition, customers want certainty that their benchmarks are not hiding flaws in a product. SPEC has for years been developing honest benchmarks that survive the crucible of debate among its large member community. SPECvirt, or more properly SPECvirt_sc2010, is the result of this vigorous debate. You can read up on SPECvirt in the FAQ released coincident with the product’s launch. But I will add a few comments and comparisons here.

1. SPECvirt costs $3000 to purchase.  VMmark is free.  But VMmark requires commercial software and versions of SPEC benchmarks that are not free.  Depending on your licensing model, you may find VMmark or SPECvirt cheaper.  But the prices of each are essentially comparable.
2. VMmark uses the most common applications in the data center (like Apache and Microsoft Exchange).  SPECvirt does not mandate application choice for the system under test.
   • This is a Good Thing, because you may now choose a configuration that models your environment by running the exact applications you run.
   • This is a Bad Thing, because five different testers may choose five different application sets in their tests resulting in incomparable results.
3. SPECvirt cannot be run against a cluster of hosts.  But VMmark cannot, either.  We will have to wait for an update to one of these benchmarks before we can properly test DRS clusters and their competitive equivalents.
4. There is only one published SPECvirt result, courtesy of IBM running KVM.  There are a boatload of VMmark results, as one would expect of a more mature product.  It will be interesting to watch the rate of submissions of these two benchmarks over the coming year or two.
5. SPECvirt runs three workloads and an idle virtual machine in its tile.  One of those workloads, tested by SPECweb, is implemented with three virtual machines.  The end product is a six-VM tile that looks very much like VMmark’s six-VM tile.

For years we have seen online and in-person griping about VMware’s misunderstood benchmark restriction in its EULA.  Both VMmark and SPECvirt can be run on any supported hypervisor.  So now its time for all the hypervisor vendors to put up or shut up.  Run one of these benchmarks on your product and compare the results against existing published results.  Then the world will know where your product stands.

On VI3, many of VMware’s customers disabled Hyper-Threading on their older, Netburst architecture Intel processors. Intel has vaguely described the new Hyper-Threading as more efficient than the previous generation and I believe this to be due to a shorter pipeline and an improved ability to context switch pipeline stage data. Long pipelines–such as the Netburst era Xeons of model numbers x1xx and x2xx–are more likely to suffer bubbles during context switches and are therefore penalized versus shorter pipeline products, such as the Core i7. Furthermore, by pushing and restoring pipeline stage data during a hardware context switch, the new HT can reduce pipeline bubbles.

But the gains vSphere users experience as a result of the new Hyper-Threading also comes from changes in ESX. ESX’s scheduler must make decisions as to when to co-locate two worlds on a physical core to take advantage of Hyper-Threading. In some conditions the scheduler will perform this co-location and in others it will allow a world to run on the core by itself. The decision to execute worlds concurrently instead of serially on a physical core can be informally called the scheduler’s trust of Hyper-Threading. The vSphere scheduler trusts Hyper-Threading more than the VI3 scheduler did. This amplifies the effect of HT.

I am now going to bore you with a disclaimer before I give you any data showing the effect of Hyper-Threading. The value of HT will vary from workload to workload and the ultimate authority of HT’s value is the end-user. The following numbers are the result of informal analysis and VMware that should only be used as a guide in your own analysis. Please do not make purchasing decisions on this information, which is devoid of the detail we would normally commit to a white paper.

In addition to the gains we informally cite here, I can say that we have not yet seen a workload where the new Hyper-Threading slows down consolidated performance. As far as we can tell, the new Hyper-Threading should be left enabled in 100% of virtualized environments.

Number 5: Comparing Intel Dual-Core and Quad-Core Using VMmark

This article received 4,700 views in 2009.  I am not surprised that this two-year-old article is so popular, though.  Even this morning at Intel’s live chat I was asked for advice on selecting core/socket configurations.  Everyone wants to know if the cores on the largest servers can be put to good use.  The article summed up findings with this graph:

Doubling cores or sockets does not exactly double performance.  In this case the quad-core system produced 70% more throughput than its equivalent dual-core configuration.

Number 4: 100,000 I/O Operations Per Second, One ESX Host

In VMware’s early days customers were concerned that ESX lacked the storage throughput for demanding workloads.  In our first effort to dispel this rumor, we partnered with EMC to demonstrate a handful of VMs driving 100,000 IOPS on a single host.  A year later we updated those results on vSphere and showed VMs on ESX demanding 365,000 IOPS from EMC’s Enterprise Flash Devices (EFDs).  But the original article remains the more popular and has received 8,000 hits in 2009.

Number 3: Virtualizing XenApp on XenServer 5.0 and ESX 3.5

I guess that people love a good argument.  Our customers have heard enough misinformation on vSphere and XenServer performance that our performance team wanted to weigh in with a rare competitive comparison.  That article has received 13,400 hits in 2009.  It details an experiment in which we used a workload that will eventually be released as View Planner to demonstrate scalability performance of VMware’s and Citrix’s respective products.

Desktop and terminal services performance is continually a hot topic.  Unlike enterprise applications, there is no industry standard for measuring virtual desktop performance.  VMware has favored an approach that uses many operations on the most common desktop applications.  Every other public comparison either restricts is measurements to a single operation or favors easy automation over application relevancy.

Number 2: Ten Reasons Why Oracle Databases Run Best on VMware

In his first public work at VMware, our Chief Performance Architect, Richard McDougall, showed us how good he is at evangelizing technology to large audiences.  His article has received 14,300 hits in 2009 despite its 2007 publication date.  Oracle databases are incredibly popular topics among VI administrators and Richard was the first to point out that VI3’s storage throughput ability was far beyond the needs of the average 4-way Oracle database:

Back in 2007 we were trying to convince everyone to virtualize their demanding Oracle databases.  We knew that VI3 could handle it and our early adopters knew, too.  But since vSphere launched, everyone has known.  If you think that vSphere cannot handle your most demanding Oracle databases, think again.

Number 1: Windows Vista Performance in VMware Workstation 6.0

I attribute the fact that fact that this aging article has received 15,000 hits in 2009 to the incredible size of consumer desktop products as opposed to enterprise software and hardware. But it is worth mentioning that this article was written by Zhelong Pan, the developer responsible for esxtop, who has also written the most popular article on the VMware performance communities. That guy has the magic touch!

I spend a great deal of time answering customers’ questions about the scheduler. Never have so many questions been asked about such an abstruse component for which so little user influence is possible. But CPU scheduling is central to system performance, so VMware strives to provide as much information on the subject as possible. In this blog entry, I want to point out a few nuggets of information on the CPU scheduler. These four bullets answer 95% of the questions I get asked.

Item 1: ESX 4’s Scheduler Better Uses Caches Across Sockets

On UMA systems at low load levels, virtual machine performance improves when each virtual CPU (vCPU) is placed on its own socket. This is because providing each vCPU its own socket also gives it the entire cache on that CPU. On page 18 of a recent paper on the scheduler written by Seongbeom Kim, a graph highlights the case where vCPU spreading improves performance.

The X-axis represents different combinations of VM and vCPU counts. SPECjbb is memory intensive and shows great gains with increases in CPU cache. The few cases that show dramatic benefit due to the ESX 4.0 scheduler are benefiting from the distribution of vCPUs across sockets. Very large gains are possible in this somewhat uncommon case.

Item 2: Overuse of SMP Only Slows Consolidated Environments At Saturation

For years customers have asked me how many vCPUs they should give to their VMs. The best guidance, “as few as possible”, seems too vague to satisfy. It remains the only correct answer, unfortunately. But a recent experiment performed by Bruce Herndon’s team sheds some light on this VM sizing question.

In this experiment we ran VMmark against VMs that were configured outside of VMmark specifications. In one case some of the virtual machines were given too few vCPUs and in another they were given too many. Because VMmark’s workload is fixed, increasing the VMs’ sizes does not increase the work performed by the VMs. In other words, the system’s score does not depend on the VMs’ vCPU count. Until CPU saturation, that is.

Notice that the scores are similar between the undersized, right-sized, and over-sized VMs. Up until tile 10 (60 VMs) they are nearly identical. There is a slight difference in processor utilization that begins to impact throughput (score) as the system runs out of CPU. At that point the additional vCPUs waste cycles which degrades system performance. Two points I will call out from this work:

• Sloppy VI admins that provide too many vCPUs need not worry about performance when their servers are under low load. But performance will suffer when CPU utilization spikes.
• The penalty of over-sizing VMs gets worse as VMs get larger. Using a 2-way VM is not that bad, but unneeded use of 4-way VMs when one or two processors suffice can cost up to 15% of your system throughput. I presume that unnecessarily eight vCPUs would be criminal.

Item 3: ESX Has Not Strictly Co-scheduled Since ESX 2.5

I have documented ESX’s relaxation of co-scheduling previously (Co-scheduling SMP VMs in VMware ESX Server). But this statement cannot be repeated too frequently: ESX has not strictly co-scheduled virtual machines since version 2.5. This means that ESX can place vCPUs from SMP VMs individually. It is not necessary to wait for physical cores to be available for every vCPU before starting the VM. However, as Item 3 pointed out, this does not give you free license to over-size your VMs. Be frugal with your SMP VMs and assign vCPUs only when you need them.

Item 4: The Cell Construct Has Been Eliminated in ESX 4.0

In the performance best practices deck that I give at conferences I talk about the benefits of creating small virtual machines over large ones. In versions of ESX up to ESX 3.5, the scheduler used a construct called a cell that would contain and lock CPU cores. The vCPUs from a single VM could never span a cell. With a ESX 3.x’s cell size of four this meant that VMs never spanned multiple four-core sockets. Consider this figure:

What this figure shows is that a 4-way VM on ESX 3.5 can only be placed in two locations on this hypothetical two-socket configuration. There are 12 combinations for a 2-way VM and eight for a uniprocessor VM. The scheduler has more opportunities to optimize VM placement when you provide it with smaller VMs.

In ESX 4 we have eliminated the cell lock so VMs can span multiple sockets, as item one states. Continue to think of this placement problem as a challenge to the scheduler that you can alleviate. By choosing multiple, smaller VMs you free the scheduler to pursue opportunities to optimize performance in consolidated environments