The first tool in the VI administrator’s toolbox should always be vCenter. Through the vSphere client you can use vCenter’s performance counters to confirm a problem with any of the four resources (storage, CPU, memory, network). vCenter’s 20 second sample window impedes its ability to eliminate a resource as a problem. This is because a three second spike in any resource will be smoothed and missed over the 20 second window. But when vCenter confirms a sustained resource bottleneck, it is sure to be the performance problem’s cause.

If vCenter fails to confirm an obvious performance problem, the administrator must next go to more precise, more time-intensive, and more knowledge-intensive tools such as esxtop and vscsiStats. esxtop takes more skill and time than vCenter but provides better resolution and more visibility into the system. vscsiStats is the most time-intensive tool and has limits with ESXi hosts but can uncover a world of detail invisible to esxtop and vCenter.

I estimate each tool’s chance of identifying a random performance problem as follows:

• vCenter: used in 90% of performance problems
• esxtop: used in 9% of problems
• vscsiStats: used 0.9% of the time

The remaining 0.1% of the time is when you engage your account team or your local VMware performance expert.

Even within each tool’s usage there is an hierarchy of investigation: storage, CPU, memory and network. My experience with troubleshooting has informed this decision.  Storage causes the most problems, then CPU, then memory, and lastly (and rarely) network. After each resource level is inspected in vCenter, a repeat of the inspection should occur on esxtop. Guest tools may be a third option for memory, CPU, and network but vscsiStats should always be consulted if the performance problem persists.

VMware’s growing array of performance management tools will change this flow somewhat. AppSpeed, for instance, adds the ability to make very educated guesses about resource bottlenecks based on inside information into the application execution.  Hyperic can provide in-guest process visibility and Ionix ADM will map application interdependenies to focus the investigation.  But, I will abstain from providing best practices on these tools until I have used them more. In all cases, however, the fundamental relationship of “easy first, precise later” remains.

VMware continues to work towards integrating all of these tools into a single view within the vSphere client. I expect that integration will improve the success rate of the performance layman in troubleshooting these problems. But I am sure that even into the distant future performance people will find their jobs secure.

ESX must expend CPU cycles to maintain running virtual CPUs whether they are being used by an application or not. This means that host efficiency drops as more vCPUs are put on the server. But applications that scale well with CPUs will deliver greater performance when their virtual machines have been given more CPUs. The administrator must therefore balance the desires of an individual application’s owner with the needs of the entire cluster’s of applications.

There are several resources that VI administrators can use to inform their decisions in virtual machine sizing. I have listed some of them below.

Bruce Herndon’s Cost-of-SMP Article

Last summer the VMmark team’s Bruce Herndon published an article on the cost of SMP. I summarized his findings in a vPivot article I wrote on the ESX 4 scheduler. There are two key messages that you can take away from these posts to inform your decisions on virtual machine sizing:

• Over-sized virtual machines only hurt system performance when the server’s CPUs are saturated.  When utilization is low, unneeded vCPUs only penalize the system’s CPU utilization, not the applications’ performance.
• Unneeded 2-way virtual machines are not very harmful to the environment.  But administrators should be very careful with 4-way virtual machines and larger.

Co-stop and Ready Time

Ready time indicates a vCPU waiting for an available core when it has work to perform. Co-scheduling stop time (or co-stop time) indicates a vCPU being paused by the scheduler to allow its sibling vCPUs to catch up. These two counters can help administrators recognize a certain kind of stress due to limited CPU resources.

Ready time is generally a sign of the unavailability of CPU. Correction usually requires the administrator reducing work on the host (migrating virtual machines, decreasing vCPU count, etc.) or increasing CPU capacity (more hosts or faster CPUs). Co-stop time is a sign that the scheduler is allowing vCPUs to develop skew while it runs portions of virtual machines on available cores. Considerable numbers for these counters are 10% ready time and 3% co-stop time. There is no guarantee that application performance is suffering if these thresholds are crossed, but a problem may be present.

The important thing about ready time and co-stop time is that they are signs that you are using all of the CPU you have available to you. This could be a Good Thing. But it could also be a surprise to you. When these counters get high it is a good time to start asking yourself if you capacity usage meets your expectations. If not, you should inspect your virtual machines to be sure that the applications are using the vCPUs you have given them. If your guest tools show poor in-guest utilization then decrease those VM sizes. That will free up resources in the cluster for more virtual machines.

Application Scalability Information

I wish we lived in a world where every ISV published data showing their applications’ abilities to scale with cores. Unfortunately for us, many software vendors have for years allowed their customers to assume that each doubling of cores would double the performance of the application. VMware has chosen to provide some scalability information so our customers know how well or how poorly applications scale. But every customer of a software company deserves to have the vendor provide guidance on sizing the server. And those vendors deserve the right to put these results out on their own products. Go talk to your ISV to get the information you need to size your virtual machines.

CPU Usage Calculations and CapacityIQ

I am belatedly updating this post with a fourth way of identifying oversized virtual machines: mathematical calculation or Capacity IQ.

When a virtual machine consistently uses only a fraction of its vCPU resources it is possible that the virtual machine can be downsized and still deliver the same application performance. The calculation to determine this is simple: multiply the vCPU count by utilization and round up. Set the virtual machine’s vCPU count to the result of that calculation.

If you own CapacityIQ it will make this calculation for you for every virtual machine in your data center. Here is an screenshot of its recommendations based on virtual machine CPU and memory utilization. Click for a clearer picture.

CapacityIQ monitors CPU and memory utilization to recommend VM downsizing.

Understanding the Bottleneck

Let me show the value of over-commit and danger of swapping by way of an example. I will choose the following typical values to demonstrate my point:

• All virtual machines are on a single host which has 32 GB of RAM installed.
• Each virtual machine is sized to 8 GB of RAM.
• Each virtual machine has 25% active memory (%ACTV in esxtop and “Active” in vCenter).

A virtual machine’s active memory is dictated by the application and its usage.  But the VI admin has complete control over the number of virtual machines in the environment which means host active memory can be influenced by adding or removing virtual machines.  Because virtual machine active memory is always equal to or less than 100% the only way to drive the host active memory to 100% is to over-commit memory.  This is why hypervisors that do not support memory over-commit are simply not viable for data centers where memory optimization is a priority.

Identifying and Correcting the Bottleneck

The ongoing occurrence of swapping is identified by a non-zero swap rate in either esxtop or vCenter.  In addition to swap rate, esxtop provides a swap wait time in its CPU panel.  When swap rate exceeds hundreds of kilobytes per second or swap wait time exceeds a couple percentage points, it is time for corrective action.

There are three possible solutions to this problem:

1. Balance the virtual machines’ memory usage by moving virtual machines from hosts with higher amounts of memory usage to hosts with lower amount of memory usage.
2. Run fewer virtual machines.
3. Buy more memory.

Designing Your Infrastructure to Simplify Memory Management

Ultimately I owe you a full white paper on memory management to provide a sufficient answer. But I want to give you two ideas of the tools and techniques that I will be describing when in this future paper.  First, place host swap files on solid state disk (SSD) stores to improve their performance.  With the right SSD device it may be possible to eliminate swap penalties.  Second, even if SSDs are unavailable consider consolidating multiple swap files onto a single store.  This will make swap rate monitoring very easy but may compound the performance penalties of swapping.

Stay tuned and VMware will provide more documentation on memory management in 2010.

The article describes a phenomena common to consolidated workloads called micro-bursting.  Micro-bursting occurs in such short periods as to go unnoticed in the sampling window of monitoring tools.  As Chad put it:

Remember that every metric has a timescale.   IOps is in seconds.   Disk service time is in ms (5-20ms for traditional disk, about 1ms for EFD).  If an I/O is served from cache, it’s in microseconds.   Switch latencies are in microseconds.    Here, the I/O periods were so short that they filled up the ESX LUN queues instantly, causing a “back-off” effect for the guest.   These were happily serviced by the SAN and the storage array, which had no idea anything bad was going on.

When these bursts happen queues overflow, messages backup, and service times briefly sky rocket.  These rapid overflows happen in a fraction of esxtop’s multi-second window and vCenter’s 20 second window.

So, what buffers are we talking about?  Take a look at Chad’s hand-drawn picture of the storage path, which is only slightly less complicated than the Republican view of Obamacare:

Chad Sakac's image showing the numerous locations of storage queues in all locations from the VM to the platter.

If you are at VI admin, you care about the LUN queue in ESX.  ESX creates one of these queues for each HBA+LUN pair.  So, multipathing to a LUN increases the effective LUN queue and using a single HBA to multiple LUNs will guarantee a queue to each LUN.  Instances of this queue will overflow if many VMs on a single server issue commands to a single LUN.  As Chad says:

In VMware land – this is usually the fact that the default LUN queue (and corresponding Disk.SchedNumReqOutstanding value) are 32 – which for most use cases is just fine, but when you have a datastore with many small VMs sitting on a single LUN, the possibility of microbursting patterns becomes more likely.

So, when will the queues overflow?  Not often:

In the example [Vaughn] used, [multi-pathing] would not help materially if there were more than 3 ESX hosts, as it would be a likely case of “underconfigured array” – not host-side queuing.

The message here is that there is only a small window of configurations will result in LUN queue overflow: many VMs on very few hosts talking to a common LUN.  This is a perfect use case for vscsiStats, which I have talked about in various forums now.  vscsiStats avoid sampling windows by recording precise information on every IO.  This means that microburst statistics will not be averaged–and lost–across a time period.

Consider the following data I pulled from a sample session on my office system:

This table shows the number of outstanding IOs as each new IO arrives in the VMkernel.  The first row means that during the collection period only two IOs arrived to a queue with one outstanding IO.  Row two says that two IOs entered when there was were two outstanding IOs.  The third row states that 50 IOs arrived while the queue had 3-4 IOs.  And so on.

This table represents a fairly healthy access pattern, showing that only 433 out of 357,402 IOs arrived while the queue had 33-64 outstanding IOs (shown on the last row).  With ESX’s default LUN queue depth at 32, vscsiStats shows that a very small number of IOs arrived to an overflowing queue.

In summary, some storage performance issues appear and disappear so rapidly as to not be visible with sampling based tools, even as fine-grained as esxtop.  As a VI admin you should consider this in your most challenging troubleshooting cases.  And remember to use vscsiStats if all else has failed.

A few days after the conference a coworker of mine shared the following quote with me, courtesy of an article by David on Internet Evolution:

The fact is, most data center managers wouldn’t trust VMware to manage their Tier 1 applications because if something goes wrong performance-wise, you still need to roll in the VMware PhDs to solve it.

Let me respond to a few of the suggestions from this quote.

“Customers Do Not Trust VMware for Tier-1 Apps”

The following chart presents data collected from 676 VMware customers in July and August of 2008.

Percentage of application instances virtualized by VMware customers.

This graph shows the large rates of virtualization of the most well-known enterprise applications.  By any definition of “Tier-1 Application”, at least one tier-1 application is mostly virtualized by this customer sample.  And the survey date bears repeating: summer, 2008.  Virtualization acceptance has greatly increased in the past 12 months.

Concerns about performance management aside, VMware customers are virtualizing their tier-1 apps today.  So let’s talk about the process of performance troubleshooting.

“A PhD From VMware Is Required to Fix Performance Problems”

I think that David must have inferred from my confident and detailed talk on a great number of performance-related technical topics that I am the cream of the crop of America’s educational system.  For the record, I went to a state school in Alabama and spent far more time drinking beer than going to class.  Nonetheless, I am sure what he meant to say was…

“A Highly-skilled Performance Expert Is Required to Fix Performance Problems”

VMware now boasts over 150,000 customers, and I only interact with a relative handful a year.  If I count the experts in our small performance community I can conclude that our performance experts touch a very small percentage of our customer base each year.  That means that the great majority of our customers are solving their performance problems without engaging us.

Customers are fixing their problems using a variety of tools that I continue to document:

• The vSphere client interface to vCenter is known to everyone and its counters operate with 20s granularity and are effective at fixing about 90% of most performance problems.
• esxtop, with its finer granularity and larger counter list, can be used to fix 95% of problems, most of which could have been fixed with vCenter statistics.
• vscsiStats is extraordinarily useful for a small percentage of problems, perhaps 10-20% of those I see.

We are currently working on collecting all of these views into the client and adding a framework, vProbes, that will enable unprecedented visibility into operating systems and applications.  But even as things stand today, we have provided documentation and tools that all of our customers can use to fix any problem.  There is always room for improvement, but no PhD is required.