Scott Drummonds on Virtualization

Performance Troubleshooting: No PhD Required!

Comments Off on Performance Troubleshooting: No PhD Required!

A couple of weeks ago at VMworld in San Francisco I squeezed a few press meetings in between the 19 sessions of the performance lab I led. In one of those meetings I talked with David Vellante and two of his colleagues to discuss vSphere performance and performance monitoring.  David and company asked some hard questions about our performance work but my knowledge of this area runs deep, so the conversation was fruitful and interesting.

A few days after the conference a coworker of mine shared the following quote with me, courtesy of an article by David on Internet Evolution:

The fact is, most data center managers wouldn’t trust VMware to manage their Tier 1 applications because if something goes wrong performance-wise, you still need to roll in the VMware PhDs to solve it.

Let me respond to a few of the suggestions from this quote.

“Customers Do Not Trust VMware for Tier-1 Apps”

The following chart presents data collected from 676 VMware customers in July and August of 2008.

Percentage of application instances virtualized by VMware customers.

Percentage of application instances virtualized by VMware customers.

This graph shows the large rates of virtualization of the most well-known enterprise applications.  By any definition of “Tier-1 Application”, at least one tier-1 application is mostly virtualized by this customer sample.  And the survey date bears repeating: summer, 2008.  Virtualization acceptance has greatly increased in the past 12 months.

Concerns about performance management aside, VMware customers are virtualizing their tier-1 apps today.  So let’s talk about the process of performance troubleshooting.

“A PhD From VMware Is Required to Fix Performance Problems”

I think that David must have inferred from my confident and detailed talk on a great number of performance-related technical topics that I am the cream of the crop of America’s educational system.  For the record, I went to a state school in Alabama and spent far more time drinking beer than going to class.  Nonetheless, I am sure what he meant to say was…

“A Highly-skilled Performance Expert Is Required to Fix Performance Problems”

VMware now boasts over 150,000 customers, and I only interact with a relative handful a year.  If I count the experts in our small performance community I can conclude that our performance experts touch a very small percentage of our customer base each year.  That means that the great majority of our customers are solving their performance problems without engaging us.

Customers are fixing their problems using a variety of tools that I continue to document:

  • The vSphere client interface to vCenter is known to everyone and its counters operate with 20s granularity and are effective at fixing about 90% of most performance problems.
  • esxtop, with its finer granularity and larger counter list, can be used to fix 95% of problems, most of which could have been fixed with vCenter statistics.
  • vscsiStats is extraordinarily useful for a small percentage of problems, perhaps 10-20% of those I see.

We are currently working on collecting all of these views into the client and adding a framework, vProbes, that will enable unprecedented visibility into operating systems and applications.  But even as things stand today, we have provided documentation and tools that all of our customers can use to fix any problem.  There is always room for improvement, but no PhD is required.

Comments are closed.