Scott Drummonds on Virtualization

VMworld 2010 Performance Lab


As you can imagine, we VMware employees are starting to ramp up for VMworld 2010 and its younger brother in Copenhagen, VMworld Europe 2010.  Last year I ran the performance lab and this year I plan on making it even more awesome than last year.  Since this blog enjoys a small following of the performance faithful and VMworld attendees, I want to put the question to you: what do you want to see in the upcoming performance troubleshooting lab?

Last year was my first year captaining the performance troubleshooting lab.  I designed the content around the performance problems I see most frequently in customer deployments. On the philosophy that every attendee should get their money’s worth in the two hour slot, I packed in more experiments than anyone could finish in the allotted time.  That worked great for experienced VI admins, but some of the more novice administrators walked out with their heads spinning.

I do want to continue challenging even the strongest admins that come to the lab, but I must provide enough basics to make the lab enjoyable for everyone.  With that requirement in your mind, maybe you can offer ideas as to how the lab should change and what content you would like to see.  Here are a few primer questions:

  • What comments do you have from last year’s lab?  Did I succeed in my struggle to make the lab interesting to VCPs and beginners alike?  If not, which side of that spectrum did I favor?
  • What content do you want to see added?  A few ideas:
    • Swap management: managing swap through swap file placement, use of SSDs, and memory compression*
    • Storage IO Control*: using shares and limits to maintain performance of critical VMs in the presence of storage bottlenecks
    • NFS performance diagnosis*
    • Managing storage alignment
    • Guest defgragmentation impact
    • Anything else you are itching for?
  • What content should we should repeat with updated examples?
  • What do you want us to avoid repeating?  Have you learned all you need to know about vscsiStats, for instance?
  • What would you think if we made this lab self-paced but promised performance experts in the room and on-call for the entire day?

(*) Coming soon to a hypervisor near you.

Put your comments here and they may influence the content of this lab.

8 Responses


Perhaps some thoughts on automated detection and remediation of issues? Recommendations on what *could* be automatically handled, and what should never be left to some decision making script for remediation?

  • Hi,
    Storage IO Control sound really interesting.

    And.. btw… waiting to use all the starred feature on an hypervisor near me 😉


  • esxtop and vscsistats are always good. Which counters are the main ones to keep an eye on and the threshold when to start worrying.

    Other 3rd party tools that can be used to diagnose performance issues in the labs.

  • Scott,
    Having just worked the Advanced Topics Lab at PXE, we asked the attendees what they wanted to see in a Perf & TS lab, and the feedback was that they were very interested in TS and perf analysis focusing on Tier 1 Apps , deeper understanding of r/ESXTOP usage for same analysis (we promoted the upcoming Manage for Performance Course!), a break/fix portion of the lab, and vscsistats in ESXi (per your post) and NFS TS was a popular request. I thought the basic TS flow from the MfP class would be good to include as well…

  • Hi,

    I’d love to see a discussion/demonstration on tuning guest and host networking for low latency messaging workloads (vs., e.g., bulk throughput) — sort of a “how low can you go?” networking latency limbo…


  • Hey Scott,
    I’d have to agree with Tom, though I wasn’t in the Advanced Topics lab at PEX. Also NFS perf and troubleshooting is HUGE for me. NFS is my favorite protocol for use in my vSphere designs if possible. So the use of vscsiStats is also big on my list.

  • I think many people would love being presented with several performance problems that they had to diagnose from esxtop and/or VC, and what are the key metrics to look for. These problems should include
    – CPU starvation due to contention
    – CPU starvation due to slow disk performance (VM A hogs the CPU waiting for disk IO while VM B accumulates Ready time, even though B has little disk demand)
    – Poor performance while swapping in, but fine performance while much data sits in .vswp files

    I have doubts about the benefits of managing swap file location for performance, since usually you either shouldn’t be swapping or maybe the swapping happens only during a bootstorm.

  • I could not get into any of the labs on VMworld 2009 because the only day I could get into them was the Developers Day on Monday, others were fully booked long time before.

    I’d say NFS performance diagnosis is interesting as well as Storage IO Control.