Scott Drummonds on Virtualization

Performance Troubleshooting Guide Update


My old friends in VMware’s performance team updated the hugely popular performance troubleshooting guide for vSphere with a vSphere 4.1 version. The previous version of this document has been viewed nearly 30,000 times to date, making it the most popular document ever produced by performance engineering. The vSphere 4.1 version has already been seen over 9,000 times which suggests its early popularity with a large audience.

This paper documents the flow used by VMware’s outbound performance engineering team when they are engaged to investigate a performance problem. The process is shown in flow charts that will lead the reader to a bottleneck. The problems associated with resource bottlenecks are documented in a “cause” and “solution” section towards the end of this publication.

Before I left VMware one of my friends in the VMware support organization told me that the first version of this document was instrumental in educating VMware support on the diagnosis of performance issues. Hal’s document became mandatory reading for anyone that faced a performance problem. With the help of Chethan, his teammate and one of my VMworld co-presenters, this update is surely to add even more value to this document.

5 Responses

Hello Scott,
I have a performance question about the PVSCSI driver in Windows OS (Server 2003/2008R2). In Windows there a two SCSI driver models: scsiport and the newer storport drivers. Which of them does PVSCSI follow?
We observed performance problems on servers (HW and VM) using scsiport drivers which have more than on one disk controllers installed. When one (logical) disk drive is heavily utilized all other disks (also using scsiport drivers) are also affected (and therefore slow). This is because ALL scsiport disks are using ONE disk queue (as I remember). For storport disks there is a queue for each disk.
So, is this relevant for the PVSCSI driver? If yes, I suggest to include a chapter in the performance guide.

    • Actually, you’ve stumped me! Where I would normally walk across the hall and research this with performance engineering, I am now across the Pacific from them. I will research and update here.

  • Hermann,
    Thanks for bringing this up. pvSCSI uses storport driver. Have you faced the problems you mentioned with pvSCSI driver?


  • Scott – Recently was made aware of the perfmon counters available at the guest level in Windows OS VM’s. We have many (over 100) vm’s. We seem to be up against an issue you had blogged about – missing performance counters. We’re running 4.1, and have tried the vtools uninstall / reinstall. The ‘fix’ you had is not longer available from the FTP site… was curious to see if you had any more recent updates on that issue, any official workaround, etc. Haven’t found much on the vmware KB thus far.

    Any information you have would be great. Thanks!!

    • Hi, Eric,

      Unfortunately I am out of the loop on this, now that I am out of VMware. When I last left the product manager was engaging engineering about this issue trying to get a fix. Engineering told him that there were no open tickets in the support system that justified investigating the problem. Their work has to be based on support tickets, not informal comments through channels like this.

      It will be slightly more painful for you. But I recommend that you open a support ticket.