I have struggled for years to give VMware’s customers a framework for diagnosing performance problems. People want a simple system to troubleshoot the unknown sources of poorly performing applications. The best attempt at documenting such a flow is Hal Rosenberg’s document on vSphere performance troubleshooting. Elegant as it may be, Hal’s document remains complex for the novice VI administrator. And it is because that document is so complex that performance people maintain their job security.
But in an effort to further obviate my own job, I will try and generalize the troubleshooting flow to add more clarity to the process.
Performance Troubleshooting Made Simple
How Many Virtual CPUs Per VM?
Virtual machine sizing is a tricky issue for many VMware administrators. It is important to find the right number of virtual CPUs to maximize application performance and minimize wasted CPU cycles. The optimal number of vCPUs can never be easily identified. But I can offer a few suggestions to help get this number right.
Processor Utilization Calculations
A little Friday esxtop trivia for the performance massive: did you ever notice your Hyper-Threaded systems have three rows showing CPU utilization in the CPU panel header? They are labeled “PCPU USED(%)”, “PCPU UTIL(%)”, and “CORE UTIL(%)”. Here is a screen shot to jog your memory:
esxplot 1.0 Released
I am pleased to announce that Geoff White has completed the first official release of esxplot, version 1.0. In an earlier blog article we discussed a beta version of this product and the response from VMware’s customers has been fantastic. Those of you that have played with esxplot know its value in assisting with esxtop-based analysis. I urge everyone to download the new version and give it a try.
Optimizing Memory Utilization
My recent series of blog articles have discussed ESX memory management the the performance specter of host swapping. My last article attempts to correct the misconception that VMware recommends against over-commit memory. In that article I suggested that memory over-commit is requirement in optimizing memory utilization. Today I want to provide a specific example to show why this is true. I am have also included tips for identifying host swapping in your environments.
Read the rest of this entry »
Solid State Disks and Host Swapping
Recently I have been thinking, talking, and writing about ESX host memory swapping a lot. ESX swaps memory under the same conditions that traditional operating systems do; the application(s) is using more memory than available on the physical hardware. Host swapping is an unavoidable consequence of this condition, whether virtualization is present or not.
esxtop Analysis With esxplot
esxtop remains a popular performance troubleshooting tool because of its fine granularity, expansive counter list, and support for interactive and off-line analysis. The biggest problem with esxtop is the huge CSV files generated in batch mode. The output is so large that Excel is unable to open the file and Perfmon can take hours to do so. But now we have a better way to manage esxtop batch files.
Micro-bursting and Storage Performance
I have been reading Chad Sakac’s article on IO queues and micro-bursting for months now. Chad is wicked technical for a manager type and after reading this post a dozen times I think I finally have it internalized. Let me put my own spin on this tome, embedded in which are several jewels of wisdom.
Performance Troubleshooting: No PhD Required!
A couple of weeks ago at VMworld in San Francisco I squeezed a few press meetings in between the 19 sessions of the performance lab I led. In one of those meetings I talked with David Vellante and two of his colleagues to discuss vSphere performance and performance monitoring. David and company asked some hard questions about our performance work but my knowledge of this area runs deep, so the conversation was fruitful and interesting.
A few days after the conference a coworker of mine shared the following quote with me, courtesy of an article by David on Internet Evolution:
The fact is, most data center managers wouldn’t trust VMware to manage their Tier 1 applications because if something goes wrong performance-wise, you still need to roll in the VMware PhDs to solve it.
Let me respond to a few of the suggestions from this quote.
Read the rest of this entry »
vSphere Is Not the Performance Problem, Your Storage Is
[This is an update to one of my favorite articles, which details my on-site investigation of SQL Server performance problems.]
Back in July I had the privilege of riding along with VMware’s Professional Services Organization as they piloted a possible performance offering. We are considering two possible services: one for performance troubleshooting and another for infrastructure optimization. During this trip we piloted the troubleshooting service, focusing on the customer’s disappointing experience with SQL Server’s performance on vSphere.