vPivot

Scott Drummonds on Virtualization

Inaccuracy of In-guest Performance Counters

27 Comments »

Every couple of months I receive a request for an explanation as to why performance counters in a virtual machine cannot be trusted.  While it is unfairly cynical to say that in-guest counters are never right, accurate capacity management and troubleshooting should rely on the counters provided by vSphere in either vCenter or esxtop.  The explanation is too short to merit a white paper but I hope a blog article will serve as the authoritative comment on the subject.

Usually this issue arises inside a new VMware customer or an established customer that has added new staff to the virtualization team. In both cases the administrators are familiar with existing tools and require good reason to retool their thinking and environment around a new measurement system.

I was discussing the response to these concerns with my friend and colleague Kaushik Banerjee, the head of VMware’s outbound engineering group.  Kaushik and I spend a lot of time thinking about communicating technical details to our customers and in this case we chose different approaches to answer the question.  Both responses are complementary, so choose the one that suits your needs.

Kaushik’s Approach: The Killer Examples

Kaushik suggested that if we show cases where the guest OS’s counters were obviously wrong that a naturally suspicious VI admin would never trust the guest counters again. To that end, I offer the following screen shots to make our point.

Guest Utilization Higher Than Host

Perfmon's counters show utilization higher in the guest than the host reports.

This screen shot shows two counters available in Perfmon inside a Windows guest with the vmStatsProvider installed (available by default since vSphere). The darker, red line is the CPU utilization as reported by the guest. The lighter, greenish (?) line is CPU utilization of the virtual machine, from the host’s perspective. This is the real CPU utilization passed up to the host by vmStatsProvider.  Notice how the host is always reporting higher utilization than the guest.  This is due to one of the reasons why guest counters cannot be trusted: they are unaware of hypervisor overheads.

This second screen shot shows a different case where the host utilization is lower than that reported by the guest.  Again, the dark red line represents the guest OS’s report of CPU utilization and the lighter line shows the real CPU utilization as reported by ESX.

Host Utilization Higher Than Guest

Perfmon's counters report a higher CPU utilization than ESX's.

The reason the host shows lower utilization than the guest is because the guest is unaware that it is only getting a fraction of the host’s CPU, time-sliced by ESX’s scheduler.  In this case the virtual machine was contending for CPU with other active virtual machines but this just the same principle would apply had a CPU limit been set.

Scott’s Approach: A Detailed Explanation

My approach to convincing VI admins to avoid guest tools is based on bottomless thirst for information that is common to technophiles. If I can provide an explanation for the underlying system of resource scheduling and manipulation, then our admins will be able to deal with the guest counter issue and maybe solve other issues with their newfound knowledge.

There are four reasons why guest counters cannot be trusted:

  1. The guest is unaware of virtualization overheads.  As screen shot one showed above, the hypervisor will increase the CPU load as it virtualizes the hardware for the guest operating system.  That additional CPU work is not seeing by guest tools.
  2. The guest is unaware that it is only seeing the portion of CPU that ESX’s scheduler is allowing it to see.  Because of contention or resource restrictions, virtual machines only get a slice of the CPU’s time.  When a guest thinks it is getting 100% of the CPU it may not know that the processor is being shared by eight other virtual machines. See the second screen shot above.
  3. Time skew in virtual machines can change the sample window for time-based counters.  This means that the guest may have measured 10 milliseconds of time passage during a read command when 12 milliseconds have elapsed.  This is more common on older versions of ESX and when the host CPU is saturated.  More on this below.
  4. The virtual machines are unaware that they are being de-scheduled when idle, which means that they appear to be working more of the time than they are.  Consider a case where a virtual machine is idle 90% of the time.  If ESX does not schedule the VM during its idle time then the guest will think that its processor queues are full 100% of the time that it is being executed.

The time drift explanation (item three) was historically the most problematic for VMware.  On older versions of our products time drift was common.  As ESX has matured we have reduced the amount of drift which has improved the accuracy of guest counters.  But the timer hardware is still being virtualized in software running on the host CPU.  This means that if host processor is fully utilized, the timer may not be scheduled on time, resulting in a delay in some ticks and a resultant skew in guest time.

References

Performance Best Practices and Benchmarking Guidelines.  This white paper was the last version that we printed that included benchmarking best practices, which contains some discussion on the need to measure performance from outside the host-under-test.

Timekeeping in Virtual Machines.  This document–not updated since VI3 but still accurate in its theory–will give the background on VMware-based time keeping and provide an explanation as to how skew can occur.

VMware vSphere™ 4: The CPU Scheduler in VMware® ESX™ 4.  This white paper provides great detail how the scheduler works which will fully explains the notion of time slicing.

27 Responses

Thank you, Scott! This is a clear concise explanation for those perf/capacity shops new to virtualization.

  • It’s bette than a whitepaper Scott, thanks a lot !

  • So, not only performance monitor, but all standard performance monitoring tools, such as IBM Tivoli, cannot be trusted ,correct ?

    • Any data that is originating in the guest and not in ESX or vCenter is suspect. Tools that are collecting guest stats should be eyed suspiciously.

      However, many third-party monitoring tools use VMware’s SDK to collect and present stats from the vSphere environment. In presenting accurate host stats side-by-side with slightly inaccurate guest stats, admins can get an excellent picture of how resources are being used.

  • Hi Scott,

    If a VM’s OS performance counters are showing its usings say 80% cpu, but in reality its only getting 40% from the host, doesn’t that just tell us that the guest OS is actually starved of CPU? Especially the figures are skewed due to CPU constraints on the host and in that case are valid?

  • As always, great info – thanks! Are there any plans to include additional counters, besides the memory and CPU ones, in the future?

  • Monitoring applications (like IBM Tivoli Monitoring for Virtual Servers, CA VPM, etc.) use VMware’s WebServices API (a.k.a. vSphere API) so their metrics are correct. Note that placing the Linux-version of a Tivoli agent (for example) in the COS (or guest) will not use the WS API and will be subject to the limitations described above.

  • I agree that the guest OS cannot see the VMkernel overhead without software assist from VMware. But I do trust the guest perfmon if looked at in the proper context. If a guest perfmon is showing 80% CPU utilization, that means 80% of whatever percetnage of physical CPU time VMkernel has provided to that VM. No, it does not show how busy the physical processor is (one needs to use the counters provided by VMware). But it does show how busy the vCPU is (and if it stays at 98% at all times, for example then that means that VM needs a second vCPU)!

  • thanks for a post.
    you should change the order of the screenshots.

  • [...] Microsoft NVSPBIND.EXECitrix release virtual appliances [...]

  • I’m curious about how you feel this applies to memory utilization.

    I agree the Anonymous user that posted about trusting CPU utilization in the right context.

    Ballooning, host swapping and TPS will skew the memory readings in the guest, but these do not always apply. In an environment where there is little to now ballooning or swapping, one only needs to consider TPS.

    In fact, one might argue that the guest memory counters can be more accurate because ESX only knows what memory has been consumed/granted or has been accessed recently (active)… In the case where an application does a lot of caching, it may be important to know that this memory is actually allocated.

    • There are not good active memory counters in any operating systems that I am aware of. For that I always prefer vSphere’s tools. But it is true that vSphere knows nothing about how the memory is being used: process memory, IO buffers, kernel memory, etc. So, some guest counters are useful in troubleshooting and capacity management.

      Scott

  • Great job! I have been in the virtualization field for years and get tired of explaining this. You’ve done a spectacular job at bringing the details to the table in a way that the average user can understand. Applause.

  • Great post, Scott. It really shows how collecting data from multiple spots (Guest OS for Memory utilization data, and it’s view of CPU utilization), vSphere for VM memory and CPU utilization and host overhead is key. If you only look at the system from one perspective, your view is easily skewed. You need a multi-dimensional approach to ensure you are getting the right information.

  • Great explanation and I’d prefer to use these stats for our monitoring system, however each time the VM counters are used the vmStatsProvider generates 2 events to the application log, EventIDs: 256 and 258, which just say the wmi stats provider has been initialized. So, as we poll the counters every 5 minutes it floods the event log with thousands of events a day. A way to suppress these rather useless informational events would make life easier.

  • I have a question…based on your explanations, what does it matter? If the OS says it has no resources left, and throttles back the applications, does it matter that the OS isn’t right?

    • Casey,

      If the only question you are trying to answer is, “Is my application resource-contrained?” then it does not matter if the counters are accurate or not. But there are other questions that require more accurate counters:

      How much CPU is available on the host? (Estimating this using tools that collate guest counters will produce inaccurate results.)
      How much CPU is application “X” using? (As the article states, guest tools are inaccurate.)

      So, there are some questions that require accurate answers. But, as you point out, some questions are simple enough that guest tools are sufficient.

  • Does this mean a guest will report the usage of it’s cpu allocation rather than anything that relates to the physical usage?

    Lets say we have a hypervisor with 2 x 1000Mhz cpu’s. We build to guests vm’s and allocate them each 2000Mhz. So while both vm’s are 50% idle would the hypervisor be 100% utilised?

  • Great post!

    Just to verify…

    The VM utilization, as reported by vSphere, includes hypervisor overhead, correct?

    And the overall utilization of a host, again as reported by vSphere, includes hypervisor overhead?

    • Correct. In fact, one of the ways you can measure overhead is to run an application on physical and compare Perfmon’s CPU measurements to esxtop’s when the application in run on virtual. The difference is the overhead.

      • Thanks, Scott. Does the same hold for the usagemhz counter? I would assume so since VM utilization is as usagemhz / (VCPUS * mhz per core).

  • Switch to our mobile site