vPivot

Scott Drummonds on Virtualization

A Performance Tip for ESX 3.0 and ESX 3.5

10 Comments »

Do you have any running instances of ESX 3.5 or older?  Are those instances running on processors that are no more than a couple of years old?  If so, I have a tip for you: update your hosts to ESX 4.0. Seriously, upgrade to vSphere already. It’s been out for a year!

All kidding aside, yesterday I wrote an article about a sneaky trick to leverage improved hardware assist performance on ESX 3.5 virtual machines that defaulted to binary translation. I learned very late in the day that the recommended guidance only works for AMD processors. The text below is from the original post but has since been updated to reflect my newly discovered information.

Begin Updated Article

Do you have any running instances of ESX 3.5 or older?  Are those instances running on AMD processors that are no more than a couple of years old?  If so, I have a tip for you: force hardware assist in those virtual machines.  In most situations application performance will improve by 10% or more.  Details follow.

ESX’s monitor presents virtual hardware to virtual machines’ guest operating systems.  VMware’s multi-mode monitor uses three technologies to do this: hardware assist, para-virtualization, and binary translation.  Hardware assist has gotten much faster over the years, as this figure demonstrates.

The latency of the VMEXIT instruction is shown on Intel VT systems. The longer this instruction takes to execute, the worse the virtual machine performs.

Johan De Gelas included his take on monitor mode performance when he reported “Virtualization Round Trip Latency” in a recent article on the new Xeon 5600.  His results reiterate the trend I have been sharing with my audiences for over a year now.

Because hardware assist was once so slow, older versions of ESX would utilize our faster-performing binary translation in many situations.  But virtualization assist in today’s processors–and here I am talking about Intel and AMD processors manufactured in the past two years–is generally faster than binary translation. This means your virtual machines running on ESX 3.5 on shiny new processors may not be reaching their full potential performance.

The fix is simple: force hardware assist for your ESX 3.0 and ESX 3.5 virtual machines running on newer AMD processors.  You can do this with the following lines in your virtual machines’ VMX files:

monitor.virtual_mmu = hardware

A reboot is then required. But this setting only works for AMD processors, where RVI (AMD’s hardware memory management unit) is available. With this setting both AMD-V and RVI are forced on. This setting is ignored on Intel processors, whose hardware MMU is not leveraged on ESX 3.5.

These changes are not needed with vSphere because its default monitor modes favor hardware assist more than ESX 3.5 and earlier. You can see vSphere’s default monitor modes in our wonderful monitor modes paper.

As one example of the magnitude of performance increase this small change can produce, look to the SQL Server performance paper we released last year.  Here is one graph I lifted from that document.

This figure shows AMD-V improving performance by 18% over binary translation on virtual machines running SQL Server 2008.  More gain is possible when the hardware memory management unit is utilized.

Because of ESX 3.5′s continued wide deployment, it may very well be running millions of virtual machines.  Many those virtual machines are running on newer AMD processors that can benefit from this change.  Go forth and reconfigure those virtual machines to claim the performance to which you are entitled!

10 Responses

Hi Scott,

Thanks for the info.

If you are running on pre-Nehalem based hardware what happens if you set monitor.virtual_mmu = hardware when they don’t have EPT?

How do we tell on ESX 3.5 which mode ESX is using for a given VM?

Cheers,
Sim

    • Sim,

      The attempt to set an Intel processor to use the hardware MMU will be ignored on ESX 3.5. Unfortunately, ESX 3.5 provides no way to confirm the running monitor mode. The .vmx setting is correct, unless an illegal configuration has been specified.

      We improved this in ESX 4.0, where you can identify the monitor mode in the vmware.log file. For more info on that see this document:
      http://www.vmware.com/resources/techresources/10036

      Scott

      • Thanks for the additional link Scott.

        According to that:
        “Both ESX 3.5 and ESX 4.0 recognize the monitor.virtual_mmu setting. Only ESX 4.0 recognizes monitor.virtual_exec. You can express all possible ESX 3.5 mode choices with the monitor.virtual_mmu option alone.”

        Does that mean that the monitor.virtual_exec setting isn’t valid for ESX 3.5 and that ALL the monitor mode settings are controlled via monitor.virtual_mmu setting for 3.5?

        Cheers,
        Sim

        • Wow….outstanding catch. That clearly contradicts the best practice I gave above. I need to go huddle with the guys I shared a coffee with earlier today. Will update the blog ASAP.

          Scott

          • No worries. That sort of catch wasn’t the result I had hoped for though. :)

            That will teach me to go looking for silver performance bullets.

  • [...] Drummonds – A Performance Tip for ESX 3.0 and ESX 3.5Because hardware assist was once so slow, older versions of ESX would utilize our faster-performing [...]

  • scott — how do we determine if the guest is utilizing hardware assist or not ?

    • For ESX 3.5 there is no way to check the monitor mode. For ESX 4.0 you can grep the log file for MONITOR and you will see statements at VM boot time showing monitor information.

  • [...] Drummonds – A Performance Tip for ESX 3.0 and ESX 3.5Because hardware assist was once so slow, older versions of ESX would utilize our faster-performing [...]

  • Good day! i am at work currently so i didn’t have the time to check out all of the article, but i do like the stuff i read and i’ll read a little more on the site when i get home.. Got a lot of things to finish at the job :) do you have a profile on facebook? ;) Regards, Resa

  • Switch to our mobile site