Understanding the Bottleneck

Let me show the value of over-commit and danger of swapping by way of an example. I will choose the following typical values to demonstrate my point:

• All virtual machines are on a single host which has 32 GB of RAM installed.
• Each virtual machine is sized to 8 GB of RAM.
• Each virtual machine has 25% active memory (%ACTV in esxtop and “Active” in vCenter).

A virtual machine’s active memory is dictated by the application and its usage.  But the VI admin has complete control over the number of virtual machines in the environment which means host active memory can be influenced by adding or removing virtual machines.  Because virtual machine active memory is always equal to or less than 100% the only way to drive the host active memory to 100% is to over-commit memory.  This is why hypervisors that do not support memory over-commit are simply not viable for data centers where memory optimization is a priority.

Identifying and Correcting the Bottleneck

The ongoing occurrence of swapping is identified by a non-zero swap rate in either esxtop or vCenter.  In addition to swap rate, esxtop provides a swap wait time in its CPU panel.  When swap rate exceeds hundreds of kilobytes per second or swap wait time exceeds a couple percentage points, it is time for corrective action.

There are three possible solutions to this problem:

1. Balance the virtual machines’ memory usage by moving virtual machines from hosts with higher amounts of memory usage to hosts with lower amount of memory usage.
2. Run fewer virtual machines.
3. Buy more memory.

Designing Your Infrastructure to Simplify Memory Management

Ultimately I owe you a full white paper on memory management to provide a sufficient answer. But I want to give you two ideas of the tools and techniques that I will be describing when in this future paper.  First, place host swap files on solid state disk (SSD) stores to improve their performance.  With the right SSD device it may be possible to eliminate swap penalties.  Second, even if SSDs are unavailable consider consolidating multiple swap files onto a single store.  This will make swap rate monitoring very easy but may compound the performance penalties of swapping.

Stay tuned and VMware will provide more documentation on memory management in 2010.

I believe we know the text that is misrepresented.  It comes from our performance best practices document and could be misunderstood by someone to whom the terms “working set” and “active memory” are unfamiliar.  Here is the sentence, quoted from the oldest available version of our best practices document:

Swapping is used to forcibly reclaim memory from a virtual machine when both page sharing and ballooning fail to reclaim sufficient memory from an overcommitted system. If the working set (active memory) of the virtual machine resides in physical memory, using the swapping mechanism and having inactive pages swapped out does not affect performance. However, if the working set is so large that active pages are continuously being swapped in and out (that is, the swap I/O rate is high), then performance may degrade significantly.

This excerpt describes the condition under which a host will swap.  That condition is best summarized as “the sum of the working sets of all virtual machines exceeds the amount of memory on the host”.  This definition presupposes that the reader understands the definition of a working set, which I think some readers may not.  For this discussion I will simplify the definition of working set as “recently active memory” and refer readers to their helpful search engine for a more complete description.

When a system’s working set exceeds available memory, the system will swap.  This is not unique to virtual, consolidated workloads.  As long as operating systems have implemented virtual memory, there existed a possibility that a working set could exceed available physical memory.  The only thing that has changed in a virtual environment is that the working set is calculated by summing the working sets from multiple virtual machines as opposed to a single application or operating system instance.

But the key here–and the reason why memory over-commitment remains so powerful–is that allocated memory (the virtual machine’s size) exceeds active memory (the working set) nearly 100% of the time.  Memory management in consolidated environments is about pushing a host’s active memory to as close to 100% as possible.  This is something that was not possible in physical environments, and cannot be done in virtual environments without over-commitment.

To sum this up, not only does VMware recommend some over-commitment, but we know that it is impossible to fully use your available memory without the flexibility provided by over-commitment.

But a recent article by my engineering colleague Chethan Kumar shows an avenue that allows VI admins to aggressively over-commit memory and avoid the catastrophic performance penalty of swapping: use solid state disks to host ESX swap files.

The fundamental problem with host swapping comes from the high latency of traditional disks compared to memory.  Data can be retrieved from memory in nanoseconds but takes milliseconds to fetch from a hard drive.  That means a single 4K memory page takes 100,000 times longer to retrieve if the operating system swapped it out.

The value that solid state disks offer to this problem is exceptional latency, as compared to traditional drives.  The SSD that Chethan used showed microsecond latencies, about 1,000 times lower than physical disks.  This means that  time spent waiting for swap activity* has been decreased to 0.1% of the time spent swapping to physical disks.

The importance of fast swap files is that it enables administrators to more aggressively over-commit memory.  Today our admins rightfully fear the VMs’ aggregate active memory exceeding the available physical memory, which results in swapping.  Today SSD technology in shared storage such as EMC’s new CLARiiONs allows our admins to cleverly place swap files and drive up memory utilization to previously unheard of levels.  This may enable standard memory overcommitment of 200% or more, with extreme over-commit being much higher than this.

In future versions of ESX we want to automate the usage of SSDs to maximize the use of available memory.  But that’s a roadmap discussion that I will leave for another day.

(*) This swap wait time has conveniently been added to ESX 4’s version of esxtop under the counter %SWPWT.  See Interpreting esxtop Statistics for more information.

ESX attempts to mitigate the impact of interacting with guest operating system memory management by randomly selecting the swapped guest physical pages.

This customer has read some of our other documentation and knows that the vSphere client and esxtop report active and touched memory, each representing a kind of working set.  “So,” the customer asks me, “if ESX is keeping working sets, why would choose swap pages randomly instead of selecting inactive pages outside of the working set?”  If ESX could choose only inactive pages then the penalty due to host swapping would drop greatly.

As it turns out, ESX does not track working sets.  For ESX to know which pages are actively being read it would have to trap every memory access, which would greatly hurt performance.  Instead, we track a small sample of the host’s memory–exactly 100 pages–to extrapolate the size of active memory.  Because sample-based calculations are accurate with the square of the sample size, a 100 page sample delivers a very high accuracy of active memory.  But, because it is only a sample, the activity on the great majority of pages is a mystery.

This means that ESX has absolutely no information on the read and write behavior of most memory on the system.  Guest operating systems fare no better in tracking working sets.  But because they can categorize pages based on usage–application heap, buffers for IO, kernel memory, etc.–guest OSes can make better decisions about which pages should be swapped and which should not.  For this reason the balloon driver can induce non-harmful guest paging which is superior to host swapping.