Scott Drummonds on Virtualization

VMware Thin Disks on EMC Virtual Provisioning


Even before I left VMware for EMC I was being asked to comment on “thin on thin”: the use of VMware thin VMDKs on virtually/thin provisioned storage.  As a VMware employee I recommended VMware’s thin provisioning but referred to storage vendors for their own best practices.

Now, as a member of the storage vendor community, I will answer for EMC. I will do so with detailed text from an outstanding TechBook I recently discovered on EMC’s Powerlink. This paper, Using Symmetrix Storage in VMware vSphere Environments (Version 7), provides incredible detail on the relationship between VMware thin disks and Symmetrix virtual provisioning. Its guidance is clear and simple.

My summary observations are as follows, with more detail below.

  • The decision about thin/virtual provisioning is primarily a management one: trust your vSphere capacity management processes?  Use vSphere thin provisioning.  Trust your array capacity management processes?  Use virtual provisioning.
  • VMware’s lazy zeroed thick disks reserve the space in VMFS.  But because blocks are left uninitialized until the first write, these disks are thinly provisioned in the virtual pool.  Choose this configuration when you have mature storage capacity management processes but questionable capacity management processes in vSphere.
  • VMware’s thin disks do not reserve space in VMFS nor in the virtual pool.  This “thin-on-thin” configuration presents the most flexibility but requires mature capacity management in vSphere and the array.
  • Even if your storage capacity management processes are mature, there are some cases where you may want eliminate the chance of an out-of-capacity problem.  This might be true with a mission critical application. In this case you have two options:
    • Do not use virtual provisioning in the array, which may greatly increase your storage consumption.
    • Use VMware’s eager zeroed thick disks on EMC virtual provisioning, which reserve capacity in VMFS and the virtual pool.  With VAAI, Enginuity records but does not perform the zeroing.  This means the zeroing is instantaneous and the block is reserved.

Here is a long quote from the TechBook, starting from page 128:

In general, before vSphere 4.1, EMC recommended using “zeroedthick” instead of “thin” virtual disks when using Symmetrix Virtual Provisioning. The reason that “thin on thin” was not always recommended is that using thin provisioning on two separate layers (host and array) increases the risk of out-of-space conditions for the virtual machines. vSphere 4.1 (and even more so in vSphere 5) in conjunction with the latest release of Enginuity integrate these layers better than ever. Consequently, using thin on thin is now acceptable in far more situations, but it is important to remember that risks still remain in doing so. For this reason, EMC recommends “zeroedthick” (for better reliability as only the array must be monitored) or “eagerzeroedthick” (for absolute reliability as all space is completely
reserved) for mission critical virtual machines.

It is important to remember that using “thin” rather than “zeroedthick” virtual disks does not provide increased space savings on the physical disks. As previously discussed, since “zeroedthick” only writes data written by the guest OS, it does not consume more space on the Symmetrix thin pool than a similarly sized “thin” virtual disk. The primary difference between “zeroedthick” and “thin” virtual disks is not a question of space, but a question of the quantity and capacity of the Symmetrix thin devices presented to the ESX server. Due to the architecture of Virtual Provisioning, binding more thin devices to a thin pool or increasing the size of them does not take up more physical disk space than fewer or smaller thin devices would although it does require a small amount more Symmetrix cache. So whether 10 thin devices are presented to an ESX host, or 1 thin device, the storage usage on the Symmetrix is essentially the same. Therefore, packing more virtual machines into smaller or fewer 128 thin devices is not necessarily more space efficient. Prior to ESXi 5, a single-extent VMFS volume was limited to approximately 2 TB and this led customers to use “thin on thin” to keep the number of devices presented to a host minimal as a way to ease management. “Thin” virtual disks allowed for fewer presented devices and a higher virtual machine density. Now with the larger allowable single-extent VMFS volume size in ESXi 5 (up to 64 TB) the need to add more thin devices is reduced as larger, more appropriately sized thin devices can be presented and used and expanded as necessary. With ESXi 5, customers therefore do not need to make the virtual disks “thin” in order to fit a large number of them on a single volume as it is no longer constrained by the 2 TB minus 512 byte limit.

However, as previously mentioned, VMware and EMC have recently ameliorated the practicality of using thin virtual disks with Symmetrix Virtual Provisioning. This has been achieved through refining vCenter and array reporting while also widening the breadth of direct Symmetrix integration through features such as the vStorage API for Storage Awareness (VASA—vSphere 5 only). Additionally, the slight performance overhead that existed with thin VMDKs
which was caused by zeroing-on-demand and intermittent expansion of the virtual disk as new blocks were written to by the guest OS has been significantly diminished with the advent of Block Zero and Hardware-Assisted Locking (ATS). The introduction of Storage Dynamic Resource Scheduler in vSphere 5 (SDRS) further reduces the risk of running out of space on a VMFS volume as it can be configured to migrate virtual machines from a datastore when that datastore reaches a user-specified percent-full threshold. This all but eliminates the risk of running out of space on VMFS volumes, with the only assumption being that there is available capacity elsewhere to move the virtual machines to. These improvements can make “thin on thin” a much more viable option—especially in vSphere 5. Essentially, it depends on how risk-averse an organization is and the importance/priority of an application running in a virtual machine.

If virtual machine density is valued above the added protection provided by a thick virtual disk (or simply the possible risk of an out-of-space condition is acceptable), thin on thin may be used. If “thin on thin” is used, alerts on the vCenter level and the array level should be configured.

If you want the full copy of this document, go to Powerlink and search for the paper’s title, “Using EMC Symmetrix Storage in VMware vSphere Environments”. If you change the “Filter by Content Type” drop-down to “Technical / White Papers” this document will appear first in your search.

4 Responses

Thanks Scott for the response; Great post on this topic!

  • I just published an updated white paper that complements this information in the techbook with more detail specific to vSphere and Symmetrix VP:

  • […] The same article has a section entitled “Thin pool management” and it has great examples on how to configure the array and vCenter to monitor the space usage of a thin Lun. All in all, using thin provisioned vmdks on thin provisioned LUNs is not a bad thing as long as you running ESX 4.1 or above and you monitor the disk usage appropriately. Here are some blogs that talk about similar topics: Thin on Thin? Where should you do Thin Provisioning – vSphere 4.0 or Array-Level? and  VMware Thin Disks on EMC Virtual Provisioning. […]