In the past few weeks I had the pleasure of getting deep into a customer’s performance problem.  Ultimately we identified some interesting issues in the environment that we traced back to an overloaded array.  Like most performance problems, the complaints started at the application layer and then shifted to vSphere.  Like many configurations, it was difficult to pinpoint why the storage was slow.  But EMC account teams pride themselves in customer responsiveness. We assembled a small team to help out. I was amazed and grateful that experts from our midtier specialists in Australia, Malaysia, and India all pitched in on the analysis!

If you are a VMware administrator you may choose to leave the nuts and bolts of storage management to your storage teams.  While this article talks about those nuts and bolts, I ask you to read on.  A little knowledge about how your array works will make you an awesome VMware administrator.  It will help you work with your storage administrators to get the most out of your array.  When your array is at its best, so are your virtual machines.

The analysis you below is the product of tools EMC can run against your EMC storage in a very short time.  The data collection took 24 hours in this case.  But the figures I will show were auto-assembled in minutes.  This is one of the many cool things an EMC technical consultant or one of our partners can do for you.

This account is all true, save the pseudonym I am using for the customer: SingB.  All customers are great customers, but I think we have a particularly good relationship with this one.  I have been working with this customer’s IT department since my first week in Singapore.  They are forthcoming with their needs and problems.  And I am always happy to spend a few minutes with them.

The analysis started with the creation of a NAR file.  The process to create a NAR file is well documented throughout the internet.  We collected 24 hours worth of data from SingB’s CX4-480.  Confirming what the VMware and storage teams already knew, the first sign of trouble from the NAR file analysis is a 24-hour summary of array latency.

You can immediately see that the 95th percentile latency is just below 50 ms.  A backup job they are running after 21:00 is driving the latency quite high.  But as that slowdown is happening at off-peak hours, they can live with it.  But there are three other times in the day–right in the middle of the work day–that latency touches or barely passes the 50ms threshold.  During those periods end users are screaming.

The most visually exciting figure auto-generated from the NAR file is the heat map.

For those of you VMware admins that find yourself in the same position I was in five years ago, the parts of the storage array may be foreign to you.  So, here’s a five-second summary of what happens in the array: commands enter the array through the SP and are sometimes serviced by the DRAM cache.  When the DRAM cache misses a sequence of reads and possibly writes are initiated against the backend disks using the bus.  All of the components involved in this flow–SP, DRAM, connecting bus, and disks–are depicted in the above heatmap. There you go.  Storage is simple, right?

Several observations from the heat map:

• The SP utilization is generally high, moving between the green 50% and the red 100%.  As I said to our friends at SingB this week, they are getting their money’s worth out of this array.
• The DRAM cache is exceptionally busy.  That is generally a good thing.  But high cache usage in write-intensive environments often causes forced flushes.  A forced flush is when the cache fills up from writes and IOs are temporarily halted while more space is made.
• Bus 1, attached to SP A, is too often at 100% utilization.  This will limit the SP’s effectiveness.
• The disks, labeled by the pool they have been added to, are not exceptionally busy.  But on the whole the pool called “LOC” shows the highest utilization.

This customer originally deployed this array for a development environment.  They followed what I would call “generic best practices”.  They wanted to suit the most varied set of workloads with a reasonable cost.  Some of the decisions they made included:

• RAID 5 everywhere.  I would say the “average” workload looks a lot like an OLTP workload but with slightly larger blocks.  Mostly read with a block size just above of 8 KB or 16 KB.  RAID 5 is usually good for this.
• Pools for the majority of the applications but a few RAID groups for applications that are being manually managed.
• FAST Cache for the VMware environment, which we would expect to meet the “average” workload I describe above.  No FAST Cache for their database environment, which has frequently table scans and backups.  Those activities are mainly sequential and do not realize as much benefit from a large cache.

Later in our report I found something interesting.  You performance sleuths out there should take a look at this summary and see if you can identify where our assumptions above have been contradicted.

The thing that jumped off the screen at me is the write-heavy ratio for pool “vmw-1″.  It is 83% write!

If you are a storage administrator, you may have already realized why the SP utilization is so high.  If you are not an admin, you will first need to understand how RAID 5 works to know what is driving up SP utilization.

RAID 5 protects data using a parity system.  For each number of data blocks in a stripe there is one more block with the data’s parity information.  A read from any block in the stripe requires reading from all the blocks so the parity can be recalculated and checked.  A write to any block in the strip requires that all the data be read first, the parity checked and then recalculated before the stripe is written back.  It is because of this fact that writes are much more expensive with parity protection like RAID 5 than mirroring, like RAID 1.

In fact, our helpful analysis tool gave us a precise calculation of the effect of RAID choices with this workload. The following table estimates the backend IOPS (those internal to the array) as a result of choosing RAID 5 versus RAID 10 protection.

As you can see, at this particular read/write ratio the RAID 5 choice is generating roughly twice as many IOs at the backend as RAID 10 would, when configured to the same capacity.  But RAID 10, using both mirroring and striping, produces less usable space than RAID 5.  Balancing performance efficiency and capacity efficiency is one of the many difficult decisions storage administrators need make.

(As a side note, the above table also shows the tremendous value of SSDs in throughput-intensive environments.  But the SSD versus HD decision does not help in this particular situation.  The array is starved for SP cycles.  Disks that respond faster do not change the fundamental number of front-end and back-end IOs that the SP much process.)

As I mentioned earlier, we were working on this analysis with our friends at VMware.  VMware setup vCenter Operations and produced some phenomenal summaries of what vSphere was generating against the array.  With that analysis we have provided a simple plan of action for the customer:

1. Storage vMotion a couple offending write-heavy workloads to another array to provide immediate relief.
2. Over the next few weeks, perform some light storage redesign to exchange one RAID 5 pool for a RAID 10 pool.  Place the write-heavy VMs there and the subsequent SP load caused by those applications will go down by more than 50%.
3. Over the coming months, upgrade to vSphere 5 (SingB is currently on an older version of vSphere), consider buying vCenter Operations, and spend some more time optimizing the storage layout to reduce SP utilization and eliminate bus contention.

So, what can you take away from this lesson?

• If you are a VMware administrator, your storage administrators’ best friend.  Educate yourself a little on the different ways arrays actually store data.  Know how your VMs’ profiles will benefit from or be penalized by being mapped to different volumes.  Consider using vSphere’s vStorage APIs for Array Awareness to automate the mapping. A recent EMC paper on VASA on Symmetrix contains a great description of what VASA does and how it can help.
• If you are a a storage admin, make sure your VMware admins are using their storage vCenter plugin, such as EMC’s free VSI.  That is the beginning of their education in storage and the little extra help they need to see that all VMFS volumes are not the same.

The EMC Channel team here in Singapore is bringing the VSPEX word to all of our partners throughout Asia Pacific and Japan. Our Cisco channel manager asked me to create a video she could use with Cisco to tell them more about the project. She told me to keep it brief–under 30 seconds–and have some fun with it.

First, here is a quote from the EMC paper on Oracle best practices:

For Databases and some other workloads, [TPS] can degrade performance if that [sic] memory actually changes frequently, as is true with Oracle memory regions, such as buffer caches.

This comment, somewhat vague and not fully supported, indirectly suggests that TPS should be disabled in memory-intensive applications.  I had never heard such claims before so I mailed my old friends at VMware’s performance engineering team.

VMware’s outbound team told me they worked with the EMC authors and raised this issue in the original draft.  As I has suspected, VMware has never seen and could not understand a TPS performance penalty in configurations like this.*  And the feature’s design suggests that this slowdown is not be possible.  VMware maintains its position that TPS should not be disabled  But the excellent and thorough folks at EMC, also subjecting their configurations to some rigor, stand by their claim.  With two expert teams in disagreement, how could we broker a compromise?

Barring a large recommitment of time by team EMC to re-run the experiments or run more, new tests at VMware’s request, the compromise was to tone down the wording in the paper.  The paper does not say TPS should be disabled, as you can see above.  It is true that a best practice of disabling TPS might be deduced from this claim.  But EMC has not written it and VMware would not support it if it were committed to print.

When I was still in the VMware performance team I ran into cases like this with several vendors and partners.  I can tell you that 95% of the time the vendors or partners made a mistake in methodology.  But in one notable case a partner identified an issue unknown to VMware.  It is because of this that VMware defends its position on performance but encourages experimentation and disagreement.  Groups like EMC’s Enterprise Solution Group and in this case EMC IT do a tremendous amount of research on vSphere and the world is a better place for it.  We all want this to continue.

But in this case time and resources did not permit execution of more experiments drive consensus to unanimity.  So the written compromise respected the work and position of both parties.  EMC published the paper and recently Michael Webster published an awesome summary of a presentation on EMC’s IT journey that discusses VMware’s position on this TPS research.

As a final note, remember this.  Siblings bicker.  But family is family.

Footnote

(*) There are two cases where TPS could impact performance.  But neither should have been present in this case:

• In a server running flat-out at 100% utilization, the nominal CPU usage of TPS (less than 1% of a single core) might be measurable.  We never have seen this cost appear in end-user throughput or response time because it is nearly impossible to drive CPUs to 100%.  With any IO on the server at all there is always some wait time and a total CPU load less than 100%.  But, in theory, it is possible.
• Once memory is overcommitted, large memory pages are broken into small for page sharing.  In this case, the small pages backing the guest large pages will underperform the original configuration.  To stop this behavior and guarantee the integrity of large pages, you need only set memory reservations.  Do not disable TPS.

First, here is a grab from the slide I shared to the Twitterverse.

In summary of this slide, I tweeted, “The average cost of downtime in a disaster is $223k/hr. Disasters: expensive. #VMware #SRM: cheap. http://pic.twitter.com/AAbIrUMS”. David saw my tweet and asked for external support for this claim. I searched, and found nothing from the claims author (IDC/Alinean).

I did, however, find a another vendor with the same data. The presentation notes in my colleague’s presentation said the $223,000 number was an average from a range including costs from $42,000 to $600,000. By searching for these numbers I stumbled across a conference session with more detail. In this second presentation, apparently delivered by SAP at a CA conference, cost of downtime is reported by application type. Here is their slide.

So, it seems EMC’s $223,000/hour number was calculated from the range above. A flat, basic, stupid average from the above table calculates a cost of downtime of about $193,000/hour. Finding the average cost by taking the average of those numbers is dumb, of course. But we could weight these application types in a hundred different ways. I feel fairly confident that somebody tried to collapse all eight numbers into a single, monolithic assessment that works in marketing slides. With slight weighting, they could have produced a $223,000/hour number.

So, to David and the rest of my readers, I do not have a public link from Alinean that I can point you to. I suspect that Alinean provided EMC and SAP these numbers for us to use with our customers. I conclude that we can trust the cost of downtime to be near $223,000/hour to the extent that we can trust Alinean.

But for the time being, the world is using server cache to speed up applications while living with mobility limitations.  Because of my performance background I am still a speed junky.  My long time in that field makes me a bit of a cynic, too.  When I saw a version of the following chart used in an internal EMC presentation I was skeptical.  Take at look at this and ask yourself if you believe it.

The benchmark is an Oracle OLTP workload.  VFCache is EMC’s PCI card that provides solid state cache at the server.  FAST Cache is EMC’s array-based solid state read/write cache.  At the request of EMC engineering I am only showing relative versions of IOPS measured from the application. This graph says that VFCache and FAST Cache provide multiplicative benefits to performance.  By multiplicative benefit I mean that when VFCache accelerates baseline performance by 30% it can also improve FAST Cache performance again by 30%.

How could this be?  I originally reasoned that the value of VFCache should be additive, not multiplicative.  I thought it would provide a total speedup to some fixed portion of the data.  A large array cache would speed up the rest of the data. In other words, VFCache speeds up 10% of the data a lot and FAST Cache speeds up 90% of the data to a lesser degree. The aggregate gain is the summary of the two, right?

Wrong.

VFCache can provide a multiplicative affect on performance. This means VFCache improves FAST Cache utilization and vice versa.  Here is why: by speeding up the rest of the data you are decreasing the application wait time.  When wait times go down, efficiency goes up.  When efficiency goes up, the utilization to each of these capabilities increases. We saw this efficiency gain when we saw VFCache throughput increase once FAST Cache at the array was enabled.

Because of the incredible performance benefits server cache provides, many customers are willing to live with the mobility limitations it imposes. Because of those mobility limitations we can really only think of today’s server cache as a precursor to the revolutionary technology we are waiting for. Eventually we will be able to federate the data on server cache, protect it, and promote and demote blocks between server and array. Eventually flash will be thought of as a true catalyst for change. But even today in its current form you can enjoy some screaming fast performance when you combine server cache and array cache.

Update: A Happy Addendum

After seeing my post one of the talented folks involved in this analysis sent me an email. He gently pointed out that the numbers I quote above are IOPS, as I wrote. These are indirect numbers for performance. The real application performance numbers, transactions per second, showed an even more dramatic benefit of the FAST Cache and VFCache configuration. The total application performance improvement for for these two technologies was 10x for this particular workload. So, take everything I said above, and multiply it again for this configuration.

At least that is how I read a press release from IDC dated 28 February, 2012.  Here are a few key points from the IDC statement:

“For the full year 2011, worldwide server revenue increased 5.8% to $52.3 billion when compared to 2010, while worldwide unit shipments increased 4.2% to 8.3 million units.”  These are the full-year numbers.  For the fourth quarter alone revenue and units were down.  But on the whole, 2011 was a good year.

But what about that fourth quarter fall?  ”Volume systems experienced a 2.0% year-over-year factory revenue decline to $8.8 billion, while midrange revenue decreased 4.6% to $1.8 billion when compared to 4Q10. Additionally, high-end system revenue declined 18.4% to $3.7 billion in the quarter.”  The cheaper systems declined the least.  Bigger systems were hit hard.  I usually equate “cheaper systems” with x86.  But its not fair to jump to that conclusion yet.

Even with the fourth quarter declines, one segment was up in the fourth quarter: “Linux server … revenue improved 2.2% year over year in 4Q11 to $2.6 billion.”  Linux servers are more commonly deployed on x86 than RISC.  Similarly Windows did well by falling less than server shipments in the fourth quarter: “Microsoft Windows server demand subsided slightly in 4Q11 as hardware revenue decreased 1.5% year over year.”  I believe these two points provide indirect support of the claim that x86 is taking over.

No point in beating around the bush any more.  The report states what IDC measured:

Unix servers experienced a revenue decline of 10.7% year over year to $3.4 billion representing 24.2% of quarterly server revenue for the quarter.

Followed shortly by:

The [x86] architecture accounted for 64.1% of all server spending. … As a result of a strong demand throughout the year, worldwide x86 server revenue for 2011 increased 7.7% to $34.4 billion.

I imagine that the 13.7% gap between the named Unix and x86 servers is accounted for by Windows on RISC and other rare operating systems.  But no matter where the chips fall in that group, x86 processors are dominating the market and growing much faster than their RISC counterparts.

But choosing between protocols does imply some design differences, limitations or benefits.  In this article I want to collect some of these items for your consideration.  As I asked my friends and colleagues about this subject I realized no one person could completely enumerate the protocol choice implications.  So, add your comments to the bottom and we will continue to update this article as a living document.

Lastly, this is a comparison of the differences between block and file in a VMware environments.  Differences in physical will not be discussed.  A comparison of different block protocols is left as an exercise to the reader.

Throughput

Throughput is usually limited by the backend storage configuration, not the protocol.  Only when an array can serve data very fast is throughput gated by the link speed.  Obviously 10 Gb is faster than 8 Gb which is faster than 1 Gb.  But here multipathing across many links gives block an advantage.  vSphere’s use of an NFS v3 client makes load balancing tricky, at best.  But multiple block paths can be simply aggregated to deliver a higher maximum throughput than a single Ethernet link.

Latency

Block is nominally faster.  I recall seeing this when I still worked at VMware but I cannot find supporting data now.  My memory is confirmed with a quote from a very old VMware paper: “The data also demonstrates [sic] that…Fibre Channel has the best performance in throughput, latency, and CPU efficiency.”

CPU efficiency

Block can be more efficient but that is not inherent to its “blockiness”.  Protocols managed an HBA use less CPU.  But NFS is more cycle-efficient than SW iSCSI.  Fibre Channel is better than HW iSCSI which is better than NFS which is better than SW iSCSI.  Check page 6 of VMware’s aging vSphere 4 protocol comparison for details.

Simplicity

This is obviously subjective. But can anyone argue that NFS is the king of simplicity?

Robustness/Availability

From vSphere’s perspective block must be considered more available.  vSphere provides no multipathing options for file storage.  Physical design of both Fibre Channel and Ethernet can be improve availability. But regardless of the physical design, vSphere layers more availability options with block’s multipathing options.

Cost

The answer here is tricky.  Ethernet is the cheapest storage interconnect and there are three protocols, two block and one file, that can use it.  But I feel the CPU cost per IO of iSCI to make it prohibitive with mission critical applications.  And Fibre Channel over Ethernet (FCoE) may introduce new requirements besides a cheap NIC and switch that drive up its cost.  So, given its undeniable simplicity and use of any network hardware I will award file the title of cheapest protocol.

Summary

If you look at the above list you’ll see that throughput, latency, and CPU efficiency are really three facets of the same gem: performance.  If we consolidate those three into a single performance field and crown block the winner, then the final score is two for file and two for block.  But because simplicity is so important in early VMware deployments I always recommend file storage for new virtualization deployments.

But to take this article full circle, the operational competency of your VMware and storage admins is much more important than any slight lead in the above list.  Stick with what you know.  When you get really great at both protocols, use block for your mission critical applications and file for the vast majority of your virtual machines.

A recent InfoWorld article summarized results of a study by Foote Partners, LLC that analyzed the changing value of IT certifications.  The InfoWorld piece provided some high-level observations that piqued my interest.  One Google search later I found the original report, which included details more relevant to those of us that work with VMware.  Below are highlights from the Foote Partners work that you may find interesting.

The free article published by Foote Partners is a bit of an infomercial.  They provide this summary as a way of advertising their $3000 report.  The $3000 report puts exact value to IT certifications.  The free report that I am summarizing only includes relative values.  Among those relative values, a decrease means the certification yields a smaller premium than the previous year.  A decrease likely means the industry is demanding this skill less or the workforce is over-supplying the skill.

What follows is my summary of “interesting” changes in the value of VMware-related certifications.  I used my own judgment in calling these certifications “VMware-related”.  I include Java certifications, for instance, because of VMware’s Spring acquisition.  I included infrastructure certifications because often VMware administrators also own other areas of the infrastructure.  I do not include a database certification because I do not consider database administration tightly connected to VMware product skills.

Declining value VMware-related certifications:

• Citrix Certified Administrator (CCA)
• CompTIA Network Technician (Network+)
• Juniper Networks Certified Internet Professional
• Juniper Networks Certified Internet Associate
• Avaya Certified Specialist
• Avaya Certified Expert
• Cisco Certified Design Associate (CCDA)

Rising value VMware-related certifications:

• Oracle Certified Professional Java Programmer (formerly Sun SCJP)
• Cisco Certified Internetwork Professional (CCIP)
• Juniper Networks Certified Internet Specialist
• Cisco Certified Design Professional (CCDP)
• Microsoft Certified Systems Administrator (MCSA)
• HP Accredited Platform Specialist (HP APS)
• Red Hat Certified Architect (RHCA)
• HP/Accredited Integration Specialist (AIS)

Highest value VMware-related certifications:

• Oracle Certified Master, java EE Enterprise Architect (formerly Sun SCEA)
• Master IT Certified Architect (ITAC/The Open Group)
• ITIL Service Manager Certification
• IT Certified Architect (ITAC/The Open Group)
• Cisco Certified Design Expert (CCDE)
• Cisco Certified Design Profesional (CCDP)
• Cisco Certified Internetwork Expert (CCIE, all variations)
• Cisco Certified Internetwork Professional (CCIP)
• Juniper Networks Certified Internet Expert
• SNIA Certified Storage Networking Expert (SCSN-E)
• Microsoft Certified Architect (MCA)
• HP/Accredited Integration Specialist (AIS)
• Citrix Certified Integration Architect (CCIA)
• IBM Certified Infrastructure Systems Architect
• HP Master Accredited Systems Engineer (Master ASE)
• Red Hat Certified Architect (RHCA)
• HP/Accredited Systems Engineer (ASE)

The VMware Certified Advanced Professional (VCAP) was recognized on page 27 but not tracked in the study.  Presumably that certification is too new.  The VMware Certified Professional (VCP) and VMware Certified Design Expert (VCDX) are recognized and measured but not mentioned in the summary.  This probably means that the value of those certifications has been stable in the previous 12 months.  It also means that they are not among the highest value certifications.

The report also highlights non-certified IT skills.  That is a much larger list so I will not copy-and-paste here.  But there are notable gains in value for Ruby on Rails and Spring Framework.  There are declines in values of J2EE, Microsoft Hyper-V, VMware Server/ESX, ESXi Server, and all non-mobile operating systems.  The trend is clear: companies are paying more for skills in frameworks and complex applications.  Companies are paying less for operating system and programming language skills.

Now, as a member of the storage vendor community, I will answer for EMC. I will do so with detailed text from an outstanding TechBook I recently discovered on EMC’s Powerlink. This paper, Using Symmetrix Storage in VMware vSphere Environments (Version 7), provides incredible detail on the relationship between VMware thin disks and Symmetrix virtual provisioning. Its guidance is clear and simple.

My summary observations are as follows, with more detail below.

• The decision about thin/virtual provisioning is primarily a management one: trust your vSphere capacity management processes?  Use vSphere thin provisioning.  Trust your array capacity management processes?  Use virtual provisioning.
• VMware’s lazy zeroed thick disks reserve the space in VMFS.  But because blocks are left uninitialized until the first write, these disks are thinly provisioned in the virtual pool.  Choose this configuration when you have mature storage capacity management processes but questionable capacity management processes in vSphere.
• VMware’s thin disks do not reserve space in VMFS nor in the virtual pool.  This “thin-on-thin” configuration presents the most flexibility but requires mature capacity management in vSphere and the array.
• Even if your storage capacity management processes are mature, there are some cases where you may want eliminate the chance of an out-of-capacity problem.  This might be true with a mission critical application. In this case you have two options:
• Do not use virtual provisioning in the array, which may greatly increase your storage consumption.
• Use VMware’s eager zeroed thick disks on EMC virtual provisioning, which reserve capacity in VMFS and the virtual pool.  With VAAI, Enginuity records but does not perform the zeroing.  This means the zeroing is instantaneous and the block is reserved.

Here is a long quote from the TechBook, starting from page 128:

In general, before vSphere 4.1, EMC recommended using “zeroedthick” instead of “thin” virtual disks when using Symmetrix Virtual Provisioning. The reason that “thin on thin” was not always recommended is that using thin provisioning on two separate layers (host and array) increases the risk of out-of-space conditions for the virtual machines. vSphere 4.1 (and even more so in vSphere 5) in conjunction with the latest release of Enginuity integrate these layers better than ever. Consequently, using thin on thin is now acceptable in far more situations, but it is important to remember that risks still remain in doing so. For this reason, EMC recommends “zeroedthick” (for better reliability as only the array must be monitored) or “eagerzeroedthick” (for absolute reliability as all space is completely
reserved) for mission critical virtual machines.

It is important to remember that using “thin” rather than “zeroedthick” virtual disks does not provide increased space savings on the physical disks. As previously discussed, since “zeroedthick” only writes data written by the guest OS, it does not consume more space on the Symmetrix thin pool than a similarly sized “thin” virtual disk. The primary difference between “zeroedthick” and “thin” virtual disks is not a question of space, but a question of the quantity and capacity of the Symmetrix thin devices presented to the ESX server. Due to the architecture of Virtual Provisioning, binding more thin devices to a thin pool or increasing the size of them does not take up more physical disk space than fewer or smaller thin devices would although it does require a small amount more Symmetrix cache. So whether 10 thin devices are presented to an ESX host, or 1 thin device, the storage usage on the Symmetrix is essentially the same. Therefore, packing more virtual machines into smaller or fewer 128 thin devices is not necessarily more space efficient. Prior to ESXi 5, a single-extent VMFS volume was limited to approximately 2 TB and this led customers to use “thin on thin” to keep the number of devices presented to a host minimal as a way to ease management. “Thin” virtual disks allowed for fewer presented devices and a higher virtual machine density. Now with the larger allowable single-extent VMFS volume size in ESXi 5 (up to 64 TB) the need to add more thin devices is reduced as larger, more appropriately sized thin devices can be presented and used and expanded as necessary. With ESXi 5, customers therefore do not need to make the virtual disks “thin” in order to fit a large number of them on a single volume as it is no longer constrained by the 2 TB minus 512 byte limit.

However, as previously mentioned, VMware and EMC have recently ameliorated the practicality of using thin virtual disks with Symmetrix Virtual Provisioning. This has been achieved through refining vCenter and array reporting while also widening the breadth of direct Symmetrix integration through features such as the vStorage API for Storage Awareness (VASA—vSphere 5 only). Additionally, the slight performance overhead that existed with thin VMDKs
which was caused by zeroing-on-demand and intermittent expansion of the virtual disk as new blocks were written to by the guest OS has been significantly diminished with the advent of Block Zero and Hardware-Assisted Locking (ATS). The introduction of Storage Dynamic Resource Scheduler in vSphere 5 (SDRS) further reduces the risk of running out of space on a VMFS volume as it can be configured to migrate virtual machines from a datastore when that datastore reaches a user-specified percent-full threshold. This all but eliminates the risk of running out of space on VMFS volumes, with the only assumption being that there is available capacity elsewhere to move the virtual machines to. These improvements can make “thin on thin” a much more viable option—especially in vSphere 5. Essentially, it depends on how risk-averse an organization is and the importance/priority of an application running in a virtual machine.

If virtual machine density is valued above the added protection provided by a thick virtual disk (or simply the possible risk of an out-of-space condition is acceptable), thin on thin may be used. If “thin on thin” is used, alerts on the vCenter level and the array level should be configured.

If you want the full copy of this document, go to Powerlink and search for the paper’s title, “Using EMC Symmetrix Storage in VMware vSphere Environments”. If you change the “Filter by Content Type” drop-down to “Technical / White Papers” this document will appear first in your search.

Very few customers and partners require the full-time support of a VCDX. Most customers with even a modest VMware environment know the VCP certification does not measure enough design and administration skills. To address this, in 2011 we saw the introduction of the VMware Certified Advanced Professional (VCAP).

As a member of the VMware partner community, I can tell you EMC highly values the VCAP certification for our pre-sales roles.  As a hiring manager, I consider the VCP the minimum proficiency for anyone that will sell into today’s enterprise.  But technical sales experts and evangelists that brandish the VCAP-DCD or VCAP-DCA can engage in deep technical discussions on customers’ virtualization plans.  This is a great asset if you want to work for infrastructure vendors, resellers, or integrators.

Earlier today Alastair Cooke announced virtual, live VCAP training and study sessions at times convenient for those of us in the Asia-Pacific region.  This will be a four-session series of interactive lessons under the ProfessionalVMware BrownBag umbrella.

Alastair and the fantastic VMware’s Iwan Rahabok (VCAP-DCD!) developed this Asia-Pacific series.  The agenda is set, and some deep technical experts have already been booked to lead the sessions.  The last thing needed is you!

You can register for the series on GoToWebinar. Alastair is still looking for hosts for some of the sessions and has solicited volunteers on his blog. Please contact him using the details on his blog if you are interested.