Scott Drummonds on Virtualization

VMware Linked Clone IO Implications


A month ago I was in Palo Alto seeing old friends and talking VMware. I spent some time with an old friend in performance engineering talking about vSphere’s implementation of linked clones. Conceptually I know linked clones require an additional map to translate the guest’s view of a contiguous file system into a reordered collection of blocks in multiple VMDK files. But I was not sure exactly how this technology worked and love hearing details. I want to share a few comments about this discussion.

Read the rest of this entry »

MLC Flash Versus SLC Flash

1 Comment »

EMC’s recent announcement at EMC World of Project Lightning documents a program to increase the use of flash devices in enterprise storage. The project includes increased use of flash storage in EMC arrays, all-flash storage configurations, and support for Multi-layer Cell (MLC) flash. This last subject–MLC flash and its difference from SLC flash–piqued my curiosity.

Many years ago I studied electrical engineering. I was an awful at it. Analog was never my thing. I much prefer ones and zeroes. But I challenge myself to think about electronics once every blue moon. So I decided to delve into SLC and MLC flash technologies to understand how they differ and why we should care.  The content below summarizes my online research and the little bit I remember from school. If you can add, correct, or update this article I would be happy to have your comments.

Read the rest of this entry »

Bring Storage and VI Admins Together


Three years ago I was not a very big fan of EMC.  We at VMware worked very hard to show the world that we were not under EMC’s thumb.  In many respects, this meant pursuing projects with EMC’s competitors before engaging EMC.  EMC was partly to blame for this. EMC’s best argument to win the VMware footprint was based on a claim of majority ownership.  That pointless claim offends VMware employees and does not translate into value in VMware environments.

But things started changing a couple years ago.  Through the heroic efforts of Chad Sakac, EMC stopped acting like it deserved the business of VMware’s customers and more like it had to earn that business.  I noticed this in the way we partnered at multiple EMC Worlds and VMworlds.  I started to hear EMC’s name mentioned when we were prototyping features and products like VAAI, SRM, and SIOC.  EMC employees started treating us more like partners than indentured servants.  And then EMC started releasing products that were making the lives of VMware’s customers easier.

Read the rest of this entry »

Improving Storage and Virtual Infrastructure Administrator Interlock

1 Comment »

Long before I left VMware for EMC I was writing articles on storage issues that were punishing virtual machine performance. As a member of VMware’s performance marketing, I spent an inordinate amount of time asking people to first check their storage to fix performance problems. While those efforts bore fruit in VMware’s more mature customers, the majority of new customers first blamed vSphere for performance problems. Storage issues went undiagnosed.

Educating VI administrators on storage troubleshooting and capacity management is a tactical fix to this problem. In the past year I realized that storage mismanagement in virtual environments is a systemic problem that is mainly experienced customers with immature VMware deployments. A solution to this systemic problem requires a more holistic view.

Read the rest of this entry »

Justifying SSDs


Ever since I saw the results of VMware’s first performance work on EMC’s Enterprise Flash Drives (EFDs) I knew the storage world was about to change.  Even though I love the idea of SSD, I still struggle with the justification of their purchase.  I have had trouble quantifying the value of an EFD and fearlessly committing customers’ money to their purchase.  In this article I want to offer a few thoughts on these devices as I formulate my own ideas as to when SSDs are needed and how we can all enjoy their benefits.

Read the rest of this entry »

Storage Consolidation (or: How Many VMDKs Per Volume?)


Part of the performance best practices talk I co-presented at VMworld in San Francisco and Copenhagen focused on answering the question, “How many virtual machines can be placed on a single VMFS volume?”  There are a lot of theories as to a best answer.  It will not surprise you to learn that no single consolidation ratio works in every environment.  Your workloads will influence the maximum consolidation.  But we know enough about how ESX virtualizes storage to provide guidance as to the right storage consolidation ratios.

Read the rest of this entry »

SIOC Event: Ignore or Panic?


A colleague of mine recently reported an alarming event witnessed at a customer of both EMC and VMware. After enabling Storage IO Control (SIOC), vCenter reported a possible problem with a datastore. The specific text of the event is:

External I/O workload detected on shared datastore running Storage I/O (SIOC) for congestion management

Read the rest of this entry »

Databases, Storage, and Solid State Disks

Comments Off on Databases, Storage, and Solid State Disks

A colleague of mine dropped by my desk on Friday to talk about storage best practices for virtualized databases (SQL Server in this case).  He observed a VMware deployment where the data and log files for a SQL Server virtual machine were consolidated on a single VMFS volume backed by a RAID 5 LUN.  “Is this a VMware best practice?” he asked.  “Should you not put the redo logs on a RAID 10 LUN?”  The answers are ‘no’ and ‘yes’, respectively.  And with the solid state disk (SSD) auto-tiering from EMC (FAST) the second answer is an emphatic “YES!”

Read the rest of this entry »

Can VI and Storage Administrators Play Well With Each Other?


This week I am in Tokyo visiting my colleagues in EMC and our good friends at VMware and Cisco.  Today in a EMC/VMware solutions exchange, I talked about the continued problems with storage configurations that are blamed on the virtual infrastructure.  These misunderstood problems slow VMware deployment, tarnish VMware’s name, and inhibit the customer’s ability to extract value from their purchase.  VMware, EMC, and the rest of the storage vendors need to do a better job at helping VI administrators identify and correct storage problems.

In many of the environments I have diagnosed, I have traced the problem to a poor relationship. VI admins lack basic storage skills and storage admins are supplying LUNs via email or web requests, not interactive design sessions. I offer one customer–protected through anonymity–whose story showed a failure of the storage/VI relationship.

It was a couple of years ago that this customer asked for my recommendation on extents.  I told him there exist no performance scalability concerns with extents, but ailing LUN diagnosis can be difficult.  He said extents were a requirement in his environment because the storage admin would only provide him standard, preconfigured 20 GB LUNs.  If he needed larger volumes, the storage admin insisted he aggregate in software (RAID, LVM, extents, etc.)  I immediately knew this lack of cooperation would doom them to failure.  Would it surprise you to hear that I heard from this customer many more times as problems were escalated to me?

It occurs to me that three things will decrease the storage mistakes that get blamed on VMware:

  1. Regular meetings with people from VMware and EMC so everyone understands these problems, can identify them, and can help each other work through them.
  2. Good VMware tools to help VI administrators recognize storage bottlenecks so they go to their storage team before going to VMware.
  3. An increase in VMware administrators’ view and control of storage so they become partners in storage decisions and not nameless, voiceless customers.

The good news is that solutions are present or imminent:

Problem Solution
EMC/VMware information sharing Meetings like I am doing in Tokyo and all over APJ
VMware storage tools vCenter, esxtop, vscsiStats, SIOC*, Storage DRS**
VI admin storage visibility and control EMC’s storage plugins and other vendors’ tools

(*) Demonstrated by VMware but not announced or committed to a release.
(**) Not demonstrated ever but we can dream, right?

OK, team. I know I have been preaching to the choir for years about fixing these performance problems. It is now time for some preventative maintenance. Storage vendors, help VMware by educating their customers on how to diagnose and correct storage problems. Customers, install the vCenter plugins from your storage vendor and be sure you understand what you are looking at. VMware, get your new features out.

OK, everyone put your hands in the circle. Shall we do this? OK, break!

Storage IO Control Video

Comments Off on Storage IO Control Video

The good folks at VMware have given another preview of a feature they are testing in their labs.  The Youtube video sets to music one of the coolest performance features being talked about right now, Storage IO Control.  I think it is perhaps the second hippest video ever made on an enterprise product from VMware.  But, hey, I am biased!

Read the rest of this entry »