Scott Drummonds on Virtualization

Improving Storage and Virtual Infrastructure Administrator Interlock

1 Comment »

Long before I left VMware for EMC I was writing articles on storage issues that were punishing virtual machine performance. As a member of VMware’s performance marketing, I spent an inordinate amount of time asking people to first check their storage to fix performance problems. While those efforts bore fruit in VMware’s more mature customers, the majority of new customers first blamed vSphere for performance problems. Storage issues went undiagnosed.

Educating VI administrators on storage troubleshooting and capacity management is a tactical fix to this problem. In the past year I realized that storage mismanagement in virtual environments is a systemic problem that is mainly experienced customers with immature VMware deployments. A solution to this systemic problem requires a more holistic view.

Read the rest of this entry »

Databases, Storage, and Solid State Disks

Comments Off on Databases, Storage, and Solid State Disks

A colleague of mine dropped by my desk on Friday to talk about storage best practices for virtualized databases (SQL Server in this case).  He observed a VMware deployment where the data and log files for a SQL Server virtual machine were consolidated on a single VMFS volume backed by a RAID 5 LUN.  “Is this a VMware best practice?” he asked.  “Should you not put the redo logs on a RAID 10 LUN?”  The answers are ‘no’ and ‘yes’, respectively.  And with the solid state disk (SSD) auto-tiering from EMC (FAST) the second answer is an emphatic “YES!”

Read the rest of this entry »

Can VI and Storage Administrators Play Well With Each Other?


This week I am in Tokyo visiting my colleagues in EMC and our good friends at VMware and Cisco.  Today in a EMC/VMware solutions exchange, I talked about the continued problems with storage configurations that are blamed on the virtual infrastructure.  These misunderstood problems slow VMware deployment, tarnish VMware’s name, and inhibit the customer’s ability to extract value from their purchase.  VMware, EMC, and the rest of the storage vendors need to do a better job at helping VI administrators identify and correct storage problems.

In many of the environments I have diagnosed, I have traced the problem to a poor relationship. VI admins lack basic storage skills and storage admins are supplying LUNs via email or web requests, not interactive design sessions. I offer one customer–protected through anonymity–whose story showed a failure of the storage/VI relationship.

It was a couple of years ago that this customer asked for my recommendation on extents.  I told him there exist no performance scalability concerns with extents, but ailing LUN diagnosis can be difficult.  He said extents were a requirement in his environment because the storage admin would only provide him standard, preconfigured 20 GB LUNs.  If he needed larger volumes, the storage admin insisted he aggregate in software (RAID, LVM, extents, etc.)  I immediately knew this lack of cooperation would doom them to failure.  Would it surprise you to hear that I heard from this customer many more times as problems were escalated to me?

It occurs to me that three things will decrease the storage mistakes that get blamed on VMware:

  1. Regular meetings with people from VMware and EMC so everyone understands these problems, can identify them, and can help each other work through them.
  2. Good VMware tools to help VI administrators recognize storage bottlenecks so they go to their storage team before going to VMware.
  3. An increase in VMware administrators’ view and control of storage so they become partners in storage decisions and not nameless, voiceless customers.

The good news is that solutions are present or imminent:

Problem Solution
EMC/VMware information sharing Meetings like I am doing in Tokyo and all over APJ
VMware storage tools vCenter, esxtop, vscsiStats, SIOC*, Storage DRS**
VI admin storage visibility and control EMC’s storage plugins and other vendors’ tools

(*) Demonstrated by VMware but not announced or committed to a release.
(**) Not demonstrated ever but we can dream, right?

OK, team. I know I have been preaching to the choir for years about fixing these performance problems. It is now time for some preventative maintenance. Storage vendors, help VMware by educating their customers on how to diagnose and correct storage problems. Customers, install the vCenter plugins from your storage vendor and be sure you understand what you are looking at. VMware, get your new features out.

OK, everyone put your hands in the circle. Shall we do this? OK, break!

From West to East


I have received several gentle nudges about the lack of content on Pivot Point of late. And a few of you have asked me what is going on with my professional life. For a while I hoped to quietly drift across continents and through companies. But I now feel an explanation is due. This description is mostly professional, somewhat personal, and a touch philosophical. But if you are interested in what moves me and what will come next for vPivot, please read on.

Read the rest of this entry »