Scott Drummonds on Virtualization

Storage Performance Analysis: SingB Case Study


Prepare to get deep into storage.

In the past few weeks I had the pleasure of getting deep into a customer’s performance problem.  Ultimately we identified some interesting issues in the environment that we traced back to an overloaded array.  Like most performance problems, the complaints started at the application layer and then shifted to vSphere.  Like many configurations, it was difficult to pinpoint why the storage was slow.  But EMC account teams pride themselves in customer responsiveness. We assembled a small team to help out. I was amazed and grateful that experts from our midtier specialists in Australia, Malaysia, and India all pitched in on the analysis!

If you are a VMware administrator you may choose to leave the nuts and bolts of storage management to your storage teams.  While this article talks about those nuts and bolts, I ask you to read on.  A little knowledge about how your array works will make you an awesome VMware administrator.  It will help you work with your storage administrators to get the most out of your array.  When your array is at its best, so are your virtual machines.

The analysis you below is the product of tools EMC can run against your EMC storage in a very short time.  The data collection took 24 hours in this case.  But the figures I will show were auto-assembled in minutes.  This is one of the many cool things an EMC technical consultant or one of our partners can do for you.

Read the rest of this entry »

Server Flash + Array Flash


With the recent general availability of VFCache, EMC has buzzed with ideas about what to do with server-based solid state storage.  Server-based solid state has been around for years.  I remember when Fusion-io visited us at VMware in 2009.  I spent a lot of time thinking about use cases, value, costs, and features. Now at EMC I am asking myself even bigger questions: how far can we go with this technology?  How much can we federate it, migrate data among nodes and within shared storage, protect it, and replicate it?  There are a lot of smart people in EMC that are way ahead of me on this.

But for the time being, the world is using server cache to speed up applications while living with mobility limitations.  Because of my performance background I am still a speed junky.  My long time in that field makes me a bit of a cynic, too.  When I saw a version of the following chart used in an internal EMC presentation I was skeptical.  Take at look at this and ask yourself if you believe it.

Read the rest of this entry »

Protocol Comparison: Block Versus File

1 Comment »

Customers have asked me to recommend a protocol for their vSphere environments more times than I can remember.  The best answer to this question is “stick with what you know”.  By far staying with an existing infrastructure is the best solution.  This leverages your existing skills, minimizes risk, and keeps costs down.  And no protocol can on its own claim to be the undisputed best choice.

But choosing between protocols does imply some design differences, limitations or benefits.  In this article I want to collect some of these items for your consideration.  As I asked my friends and colleagues about this subject I realized no one person could completely enumerate the protocol choice implications.  So, add your comments to the bottom and we will continue to update this article as a living document.

Read the rest of this entry »

VMware Thin Disks on EMC Virtual Provisioning


Even before I left VMware for EMC I was being asked to comment on “thin on thin”: the use of VMware thin VMDKs on virtually/thin provisioned storage.  As a VMware employee I recommended VMware’s thin provisioning but referred to storage vendors for their own best practices.

Now, as a member of the storage vendor community, I will answer for EMC. I will do so with detailed text from an outstanding TechBook I recently discovered on EMC’s Powerlink. This paper, Using Symmetrix Storage in VMware vSphere Environments (Version 7), provides incredible detail on the relationship between VMware thin disks and Symmetrix virtual provisioning. Its guidance is clear and simple.

Read the rest of this entry »



In today’s post I want to update and amplify thoughts from an old post on Storage IO Control (SIOC).  VMware customers that are using SIOC may sometimes see the following vCenter alarm:

Non-VI workload detected on the datastore

Or you may see the following warning in the vSphere client:

An external I/O activity is detected on datastore …

Read the rest of this entry »

Flash Or SSD? (or: Why Interfaces Matter)

Comments Off on Flash Or SSD? (or: Why Interfaces Matter)

In my three part series on flash I interchangeably used the terms “flash” and “SSD”.  In a recent article on this subject, Steven Foskett on IBM’s Storage Community successfully convinced me that I should stop using these terms interchangeably.  He then suggested that flash would persevere while SSD would not.  I disagree.

Read the rest of this entry »

The Flash Storage Revolution: Part III


In this final installment of the series, I will provide some detail behind flash storage sizing.  My previous entry contained an analytical and theoretical approach to sizing flash in today’s storage.  When I first studied the ideas I introduced in that post, I thought the flash sizing exercise was hopeless.  After all, how are customers to measure data cooling?  How could a storage admin quantify skew?

As it turns out, familiarity with these abstract concepts is not needed to size flash in your environment.  The same principles that Intel and AMD apply in sizing microprocessor cache can be applied to storage.  There are generalizations that will suit the majority of deployments.

Read the rest of this entry »

The Flash Storage Revolution: Part II


In the previous entry on this ongoing series covering the flash storage revolution, I concluded that flash is now an essential part of enterprise storage. But its value proposition is hinged on high utilization. High utilization cannot be sustained without efficient auto-tiering or accurate cache sizing for flash-based cache.

This article will describe the theory behind optimal cache sizing.  Practical guidance will follow in part three, the last entry in this series. I will again lean heavily on Denis Vilfort’s presentation that I offer for download on my blog.

Read the rest of this entry »

The Flash Storage Revolution: Part I


Six weeks ago I finally upgraded my MacBook to solid state storage.  The change in performance is so dramatic, to say the least.  I have been selling flash storage to EMC’s customers for over a year now and they have been loving it.  But I did not really get how valuable flash is until I saw it on my own laptop.

After this revolution of my own mind, I want to dedicate a few blog entries to the issue of solid state storage in the enterprise.  First I want to frame the problem that flash both solves and causes.  In the second entry I will introduce some of the theory behind flash sizing.  My last article will give you some very simple practical advice on how to use flash in your enterprise.

Read the rest of this entry »

vStorage APIs for Storage Awareness (VASA)


I am currently in Beijing halfway through a five city roadshow to present and listen to EMC’s technical pre-sales team. One of my roles in this traveling show is to talk about vSphere 5, and all of the great things EMC is doing to make it even better for customers. A big part of my talk is centered around the new vStorage API for Storage Awareness, or VASA. I think this new API is going to provide value far beyond what most people realize.

Read the rest of this entry »