Pivot Point

Scott Drummonds on Virtualization

Can VI and Storage Administrators Play Well With Each Other?

2 Comments »

This week I am in Tokyo visiting my colleagues in EMC and our good friends at VMware and Cisco.  Today in a EMC/VMware solutions exchange, I talked about the continued problems with storage configurations that are blamed on the virtual infrastructure.  These misunderstood problems slow VMware deployment, tarnish VMware’s name, and inhibit the customer’s ability to extract value from their purchase.  VMware, EMC, and the rest of the storage vendors need to do a better job at helping VI administrators identify and correct storage problems.

In many of the environments I have diagnosed, I have traced the problem to a poor relationship. VI admins lack basic storage skills and storage admins are supplying LUNs via email or web requests, not interactive design sessions. I offer one customer–protected through anonymity–whose story showed a failure of the storage/VI relationship.

It was a couple of years ago that this customer asked for my recommendation on extents.  I told him there exist no performance scalability concerns with extents, but ailing LUN diagnosis can be difficult.  He said extents were a requirement in his environment because the storage admin would only provide him standard, preconfigured 20 GB LUNs.  If he needed larger volumes, the storage admin insisted he aggregate in software (RAID, LVM, extents, etc.)  I immediately knew this lack of cooperation would doom them to failure.  Would it surprise you to hear that I heard from this customer many more times as problems were escalated to me?

It occurs to me that three things will decrease the storage mistakes that get blamed on VMware:

  1. Regular meetings with people from VMware and EMC so everyone understands these problems, can identify them, and can help each other work through them.
  2. Good VMware tools to help VI administrators recognize storage bottlenecks so they go to their storage team before going to VMware.
  3. An increase in VMware administrators’ view and control of storage so they become partners in storage decisions and not nameless, voiceless customers.

The good news is that solutions are present or imminent:

Problem Solution
EMC/VMware information sharing Meetings like I am doing in Tokyo and all over APJ
VMware storage tools vCenter, esxtop, vscsiStats, SIOC*, Storage DRS**
VI admin storage visibility and control EMC’s storage plugins and other vendors’ tools

(*) Demonstrated by VMware but not announced or committed to a release.
(**) Not demonstrated ever but we can dream, right?

OK, team. I know I have been preaching to the choir for years about fixing these performance problems. It is now time for some preventative maintenance. Storage vendors, help VMware by educating their customers on how to diagnose and correct storage problems. Customers, install the vCenter plugins from your storage vendor and be sure you understand what you are looking at. VMware, get your new features out.

OK, everyone put your hands in the circle. Shall we do this? OK, break!

Storage IO Control Video

No Comments »

The good folks at VMware have given another preview of a feature they are testing in their labs.  The Youtube video sets to music one of the coolest performance features being talked about right now, Storage IO Control.  I think it is perhaps the second hippest video ever made on an enterprise product from VMware.  But, hey, I am biased!

Read the rest of this entry »

Storage IO Control

19 Comments »

Last year at VMworld 2009, Irfan Ahmad and Ajay Gulati presented a preview of an unreleased technology VMware is calling Storage IO Control (SIOC). SIOC is a feature aimed squarely at the number one cause of VMware performance problems: underperforming storage. Year after year I see misconfigured storage slowing virtualized applications with VMware blame for the problem. Now VMware hopes to add a new tool to our administrators’ toolboxes to help them identify and mitigate underperforming storage.

Read the rest of this entry »

Windows Guest Defragmentation, Take Two

25 Comments »

I have received questions about guest defragmentation tools for years.  Until today I could only pose theories as to the value of guest defragmentation.  But previous theories spawned new research and one of VMware’s partners is now putting data behind their argument that file systems in Windows virtual machines require defragmentation.  This partner, Raxco Software, shared early results of this investigation with me.  Raxco used their NTFS defragmentation tool PerfectDisk to evaluate the impact of guest defragmentation on a single virtual machine.

Read the rest of this entry »

Upping VMware's Storage Game

3 Comments »

If you have heard me talk in the past year, you have heard me relate stories of storage issues causing performance problems that are wrongly blamed on VMware.  Mark Bowker of ESG recently presented a generalized version of a customer story that relates the same tale.  Luckily (for me) the customer caught the problem before completing the deployment.  This may have saved me a customer call to unravel a basic problem.  But the customer was burned and virtualization progress slowed if not stopped.

Read the rest of this entry »

Windows Guest Defragmentation

10 Comments »

Today at VMware Partner Exchange I had a lunchtime discussion with a partner of ours that makes a Windows file system (NTFS) defragmentation tool. He related anecdotes of incredible performance acceleration credited to defragmentation and quoted a few numbers based on his test environment. When he asked me what VMware’s recommendations were on the subject I remained uncharacteristically silent. Do we have best practices on this?

Read the rest of this entry »

PVSCSI and Low IO Workloads

19 Comments »

Scott Sauer recently asked me a tough question on Twitter.  My roaming best practices talk includes the phrase “do not use PVSCSI for low-IO workloads”.  When Scott saw a VMware KB echoing my recommendation, he asked the obvious question: “Why?”  It took me a couple of days to get a sufficient answer.

Read the rest of this entry »

Virtual Storage Design: Application Consolidation

4 Comments »

Fixed recommendations for consolidation ratios are cancerous.  Whether we are talking about vCPUs per core, virtual machines per host, or VMDKs per LUN, there is no single number the represents the “right” ratio.  Accurate guidance requires workload characterization and fine tuning using vSphere’s performance counters.  Today I want to highlight one experiment that shows application choice impacting VMDK-to-LUN consolidation.  The inescapable conclusion is that sequential access data must be separated from random access files!

Read the rest of this entry »

Performance of Thin Provisioned Disks

14 Comments »

I have received questions about thin provisioned disk performance since we announced their full support in vSphere.  The questions usually center around scalability issues.  VMware’s customers fear locking contention as new blocks are created for growing disks and think that growing thin disks may punish the performance of all virtual disks on a volume.  Well performance engineering has finally released a paper on the subject and we are glad to say that thin disk performance is insignificantly different from thick disks.

Read the rest of this entry »

Another Day, Another Misconfigured Storage

7 Comments »

As I continue working with VMware’s Professional Services Organization (PSO) as they develop a performance troubleshooting service, I see repeats of a theme I wrote about a couple of months ago:  storage is causing most performance problems, not VMware.  In fact, I have yet to see a correctly-configured vSphere deployment that is running any application with unsatisfactory performance with respect to native.

Read the rest of this entry »

Blog WebMastered by All in One Webmaster.

Switch to our mobile site

WP SlimStat