Comments on: Another Day, Another Misconfigured Storage

By: MemTrimRate for ESX VMs | vReference

MemTrimRate for ESX VMs | vReference — Wed, 16 Dec 2009 15:01:45 +0000

[...] some experts point out, ESX performance issues are often caused by mis-configurations with the storage arrays. There [...]

By: EMC Storage Viewer « DeinosCloud

EMC Storage Viewer « DeinosCloud — Mon, 30 Nov 2009 13:30:06 +0000

[...] 30, 2009 by deinoscloud As Chad Sakac said in a response to a blog post at vpivot.com : “We are planning to add performance detail to the (free) EMC [...]

By: deinoscloud

deinoscloud — Mon, 23 Nov 2009 14:22:07 +0000

Hi Chad, when this new version of the EMC Storage Viewer will be out? Performance details will be available for ESX3.x/VC2.5 as well?

Thx,
Didier

By: Storage Performance Monitoring – Relevant Counters « DeinosCloud

Storage Performance Monitoring – Relevant Counters « DeinosCloud — Mon, 23 Nov 2009 14:08:43 +0000

[...] Vpivot.com, VMware.com, VMware.com, [...]

By: Chad Sakac

Chad Sakac — Sun, 22 Nov 2009 17:20:46 +0000

Disclosure – I’m an EMC employee

Good article Scott – misconfigured storage is indeed often the problem. IMO, the issue is most people ask/think about storage in “GB”, when in most cases, “MBps” and “IOps” are generally the constraints and should be the design focus for datastores and virutal disks. Regardless of datastore type, thinking about bandwidth (pipe to storage) and throughput (IOps, and also a function of latency – bandwidth consumed is a function of IOps x IO size, and modulated by latency) is **VERY** important.

This isn’t hard, but when there is a “chinese wall” between the “cylinders of excellence” of the virtualization silo vs. app silo vs. storage silo, this can be tricky.

We are planning to add performance detail to the (free) EMC storage viewer vCenter plugin shortly to help (see into the perf stats from the array, and link it to the vSCSI and ESXtop results). In the meantime, I would strongly start when troubleshooting with:

1) quick glance at guest level IO counters. This is as simple as looking at top in linux, or perfmon in windows.

2) if it looks bad (latency >20ms is usually bad), take a look at the vSCSI and ESXtop results. If they are bad….

3) talk to your storage team, and if you are the storage admin, every array has tools to quickly analyze the storage load.

If you find that the storage iteself isn’t huffing and puffing, but the ESX server is, look at IO queuing (there’s a post I wrote on virtualgeek on this topic of “microbursting”), or if using NFS/iSCSI – look at bandwidth and multipathing.

By: David Francis

David Francis — Thu, 19 Nov 2009 03:50:03 +0000

Excellent post couldn’t agree with you more , being a Storage Admin and having to monitor and build RAID groups , for the “unknown” guest which may come along.

At the moment I am designing RAID groups for VDI , which is a really interesting.

By: Barrie

Barrie — Tue, 17 Nov 2009 00:22:37 +0000

The one major problem with IOP analysis with any tool (SWAT or perfmon disk transfers) is that your peaks are affected by cache hits which can mean you get a peak of 2500 IOPs on a RAID5 array capable of only 900 (just an example I had), but the average was only 85 over an 8 hour period – hard to judge what is a real physical limitation