This week I am in Tokyo visiting my colleagues in EMC and our good friends at VMware and Cisco. Today in a EMC/VMware solutions exchange, I talked about the continued problems with storage configurations that are blamed on the virtual infrastructure. These misunderstood problems slow VMware deployment, tarnish VMware’s name, and inhibit the customer’s ability to extract value from their purchase. VMware, EMC, and the rest of the storage vendors need to do a better job at helping VI administrators identify and correct storage problems.
In many of the environments I have diagnosed, I have traced the problem to a poor relationship. VI admins lack basic storage skills and storage admins are supplying LUNs via email or web requests, not interactive design sessions. I offer one customer–protected through anonymity–whose story showed a failure of the storage/VI relationship.
It was a couple of years ago that this customer asked for my recommendation on extents. I told him there exist no performance scalability concerns with extents, but ailing LUN diagnosis can be difficult. He said extents were a requirement in his environment because the storage admin would only provide him standard, preconfigured 20 GB LUNs. If he needed larger volumes, the storage admin insisted he aggregate in software (RAID, LVM, extents, etc.) I immediately knew this lack of cooperation would doom them to failure. Would it surprise you to hear that I heard from this customer many more times as problems were escalated to me?
It occurs to me that three things will decrease the storage mistakes that get blamed on VMware:
- Regular meetings with people from VMware and EMC so everyone understands these problems, can identify them, and can help each other work through them.
- Good VMware tools to help VI administrators recognize storage bottlenecks so they go to their storage team before going to VMware.
- An increase in VMware administrators’ view and control of storage so they become partners in storage decisions and not nameless, voiceless customers.
The good news is that solutions are present or imminent:
| Problem | Solution |
|---|---|
| EMC/VMware information sharing | Meetings like I am doing in Tokyo and all over APJ |
| VMware storage tools | vCenter, esxtop, vscsiStats, SIOC*, Storage DRS** |
| VI admin storage visibility and control | EMC’s storage plugins and other vendors’ tools |
(*) Demonstrated by VMware but not announced or committed to a release.
(**) Not demonstrated ever but we can dream, right?
OK, team. I know I have been preaching to the choir for years about fixing these performance problems. It is now time for some preventative maintenance. Storage vendors, help VMware by educating their customers on how to diagnose and correct storage problems. Customers, install the vCenter plugins from your storage vendor and be sure you understand what you are looking at. VMware, get your new features out.
OK, everyone put your hands in the circle. Shall we do this? OK, break!
My experience is different “Storage admins” tend to be the VI admins and communicating with the Telecommunications department tends to get “disconnected”. Storage and Servers tend to work hand in hand. It’s the networking team that works independently, from routing and VOIP to making sure switch ports are trunked and vlans presented, they have their own headaches keeping everything talking and diagnosing communication issues (even with the nexus 1000).
I’m off topic here but I find the idea of storage administrators being detached odd. Most storage administrators are versed in many OS’s and infrastructure designs out of necessity and are part of the server team.
In your scenario I question the wisdom of the departmental layout.
I’d have to say that in my work environment it is the storage admins that are disconnect like Scott mentioned. Not as bad as the example company he used but it’s there. The network team seems to work with me more and actually listens more to the virtual requirements. I think its a case of our storage admin being more set in his ways than anything else though.