Scott Drummonds on Virtualization

vShield, vCenter, and Management Clusters


I recently tweeted looking for VMware customers using the vShield products in production. There was very little response. It seems that vShield Edge and App are barely being used at all in production. It is possible that some rough patched need to be sanded down to polish vShield to its appropriate luster. But if you are not playing with these products now, you owe it to yourself to download the trial versions.

Our fantastic APJ vSpecialist organization has been spending with vShield. Many of us believe vShield to be one of the most exciting editions to the VMware portfolio in a long time. It is simple, elegant, and very powerful. But there is a very big danger with vShield: improper use can disable the network connection of vCenter virtual machines. Fixing this problem is not intuitive.

We first observed this issue in our quarterly hands-on workshop in Tokyo with multiple people toying with the vShield products (Edge, App, EndPoint) on a single cluster. Some sequence of operations disconnected the vCenter’s vNIC. We opened the VM’s settings, clicked the vNIC’s “Connected” checkbox, and tried to go back to work. But the vNIC remained unconnected. No matter how many times we repeated this process, the vCenter VM remained offline. With vCenter offline, the infrastructure was unmanageable.

The general problem is that installing vShield App or Zones on an ESX host running vCenter introduces a circular dependency that can make installation, operation, or uninstallation fail. A warning against doing this is documented in a small cautionary note on the bottom of page 13 of the current vShield Administration guide. Unfortunately, it is an older version of this administration guide (which lacks this critical warning!!!) that appears first in response to the obvious Google search.

This same problem is described as an uninstallation issue in VMware KB article (1028151). That KB also includes suggestions on resolution.

While vShield App is not fundamentally incompatible with management of a vCenter VM, this problem raises an interesting point. In even moderate environments it makes sense to create a separate management cluster for VMware’s management products. This includes vCenter, vCloud Director, CapacityIQ, the vShield Manager, and a large number of helper VMs for other products. The key here is that these clusters are protected by HA and vCenter Server Heartbeat, but they are not managed by the tools in the cluster.

Tools like AppSpeed, CapacityIQ, vShield and others should be deployed to measure and manage the production and test clusters, but not the management clusters. This prevents circular dependencies that could result in difficult-to-correct network problems or inefficiencies like AppSpeed monitoring management traffic. This is obviously a very high level description of something that needs a reference architecture. More to come on that next week.

For those of you that share our interest in vShield, I want to direct you to the blog of a colleague of mine in Australia. Roman Tarnavski, a Sydney-based vSpecialist, has been digging into vShield and publishing some tutorials and hacking guides. Roman and I met with VMware engineering and we think that Roman will likely soon be showing you some amazing possibilities when hacking to improve vShield Edge.

Roman has already written a couple articles on the vShield and networking:

Check back here and read Roman’s blog regularly if you want to hear more about vShield.  Roman will have some incredible information on the vShield Edge internals that all of you hackers out there will not want to miss.

8 Responses

  • Scott, the problem you described happened to me while testing vCloud Director. I blogged about it last year here: http://fojta.wordpress.com/2010/09/19/vshield-troubles/. I agree about the separation of management cluster which I did not do in my lab environment.

    • Tomas,

      Its surprising to me, and a bit disappointing, that vShield was released with such a dangerous flaw. I am glad you linked to your blog here and that I have linked to Roman’s. We need to build a web of discussions around this topic.


  • […] was reading Scott’s article about using dedicate clusters for management applications. Which was quickly followed by a bunch of […]

  • Hi,

    Two or three weeks ago I was partcipating vSphere 4: Design Workshop – a training dedicated mostly to VMware partners. It was quite a new idea to me to have a separate management cluster. It makes sense, but only for medium and big vSphere installations. For small customers this idea is not acceptable – dedicating 2 physical servers for mgmt only cannot be justified.

    vShield needs a few major improvments, this is true. One of them is to make it more difficult for the user to run into vCenter isolation problem. A small caution in the documentation seems not to be enough.


  • […] cluster” has been getting some attention recently. First was Scott Drummonds, with this post and this follow up. Duncan responded here. My take? I’ll agree with Duncan’s final […]

  • […] First thing I came across was a fellow vSpecialist Scott Drummonds blog post over at vPivot.com about it:  vShield, vCenter, and Management Clusters […]