Scott Drummonds on Virtualization

Physically Separate Management Cluster


My recent foray into the world of vShield produced a contention recommendation from my friends at VMware. Today I want to ask you to educate me and our peers. I want to know: do you think it is a best practice that management products should be run on their own in a vSphere cluster?

The recommendation that all management products should be placed in a separate physical cluster came from friends at VMware that I greatly respect. This guidance was raised as the proper workaround to the “vShield App will kill vCenter networking” issue. With vShield only managing test and production clusters and vCenter running in a separate management cluster, the network-killing circular dependency between vShield and vCenter is avoided.

Building a separate management cluster sounded plausible enough when I first heard it. But the more I thought about it the less I liked it. Did I not spend four years at VMware telling customers to consolidate their workloads into a shared infrastructure? Was I not recommending that separation should be done at the virtual layer, not the physical one?

After talking with colleagues David Lloyd and Roman Tarnavski, I now more strongly believe that physical resources should be pooled and not separated for any reason in a modern virtual-aware environment.

I would like you to share your opinion on this subject. Help me understand why you do or do not like the concept of management cluster separation. Here are a few questions to ponder:

  • Do management clusters have different availability requirements?  We concluded that availability tended to be a function of the application or guest operating system, not business unit ownership.
  • Do management clusters have differing hardware requirements?  We believe that production and test workloads span all possible configurations and that management workloads cannot be thought of as universally lighter or heavier than others.
  • Are there security policies that would require an air gap between management and the rest of the workloads?  Not in our experience.
  • Is there a reason–besides working around the vShield issue–to physically separate workloads?  We feel no.  In fact, we believe the VMware mantra that separation should be performed at the virtual level, not the physical.  Separation introduces inefficiency and should only be done with great cause.

There are many more dimensions from which this argument can be pondered.  Please tell me your thoughts.

43 Responses

Please educate me! Do you place management apps in a separate vSphere cluster? http://vpivot.com/2011/02/09/physically-

    • @drummonds I never did. Only concern was vCenter, and I got so that I didn’t mind having vCenter in Prod Cluster either.

    • @drummonds not currently but plan to when we get some new hardware. can’t justify the cost with current hardware.

    • @drummonds I see both in and outside of cluster. Usually all about policies, mgmt layer cannot be on same platform what it is managing

      • @DuncanYB Besides vShield problem, why can management not be on same platform?

        • @drummonds I am not saying they cannot be, it is just Policy in many enterprise organizations to not host management on the platform it is

          • @DuncanYB Agree that there is no good reason to separate. As far as policy, policy used to be “don’t virtualize”. Nuts to policy! 🙂

          • @drummonds @DuncanYB I haven’t tried it yet but it seems like there are a lot of “chicken and egg” problems with a virtual virtual center.

          • @drummonds cant dm 🙂

          • @drummonds maybe but some battles can’t be fought straight away, when your out in the field you need to evaluate how much a cust can take

          • @DuncanYB Yielding to a customer issue is one thing. #vmware stating a best practice is another. We’re talking about the latter.

          • @drummonds @duncanyb Reality Check… How it arrived or is determined to be a best practice is a factor also.

          • @RickVanover That’s what I am looking for. The reasons that drive one to conclude a management cluster should be created.

          • @drummonds “In my dreamy world” I’d always have 2 datacenters. Each w/ everything. Each managing and monitoring and failing over the other.

          • @drummonds Specifically to a “management cluster” I do think that is overkill.

          • @drummonds I do however think management tools off of what it is managing, is a good idea. Especially for something like alerting.

          • @drummonds @DuncanYB Specifically separation is something that customers know understand & meet requirements with-Takes a lot to change that

          • @RickVanover But, again, VMware would not be making that recommendation. The customer would.

          • @drummonds Depends on whose FUD

          • @RickVanover And the customer would be asserting it usually because of FUD.

          • @drummonds Security/management type like separation. It comes down to the organization OK to foot the bill for the separation recommendation

          • @drummonds Something I used to say.. “I can protect and manage to the level I am funded.” -> I seriously said that.

          • @drummonds Absent of that funding, I’d have to go all in single cluster.

          • @drummonds We can tweak this a bit and say Dev/Prod. I prefer separate. You prolly say consolidate and leverage DRS to maximum potential.

          • @drummonds @duncanyb VMW explictly recommends that large deployments ~50 or more should have a three node mgmt cluster. I like it personally

          • @drummonds @duncanyb can you put all of your eggs in one basket, yes, but depending on the business, might be better

        • @drummonds managing. Technically speaking I don’t see an issue myself and I feel it is overdone and unnecessary

        • @drummonds it’s really a pro’s and con’s argument, but they will be different in many environments. I see both sides, but it depends…

    • We do not have a separate management cluster. We try to spread all applications across multiple vSphere clusters so that IF an issue affected a cluster it would not take down an entire applicaiton. We do the same thing for the mgt layer.

      I have heard of this recommendation from VMware recently, but I feel more comfortable keeping things spread around and not putting all the management systems in one basket (cluster). We do create a daily report that shows vm placement so if we did have an entire site outage we have a good idea where the virtual machines reside that need to come online first such as vCenter, vCenter DB, etc.

    • @drummonds In big enviro I can see it, but VMW seem to be pushing this too strongly. Heard a # of partners discussing this at PEX

  • We just run one happy pool of resources. With all the advancements with NIOC and SIOC and other features of Enterprise plus seems like a waste to have them separate.

  • Scott,

    I quickly replied on Twitter, but as I got to thinking more about this I thought of what you said about separating at the “virtual” layer I came up with another idea…

    How about separating in a nested ESX instance?

    This would fulfill the requirement to separate the management, or what I call the “virtual infrastructure” VMs, from the Production VMs.

    The limitations I see around this currently is the lack of 64-bit OS support in the nested VMs and VMware’s official support stance.

    Of course we can speculate if this is something VMware is actively testing and plan to support. I think most would agree that this is an extremely useful feature and some time should be invested no only for this purpose but for setting up lab environments on- premises, etc. This would also drive more CPU and memory utilization and drive consolidation ratios even further. Please tell me if I am completely off-base here!

    That said, I will only separate the management apps into a separate cluster if I have some old hosts and extra licenses lying around. I can’t justify the cost of having a separate management cluster in our current 8 host environment.


  • Elaboration: Scott, With the hardening of HA and DRS, plus the robustness of tools such as storage vendors like EMC and NetApp have built into their vPlex and MetroClustering toolsets, the ability to maintain an uptime for these tools in the exact same way as we maintain an uptime for all of our other mission critical tools, databases, and network monitoring devices.

    Even back before these tools were made available, we began placing our monitoring and management tools for our virtual environment in our virtual environment. Our faith in what was available to us was such that it couldn’t be shaken.

  • […] This post was mentioned on Twitter by Xinity Bot and Eric Siebert, Scott Drummonds. Scott Drummonds said: Please educate me! Do you place management apps in a separate vSphere cluster? http://bit.ly/f5g2ow […]

  • We have management mixed in with production. Although have run into a similar situation as the aforementioned vShield killing vCenter issue: when upgrading virtual hardware in the move from ESX3.5 to 4.1. vCenter was setup in a new VM during upgrade, so no problem there, but the database runs on our MSSQL server VM. This VM is VM hardware v4, but can’t be upgraded to hardware v7 now, because the upgrade requires vCenter to be running, but it shuts down when the database server is restarted to perform the upgrade. And the upgrade can’t be done without vCenter running. Didn’t even occur to us as an issue until we tried it. Ended up having to go into the host directly to manually restart the SQL VM to get vCenter responsive again, and quickly kill the upgrade task, before it tried again. Eventually we’ll get things re-arranged, but it makes me consider at least moving back to MSSQL express edition on the vCenter VM…at least that keeps all the prerequisites on 1 VM together, less likely we’ll get ‘caught out’ forgetting about necessary components running on another VM. Still despite that issue, costs wouldn’t really be feasible to let us run management in a separate cluster. Only have 3 hosts…

  • Internal Politics. Architectual policy says a large enough deployment needs separation for its management components. In most cases, the management cluster can be leveraged for other management workloads such as DCs, DNS servers, LDAP, vMA appliance, PXE boot services, etc.

    It’s not always my decision. The way I rationalize it is that a separate management cluster is still a step above physical management infrastructure (physical vCenter, SQL, DCs, DNS servers, LDAP, vMA appliance, PXE boot services, etc.)

  • In an environment big enough to justify it, I think all clusters should be grouped by SLA, because some cluster level settings lend themselves to that model. High SLA = no overcommit, lower DRS threshhold, resource pools with reservations and an admission policy to now power on VMs that would violate limits. Low SLA = pretty much the opposite, with several in between. Let service level drive the cluster config, and create resource pools for BUs or orgs.

    In a perfect world, I would create a management cluster for all management tools…not just vCenter. Network monitoring tools, backup tools, inventory mgmt tools, etc. Group them all together, because they all have a similar SLA. If there are not enough management tools to justify that, then match them with other VMs that have a similar SLA, regardless of category or applications hosted on them.

    Good reasons for and against, but that is my “perfect world” view.

  • In these sorts of discussions it’s easy to argue or present both sides of the discussion. In reality most of the environements ive seen will make up their own mind and yes, generally policy decisions will win out over architectural ones.

    I dont necessarily believe the mantra that separating management apps into a separate cluster for the hell of it. I’ve had a customer refuse becuase it was a complete overkill to provide x hosts for a separate cluster for such a small amount of VM’s (for cost and other reasons). Their arguement was also they were sold on the fact that vSphere could be architected to provide HA and protect itself to an extent, (taking vCenter out of the equation). For them there was no requirement for separation on dedicated hosts/clusters.

    In essence I think there will be different requirements for different apps. Most I’ve seen they are happy to live in a specific RP or even within the a shared cluster. Separating these apps, and we can extrpolate to any specific workload, can have its benefits but ultimately it’s going to boil down to 1. What are you trying to achieve/protect and 2. Is there a clear business reason you have to do this. If you can answer those two questions then either way you will be able to justify your answer.

    Personally, I understand the idea even from an asthetic point of view and I can see the business feeling warmer and fuzzier. However it almost goes against what I understand as to the mechanism’s of vSphere which could/would alleviate 99% of the issues a separate management cluster could provide.

    I would like to hear the justfication from the guys you were talking to as to why it was their recommendation. For me, unless it’s required and the environment is big enough to warrant it, I wouldnt bother.

  • I think there could certainly be a use case for a separate physical cluster for management resources however, the problem I see is how are you going to secure it? If the whole linchpin of security is vShield and you can’t put vShield in the cluster where your most important asset resides (vCenter) what’s the point? All someone needs to do then is breach vCenter and they have access to all of your resources.

    If vShield is the security for the cloud then it needs to be applicable to all elements of the cloud, including their management parts.