Scott Drummonds on Virtualization

Maximum Concurrent VMotions


A VMware customer and attendee of a talk I gave at a performance roundtable asked me for a preview of unreleased features*.  When I talked about the amazing improvements to VMotion that would enable as many as eight concurrent VMotions the customer said, and I am paraphrasing here, “Yawn.  I can already do that.”  Really?  I had no idea customers could do this.  As it turns out, many of us at VMware did not know that customers knew how to do this.

VMware’s dedicated and curious customer base somehow obtained information on an undocumented parameter that limits the maximum number of concurrent VMotions.  Information on modifying this parameter is sprinkled around the internet tubes but my favorite comes from Jason Boche.  Jason and others have identified how you can change the current limit of two VMotions per host.  I want to give you an explanation of why you should not do this and show you what you can expect from future releases of vSphere.

The concurrent VMotion limit was set after careful analysis of the capabilities of existing hardware, VMotion implementation details, and a large number of enterprise applications.  It has been set at two to provide a near 100% guarantee of no downtime during the migrations.  On some workloads or on older hardware, it is possible that three or more concurrent migrations could saturate a system resource and result in downtime.  If your hardware is brand new and your application is not wildly touching memory, it is possible that you can somewhat safely increase the concurrent VMotion limit.  But, as Kit Colbert told me, “I think allowing four simultaneous VMotions is probably OK in most scenarios, but if the VMs are really large and/or have very big working sets, then I’d dissuade customers from bumping up the limits.”

And if Kit’s gentle reminder is not enough to dissuade you from making this change to your production environments, I will point out that problems that arise as a result of changing the concurrent VMotion limit are not supported by VMware.  We simply cannot promise the unfaltering quality of VMotion if end users increase this limit.

Now, back to the feature preview in my ongoing performance road show (today: Bellevue, WA!).  Using an in-house development version of ESX, we are running eight concurrent VMotions on a single host with the even better quality than that of vSphere 4.  A phenomenally dedicated group of engineers has drastically improved the throughput of VMotions and decreased the already tiny virtual machine stun time.  This means that we can maintain our zero downtime commitment while upping the number of concurrent VMotions by a factor of four.  Furthermore, total migration time has also decreased, so our development host can evacuate a large number of virtual machines almost order of magnitude faster than vSphere 4.

(*) Any time I talk about this or other unreleased features the standard VMware disclaimer applies.  We are not committing this feature to any specific product nor committing any product to a specific date.  Not my rules, guy

4 Responses

6 per host concurrent in my lab.

  • How about limiting the number of migrations to one? How would you do this? This might be useful if you want to keep a tight lid on resources used.