Scott Drummonds on Virtualization

vSphere Performance Leadership with Terminal Services


Project VRC’s latest update (document available with registration) to their ongoing analysis of Terminal Services and XenApp performance in virtualized environments supports VMware’s claims of industry-leading performance.  There are two main conclusions of the revised version: (1) VMware outperforms XenServer and (2) previous performance measurements of XenServer were in error, reporting artificially high results on that product.

The story of Project VRC’s work goes back nearly a year.  This group of Dutch consultants created an ambitious project to quantify the performance of desktop virtualization platforms.  Their workload simulates multiple users running desktop applications in a Terminal Services environment. A few operations—including window appearances, Windows calculator load time, and others—are timed and the number of users is increased until a specified response time limit is reached.

While this may appear to be straightforward, benchmarking is deceptively difficult, as Project VRC will attest. Through engagements with VMware they have identified a performance gain by using ESX’s hardware assist monitor mode and acknowledged a timing bias in XenServer.  Furthermore, Project VRC has engaged other partners to improve the workload, including minimizing “stuck sessions” and reducing the impact of in-guest sleeps, which cause a variety of problems.

With this new paper, we can now make a few observations about the Project VRC workload and the platforms it runs on:

  • The previously reported XenServer numbers were incorrect.  The old results showed a significant performance lead for XenServer but the update reports that vSphere 4.0 outperforms XenServer 5.5 by 3.5%.
  • Project VRC has recognized the inaccuracy of sleep system calls in virtual environments so they have removed them from within timed operations.
  • Some system sleeps remain in the benchmark, which will result in unpredictable operation density from run to run and across platforms.

Project VRC plans to continue to develop their workload until it closely matches customer environments.  This includes fixing any remaining in-guest timing issues that may continue to bias their results.  VMware will continue to work with them to help them eliminate issues and produce a benchmark we can all stand behind.

6 Responses

Very nice work.

Just wondering, have you guys tried working with Citrix as well as VMware to get the best outputs?

  • The work I reported on here was done by Project VRC, a group of Dutch consultants. They spent about four weeks with Citrix once they acknowledged the measurement error in XenServer. This allowed Citrix to put their best foot forward.

  • Beautiful !
    Citrix only advantage is gone forever, and they can stop talking about this as a broken record.

  • 3.5%

    Margin of error numbers.

    Not worth the cost of vSphere licenses.

  • Bruce,

    First, this performance is available with ESXi for free.

    Second, virtualization slows down performance in many cases. But at least six ISVs are making money in this space because they offer more to their to their customers than just performance. As to whether their respective features are worth the cost, the market will decide.


    • Funny, you say the performance ESXi is free? But what do you get with ESXi that you dont get with the XenServer 5.5 free version? Calculate on that! 🙂
      Furthermore, the story is different… when it comes to vmware. you will always have to read between the lines….always..