• Monday, 30 Aug, 3:00-4:00PM: EXPERTS – 02 – Knowledge Experts One on One (Moscone West – Level 2)
• Tuesday, 31 Aug, 11:00AM-12:00PM: EA7726 – Virtual Machines Outperforming Physical Machines (Moscone South Room 307)
• Tuesday, 31 Aug, 2:00-3:00PM: TA7171 – Performance Best Practices for vSphere (Moscone South Room 104)
• Wednesday, 1 Sep, 12:00-1:00PM: EXPERTS – 08 – Knowledge Experts One on One (Moscone West – Level 2)
• Wednesday, 1 Sep, 3:00-4:00PM: EA7726 – Virtual Machines Outperforming Physical Machines (Moscone South Room 308)
• Thursday, 2 Sep, 12:00-1:00PM: GD 35 – Performance with Scott Drummonds (Moscone West Alcove 2)
• Thursday, 2 Sep, 1:30-2:30PM: TA7171 – Performance Best Practices for vSphere (Moscone South Room 302)
• Thursday, 2 Sep, 2:30PM-1:00AM: XX1234 – Post Conference Celebration (Various Bars in SF)

The registrations for the performance best practices talk are insane!  1100 people have already signed up for the talk and we are positively thrilled and humbled at the interest.  I know that you will not be disappointed with this year’s talk, because I am bringing a secret weapon.

This year, VMware’s head of outbound engineering will be leading the session.  Kaushik Banerjee has been at the helm of the outbound group since before I joined the company (Jan. 2007).  Every time I talked, blog, tweeted, wrote, or presented performance information Kaushik was behind me doing the research and providing the technical credibility.  It was his team that did the research for the Exchange 16K, 100K IOPS, 365K IOPS, DRS, and SPECweb announcements, among dozens of other white papers and blogs.  It is my pleasure to be able to share the stage with him next week.

On top of that, my good friend and former colleague Dave Korsunsky has asked me to co-present with him on the other topic, “Virtual Machines Outperforming Physical Machines”.  Dave and I co-presented the record-breaking “16,000 Exchange Mailboxes” talk at VMworld Europe 2008 (the data for which Kaushik gathered).  Dave’s expertise as a manager in VMware’s alliances organization goes beyond performance and he has worked on dozens of applications on a wide variety of configurations over the years.  I am again lucky to be invited to join him on stage.

As you can see it is going to be a fun show, and I am looking forward to it.  I’ll be on Twitter and trying to keep up with email so let me know if our paths will cross.

Several years I read an interesting book on the Cold War that documented some of the insane behavior of the superpowers that nearly culminated in the annihilation of mankind.  The book was rich in stories of espionage and assassination and political machination.  In one paragraph, buried somewhere in the book’s meaty center, the author includes an almost throwaway reference that has stuck in my memory ever since.

The story describes the USSR’s mining of titanium, which was used to produce a variety of products.  With no free market, the USSR lacked a proper valuation for the metal and the human energy required to process it.  Titanium is abundant in the region and there was a great demand for shovels.  With so much titanium, and no clear understanding of the value of the material and the cost of processing it, the USSR produced a lot of titanium shovels.

Without a proper value assigned to the product, the titanium shovels were sold and traded to the country’s neighbors at the going rate for “normal” shovels.  When the rest of the capitalist-driven world discovered the incredible bargain of titanium shovels and steel shovel prices, they bought USSR’s shovels at rock-bottom prices and converted the metal into high value items like rockets and planes.

The net effect was a tremendous waste of limited resources.  The valuable metal and limited human capital were directed into a venture that was not profitable and only beneficial to the USSR’s neighbors.  In addition to the story of shovels, the book related tales of unneeded refrigerators that piled up by the hundreds of thousands in warehouses and workers that dug canals through solid stone with wooden tools. These economic inefficiencies were among the many causes of the country’s collapse.

Here is where I get to IT.  If you are today running your environment with no clear understanding of the costs of your business with respect to value of your products then you are doomed to the same slow failure.  Your product–in this case virtual machines–have a value on the free market and will be bought, traded or wasted based on the cost you set.  Produce a “titanium” virtual machine for free and I promise you your customers will order it and waste it.  Produce a steel virtual machine at titanium costs and your customers will go elsewhere for their product.

To make your virtual infrastructure competitive in the free market you have to do two things:

• Establish a currency that can be used by everyone in the industry.
• Properly map your products to their costs and value.

Obviously what we are talking about here is a realistic chargeback system.  I know that many of you roll your eyes and shake your heads each time I talk about this.  But you have to realize that the free market is coming to you whether you like it or not.  Your internal customers are today being courted by public cloud service providers and if you cannot offer a better product at a competitive price, you are going to lose those customers.

Today you can survive without properly valuing your product (virtual machines) because you run a monopoly and few of your customers can escape your fiefdom.  But cloud service providers are setting up their systems at your gates, calling to your citizens with cheap jeans and Coca Cola, and laughing at your titanium shovels.  There is only so long that you can keep your internal customers locked up behind a Berlin wall before the economic gravity of the free market pulls your monopoly apart.

I have been nurturing a theory about chargeback that you will certainly read more about here.  But I now ask for your help in the formation of this idea.  Can you give me 30 seconds to share your experience with virtual infrastructure chargeback models?  If you have never implemented chargeback in your environment, this survey will take you five seconds.  But even those three multiple choice responses will help my investigation.

Here is the survey.  I much appreciate your thoughts.

A VMware customer in Australia at the Sunshine Coast Council recently produced a video showing his Lab Manager install being used to deploy dozens of virtual machines using 2.6 TB…in less than an hour.  I hope that, like me, you have not become so enamored with the bleeding edge of VMware’s technology to forget how awesome linked clones, resource pools, and Lab Manager can be.

VMware Lab Manager Demo from Phil Helmling on Vimeo.

If you want this kind of flexibility added to your environment look no further than Lab Manager. The Lab Manager architecture is being extended into VMware’s unreleased cloud management software, codenamed Project Redwood. Lab Manager is your private cloud today, using generally available software. If you are not running lab manager now and want to take it for a spin, let me know and I will connect you to a vSpecialist in your town.

It was during discussions with advanced VMware customers in Melbourne that a solution to this problem occurred to me. I have reasoned it through and I think it holds water. I have socialized it with more customers and my colleagues and we think it stands. So I want to introduce a system for implementing virtual machines with a better assurance of a performance SLA.

The key to this process is that minimum performance can be measured using limits and that performance can be assured using reservations. You can develop and document virtual machines with performance SLAs using the following procedure:

• First, as always, define a small number of strictly-sized virtual machines to be used by all applications in your environment. Often these look something like small VMs of 1 vCPU and 4 GB RAM, medium VMs of 2 vCPUs and 8 GB of RAM, and large VMs of 4 vCPUs and 16 GB of RAM. Tune these numbers for your environment, as needed.
• For any application, benchmark its maximum performance against each of these virtual machine configurations on an unloaded system. Chose an ISV-supplied benchmark or a well-known third party tool. This sets your high water mark for throughput for each application in its virtual machine.
• For each configuration, set a CPU limit at 50% of the available CPU and a memory limit of 50% of the available memory. Retest the application against this smaller, limited configuration.
• During the applications’ deployment, change the limits to reservations. That is, remove limits and set reservations equal to the limits’ previous values, in this case 50%.
• Your application now has a maximum performance defined in bullet two, and a “guaranteed” performance measured in bullet three.  This is your application’s performance SLA.

The concept is simple: limits can be used to measure the performance of an application in the presence of that degree of contention. Reservations ensure that those resource amounts are always present. Here are some notes on this process:

• This is not a true guarantee since network and storage throughput may drop. No tool can eliminate this risk entirely but SIOC and NetIOC can reduce the risk of a network- or storage-induced performance failure.
• The memory test is going to be highly dependent on the working set created by your load generation tool. Your mileage will vary depending on your application owners’ use of the virtual machine.
• vCenter will guarantee that the reservations are always available through a process called admission control, which checks the cluster to ensure that enough CPU or memory is available to run the virtual machine immediately and in the event of a server failure.

As I said above, this is not a true guarantee of application performance. But it is as close as we can get until AppSpeed or a replacement evolves into universal application latency measurement that is fed into vCenter. And this is another in a growing list of reasons why CPU and memory reservations should be part of all VMware deployments.

VMmark is clearly the reigning king of consolidation benchmarks and anything that enters its arena must stand against its standard. VMmark pioneered a new method of benchmarking that resonates with virtualization experts. It tests system performance by adding fixed load virtual machines instead of scaling up a single application to system saturation.  Traditional benchmarks tune up their load generation against a single instance but VMmark piles on virtual machines until the system is capable of no more work.

VMmark is one of VMware’s many industry-leading initiatives and was started when VMware worked closely with server vendors that wanted to benchmark their servers’ ability to run virtual machines. VMmark was conceived many years ago, well before VMware had competition. It is because of this fact that I scratch my head at claims that VMmark is biased towards VMware. There was no commercial implementation of Xen when VMmark was specified and Microsoft was only dreaming of entering the market.

But even in an environment devoid of competition, customers want certainty that their benchmarks are not hiding flaws in a product. SPEC has for years been developing honest benchmarks that survive the crucible of debate among its large member community. SPECvirt, or more properly SPECvirt_sc2010, is the result of this vigorous debate. You can read up on SPECvirt in the FAQ released coincident with the product’s launch. But I will add a few comments and comparisons here.

1. SPECvirt costs $3000 to purchase.  VMmark is free.  But VMmark requires commercial software and versions of SPEC benchmarks that are not free.  Depending on your licensing model, you may find VMmark or SPECvirt cheaper.  But the prices of each are essentially comparable.
2. VMmark uses the most common applications in the data center (like Apache and Microsoft Exchange).  SPECvirt does not mandate application choice for the system under test.
   • This is a Good Thing, because you may now choose a configuration that models your environment by running the exact applications you run.
   • This is a Bad Thing, because five different testers may choose five different application sets in their tests resulting in incomparable results.
3. SPECvirt cannot be run against a cluster of hosts.  But VMmark cannot, either.  We will have to wait for an update to one of these benchmarks before we can properly test DRS clusters and their competitive equivalents.
4. There is only one published SPECvirt result, courtesy of IBM running KVM.  There are a boatload of VMmark results, as one would expect of a more mature product.  It will be interesting to watch the rate of submissions of these two benchmarks over the coming year or two.
5. SPECvirt runs three workloads and an idle virtual machine in its tile.  One of those workloads, tested by SPECweb, is implemented with three virtual machines.  The end product is a six-VM tile that looks very much like VMmark’s six-VM tile.

For years we have seen online and in-person griping about VMware’s misunderstood benchmark restriction in its EULA.  Both VMmark and SPECvirt can be run on any supported hypervisor.  So now its time for all the hypervisor vendors to put up or shut up.  Run one of these benchmarks on your product and compare the results against existing published results.  Then the world will know where your product stands.

Calling attention to a subset of the performance features in this launch, I will augment the published documentation with my own comments.

Wide VM NUMA Support

A “wide VM” is defined by VMware as a virtual machine whose memory is too large for a single NUMA node.  In this case, some of the memory must be placed on a remote node, which has a relatively higher memory latency.  ESX 4.0 would place as much memory as possible on a single node, then arbitrarily spill the rest over to other nodes.  ESX 4.1 now recognizes memory locality of reference and places frequently accessed memory on the local node, potentially eliminating remote memory access penalties.  Expect big gains with wide virtual machines running Java or very active databases.

Memory Compression

When I wrote on Steve Herrod’s preview of memory compression at PEX, I am sure you knew this feature’s release was imminent.  VMware’s documentation is sufficient on the gains provided by this feature, so I will not repeat those gains here.  The key thing to remember about memory compression is that it greatly reduces the need for swap.  For years VMware administrators have feared the spectre of memory swapping and have left memory woefully underutilized, even in consolidated environments.  With memory compression in place, you should more confidently push active memory closer to 100%.

Storage IO Control (SIOC)

Before the new VMware documentation, you had my article on SIOC and a delightful video previewing the feature to whet your appetites.  Now that the feature is out, I want to repeat the moral of the story: SIOC will save your high priority applications’ in the event of storage contention.  But if you storage performance stinks before SIOC, it will continue to stink with SIOC.  SIOC just buys you time for your mission critical applications so you can correct that storage problem.

Faster vMotion

(I’ll take a break here and point out the change of spelling from VMotion to vMotion.  This innocuous change will surely be missed by the large numbers of people that misspell VMWare [sic].  In truth, the case of vMotion is not particularly critical, but those of you grammatical pedants like myself take note.)

Many customers, already happy with vMotion, will scratch their heads as to what is left to be improved in this feature.  But a large number of you have tried evacuating 100 virtual machines from a host.  At two virtual machines at a time, this evacuation would have taken tens of minutes.  VMware was not limiting the vMotion concurrency for no good reason; they wanted to guarantee 100% correctness.  Careful evaluation, experimentation, and critical code improvements allowed the vMotion engineering team to greatly improve the efficiency of a migration in vSphere 4.1.  The result is that virtual machines more efficiency use the vMotion network which means VMware can qualify and support more virtual machines being concurrently migrated.

Part of this efficiency change included a decrease in the virtual machine switchover time, during which the application is unresponsive.  In every production environment I have seen, this switchover time was quite small, resulting in no application downtime.  But as processor performance and memory access time improved, and with vMotion efficiency remaining flat, eventually pages would be touched faster than vMotion could migrate them. This would result in vMotion failures.

The new vMotion efficiency improvements have dropped application switchover times to minuscule levels, guaranteeing zero application downtime for many years to come.

Network IO Control

Missing from the recently published performance document is an overview on Network IO Control (NetIOC).  In truth, I may be responsible for its lack of inclusion.  Apologies.  But luckily performance engineering released a best practices document on this wonderful new feature.

NetIOC is the network version of SIOC and may be even more important than SiOC in 10 Gb network environments and infrastructure using converged network adapters.  Let us be honest: the best practice of giving dedicated network hardware to each vSphere network traffic stream is so 2007.  It’s time to consolidate network and put everything on fewer 10 Gb adapters.  But this is going to create occasional network contention that would benefit from the same resource prioritization that CPU and memory shares have provided for years.

NetIOC will help prioritize your network streams in such an environment. Converge your networks and investigate NetIOC.

So, what exactly are we looking for? We want technical experts to work with myself and regional pre-sales resources to help close VCE-related deals. This means an ideal candidate will know all three of these technologies. But, in truth, VMware skills are most direly needed now. Time exists to ramp up on Cisco and EMC technologies.

We want people that love technology. We want guys and girls that are enthusiastic, and have had their coworkers telling them this forever. We want people that show customers a new product or feature with the excitement of a child handling a new toy. We want people that build out home labs with a software infrastructure that could support a medium sized business.

What follows is the long (or thorough, if you’re an optimist) description we have been using internally to describe what we are looking for.

Position Scope: This is an exciting new initiative to assist in the design and build of next generation data center architectures for EMC’s largest clients who are looking for consolidation, virtualization and automation of their complex data center functions. These solutions will be built by a collaborative team, the Virtual Computing Environment (VCE), which includes VMware, Cisco, and EMC. EMC is looking for senior IT and Data Center experienced individuals who have designed, built or successfully sold virtual infrastructures to large scale data centers. Strong consultancy skills are a must for this position along with the ability to develop low level repeatable services methodologies.

The VCE Technical Consultant will be responsible for technical decisions that will affect the successful sale of Vblock systems, project timelines, functional unit and/or deadlines. He or she will be viewed as a technical resource by peers within EMC, and will partner with customers, field sales & pre-sales, and the VCE coalition in a team setting, taking ownership of their objectives to achieve business results. This role evaluates and sells alternative solutions, effectively influencing outcomes and uses acquired professional knowledge to determine method for issue resolution. The Technical Consultant exercises judgment within defined standards of the industry and uses expertise and creativity for innovative product recommendation and solutions and develops clear low level services methodology for the data center solutions.

The VCE Technical consultant will serve as an evangelist for the VCE coalition within region. This means developing presentations, delivering demonstrations, and leading interactive workshops aimed at increasing the knowledge of VCE technical capabilities within VMware, Cisco, EMC, and their customers.

Position Responsibilities: The VCE Technical Consultant will report directly into the VCE Solutions Team with primary responsibility for increasing EMC and Cisco sales in VMware environments. The Technical Consultant’s primary role is positioning, architecting and delivering VCE solutions around, but not limited to, EMC’s Vblock technology and specifically integrating those technologies into the customer’s overall datacenter architecture and detailed design. This position will help drive sales, serve as content knowledge expert, participate in proposal preparation, assist in the refinement of detailed services methodologies, deliver customer presentations, mentor other technical consultants and assist with development of technical collateral around server and application integrated designs. Since this is a high profile position working with EMC’s key customers across the country, some travel will be required.

Key responsibilities include:

• Complimenting existing EMC pre-sales resources with skills and experience outside that of traditional EMC technologies like VMware virtualization.
• Helping influence our customers’ decisions around choice and implementation of core datacenter technologies from VMware, Cisco, and EMC.
• Playing a key role in EMC securing the IT architecture in our customers’ datacenters.
• Helping influence the decisions of EMC Channel partners around the technical merits of combining VMware, Cisco, and EMC technologies for their customers.
• Working closely with VMware, Cisco, and EMC sales, pre-sales and Services teams on the top VCE opportunities in their geography.
• Fostering strong VMware, Cisco and EMC partnerships through constant communication, joint training, and transparent business operations.
• Leading technical discussions and training on VMware and Cisco technologies to improve the VMware, Cisco, and EMC field’s understanding of those products.
• Evangelizing the thought leadership and technical depth of the EMC VCE team in the regional market.

Functional/Cross Functional: Interfaces cross-functionally and leads collaboration, extending beyond immediate VCE team. Customer interaction requires excellent verbal and professional presentation skills. The Technical Consultant should have the ability to represent VCE by communicating technical and business capabilities and strategies externally.

Experience/Skills: The ideal candidate will have expertise with VMware virtualization including the installation, configuration, deployment and administration of ESX Server, vCenter Server, Site Recovery Manager, CapacityIQ, Lab Manager, and others. VCP certification or commensurate level product understanding is a requirement. Candidates that have completed work towards a VCAP or VCDX are highly desired.

The candidate should have experience deploying and maintaining servers, applications, network and storage, and should have an industry wide understanding of datacenter infrastructure applications (operating systems, databases, replication/disaster recovery, and data protection), expertise with server operating systems, databases, business applications, storage infrastructure, and application integration with respect to infrastructure.

In addition, experience with developing business cases and ROI, understanding of datacenter physical requirements like power, cooling, cabling etc., operations experience and working knowledge in implementing management and monitoring infrastructure for the datacenter are a plus. Strong consulting skills are a requirement for this position as customer presentations of services and solutions will need to be performed in a daily basis. Knowledge and experience of virtualization is also a requirement.

As such we are looking for the successful candidate to take responsibility for positioning and delivering Data center Solutions including server virtualization, storage consolidation, data protection, networking, security and management & automation. The successful candidate needs to have a track record of successful executive interaction, consultative selling and leading the delivery of successful consulting engagements in this area.

If you think this describes you, and you want to be a vSpecialist in the Asia Pacific region, drop me a note and we’ll take it from there.

In many of the environments I have diagnosed, I have traced the problem to a poor relationship. VI admins lack basic storage skills and storage admins are supplying LUNs via email or web requests, not interactive design sessions. I offer one customer–protected through anonymity–whose story showed a failure of the storage/VI relationship.

It was a couple of years ago that this customer asked for my recommendation on extents.  I told him there exist no performance scalability concerns with extents, but ailing LUN diagnosis can be difficult.  He said extents were a requirement in his environment because the storage admin would only provide him standard, preconfigured 20 GB LUNs.  If he needed larger volumes, the storage admin insisted he aggregate in software (RAID, LVM, extents, etc.)  I immediately knew this lack of cooperation would doom them to failure.  Would it surprise you to hear that I heard from this customer many more times as problems were escalated to me?

It occurs to me that three things will decrease the storage mistakes that get blamed on VMware:

1. Regular meetings with people from VMware and EMC so everyone understands these problems, can identify them, and can help each other work through them.
2. Good VMware tools to help VI administrators recognize storage bottlenecks so they go to their storage team before going to VMware.
3. An increase in VMware administrators’ view and control of storage so they become partners in storage decisions and not nameless, voiceless customers.

The good news is that solutions are present or imminent:

(*) Demonstrated by VMware but not announced or committed to a release.
(**) Not demonstrated ever but we can dream, right?

OK, team. I know I have been preaching to the choir for years about fixing these performance problems. It is now time for some preventative maintenance. Storage vendors, help VMware by educating their customers on how to diagnose and correct storage problems. Customers, install the vCenter plugins from your storage vendor and be sure you understand what you are looking at. VMware, get your new features out.

OK, everyone put your hands in the circle. Shall we do this? OK, break!

Sanjay was talking about the allocation of resources to IT projects.  His whiteboard figure showed one stack where project resources were in silos dedicated to the lines of business (LOBs).  In this model resources are not commonly shared across multiple projects. Here is a version of the figure Sanjay drew on the whiteboard.

Hardware and people are dedicated to the needs of the LOBs. There is little or no sharing between projects.

Most of you are thinking it ludicrous that any company today would fully separate infrastructure and people this way. Surely storage and network sharing is common across a company, right? I can tell you that in more companies than you think there are still rigid lines of separation between different business units. The most common reason for this is, “our engineers require special hardware and support that corporate IT cannot provide so we give the engineers their own IT.”

EMC includes in its vision of the journey to the private cloud a step one that shows the consolidation of network and storage. Most companies today place most storage on a shared infrastructure. And when VMware is first deployed, servers are quickly consolidated to reduce redundancy and improve utilization. I see that most of VMware’s customers show the following silos and shared infrastructure.

In phase one of the VMware journey, IT production, customers are consolidating servers that run IT's applications. The application owners are still resisting consolidation and are given customized applications on dedicated hardware.

The beauty of virtualization is that not only can the physical resources be shared, as any VMware demonstration will prove, but the people that support the infrastructure can be shared, too.  This concept is already understood by VMware’s more mature customers, who have been telling VMware for years that virtualization can save more money in operational expenses than capital expenses.  These savings are coming after thinning the ranks of dedicated infrastructure specialists and refocusing them on higher value opportunities.

But after hardware has been consolidated and is shared across LOBs, how is it possible to do a similar consolidation with people, the most expensive part of the higher levels of the stack? Sanjay says the answer is through application rationalization. Applications must go into the private cloud and they must be shared by common resources. This means applications must be chosen to:

• Fulfill the greatest number of requirements for the greatest number of LOBs.
• Be supportable in a virtual environment.
• Run on a small number of OS builds also selected for their supportability in the cloud.

Success in this space greatly reduces the aggregate size of application support teams and can reduce software licenses through the power of large purchase discounts.  The result is further consolidation.

At this phase a smaller number of OS images are supported and application versions and vendors are limited. But the support structure is greatly decreased as a result.

So, through VMware’s virtual machines, it is easier, cheaper, and faster to deploy infrastructure to the LOB.  But the system retains one bottleneck: the interface between the LOBs and the virtual infrastructure.  Speaking as a performance person, this interface is an obvious source of latency and inefficiency.  Any person at this interface will simply stand in the way of virtual machine deployment.  If there are too many people in this role there is waste.  If there are too few, the interface presents a bottleneck to throughput.  The solution is to allow the LOBs to self-service.

Enter the private cloud.  Building a private cloud where the LOBs can self-service removes the last bottleneck and eliminates more dedicated resources (people).  The end product of a private cloud deployment is shown here.

Everything is shared in a private cloud deployment: hardware and people. Sharing increases the utilization and efficiency of both.

In my opinion the most important part of the private cloud is one of the least talked about: a transparent system of showing LOBs the cost of their use of the infrastructure.  With a proper chargeback system, with a fully transparent model, LOBs can understand what portion of the hardware and people they are consuming.  And by exposing pricing, IT organizations around the world enter the free market and show their ability to compete with public cloud providers.

VMware vCenter Chargeback is as important to the cloud as any end-user portal.  It is with its correct usage that owners of a VMware private cloud will either blossom or wither.  My customer discussions of late are starting to convince me that proper chargeback is not just a step in the cloud, but a key requirement.