I have received questions about guest defragmentation tools for years. Until today I could only pose theories as to the value of guest defragmentation. But previous theories spawned new research and one of VMware’s partners is now putting data behind their argument that file systems in Windows virtual machines require defragmentation. This partner, Raxco Software, shared early results of this investigation with me. Raxco used their NTFS defragmentation tool PerfectDisk to evaluate the impact of guest defragmentation on a single virtual machine.
Before I describe the test and its results, I want to share an important point on guest defragmentation. Most of the computer literate are aware that file fragmentation–the separation of logically contiguous pieces of a file–can hurt storage performance. But many may not realize that free space fragmentation is as big of an issue. When free space is fragmented, writes take longer and files are re-fragmented rapidly. PerfectDisk defragments files and free space and the results below benefit from both of these improvements.
The following steps were used to setup the test environment:
- A new virtual machine was constructed with Windows Server 2008 on a single 50 GB virtual disk. 29 GB (58%) of the disk was populated with he OS and miscellaneous user files. 21 GB of free space remained.
- They ran a tool that simulated months of user activity by reordering blocks to fragment files and free space.
- The fragmented virtual machine was cloned.
- PerfectDisk was run on the second virtual machine to produce a defragmented virtual disk.
The Raxco team then compared the performance of the fragmented virtual machine with the defragmented one by measuring the installation time of Microsoft SQL Server 2008. This workload was chosen because it produces a bounded, write-intensive test that can easily be monitored with vscsiStats. Only one virtual machine was running on the host.
Let’s take a look at a few key graphs.
This first histogram shows IO counts by size. You can see that IO counts for all but the largest bucket have decreased because Perfect disk is reordering files to minimize small IOs and maximize large IOs. For the storage hardware this means greater efficiency in processing IOs. But it also means two things to ESX:
- More host throughput as the fixed HBA queue is now holding larger commands.
- Fewer virtual storage stack traversals resulting is lower CPU utilization.
However, PerfectDisk not only consolidates small IOs into large IOs, it also makes files logically contiguous as they are seen by the NTFS file system. This means less work for disk controllers when mapping logical to physical clusters on disk.
Next we have the vscsiStats seek distance histogram which shows the shift from random to sequential IO.
The seek distance histogram shows a clear increase in the number of IOs that were exactly one block after the previous IO. This pattern, demonstrated in the bucket “1” increase, reflects the increased sequential nature of accesses to the defragmented virtual disk. In this case the fragmented disk access was 15% sequential while the defragmented disk was 30% sequential.
Let us next look at latency.
The latency histogram shows a decrease in all IOs across the board and a near elimination of IOs that took more than 30,000 microseconds (30 ms). Those very slow IOs accounted for 15% of all the commands in the fragmented case. By eliminating the 15% slowest IOs, you can imagine that the total IO performance and application execution time would greatly improve. That is exactly what happened, as the following data show:
|Metric||Fragmented Disk||Defragmented Disk||Comment|
|Total IOs||166412||105620||A decrease in total IOs is a result of Windows making fewer requests for larger IOs in the defragmented case.|
|Mean IO Response Time||58.5 ms||3.5 ms||The best application metric for this test showed a 33% decrease in execution time.|
|SQL Server 2008 Install Time||45 minutes||30 minutes||The best application metric for this test showed a 33% reduction in execution time.|
Let me repeat one of those amazing data points: the average IO latency dropped from about 55 ms to less than 4 ms. While this is a phenomenal number, the increase depends on characteristics of the storage system. Since these improvements are configuration dependent, your results may vary considerably.
As Raxco continues its investigation I remain cautiously optimistic of the value of guest defragmentation. I think the exchange of small IO for large IO is indisputably a Good Thing. However, virtual environments are very complex and I harbor some concerns about guest defragmentation in virtual environments that must be considered. For instance:
- Defragmentation in your virtual machines backed by linked clones may explode those VMs’ VMDKs’ consumption of their VMFS volumes.
- The value of increased sequential access in a single virtual machine will decrease some in consolidated environments. This is because multiple virtual machines’ sequential access gets interleaved at the array, increasing the randomness of the IO from the array’s point of view.
- Guest block reordering may have negative consequences to your array, as Vaughn Stewart argued in a comment to my first entry on the subject.
- The value of file defragmentation may be limited when applications produce small random block access to data files, as some databases tend to do.
Raxco is continuing to investigate guest defragmentation to respond to some of these concerns and provide best practices for PerfectDisk’s usage. I will update you as the research continues.
ESX Server Configuration
- ESX Version: 3.5.0 Update 1
- Motherboard: Intel S5000PSL
- CPU Type: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
- Number of CPUs: 2
- Cores per CPU: 4
- Logical Processors: 8
- Memory: 4 GB
- RAID controller: Adaptec RAID 3805
- Number of Drives: 4
- Drive Type: WD1001FALS 1TB 7200 RPM 32MB Cache
- Total Capacity: 4.0 TB
- Number of LUNS: 2
- LUN 1 RAID level: 5
- LUN 1 Capacity: 2.00 TB
- LUN 1 Partitions: 1
- LUN 1 Name: IOTesting
- LUN 2 RAID level: 5
- LUN 2 Capacity: 744.75 GB
- LUN 2 Partitions: 1
- Number of Datastores: 2
- Datastore 1 Name: IOTesting
- Number of VMs: 2
- Capacity: 2.00 TB
- Target LUN: LUN 1 (from Storage configuration)
- Datastore 2 Name: ISO
- Number of VMs: 0
- Capacity: 744.75 GB
- Target LUN: LUN 2 (from Storage configuration above)
- Number of VMs: 2
- Operating System: Windows Server 2008 R2 (64-bit)
- Memory: 2GB
- Number of CPUs: 2
- SCSI Controller: LSI Logic (no SCSI bus sharing)
- Number of Disks: 1
- Size of Disk: 50 GB
- Provisioning Type: Thick
- Backing Datastore: IOTesting
- Virtual Memory: none (pagefile disabled)
- Network: disabled