vPivot

Scott Drummonds on Virtualization

Protocol Comparison: Block Versus File

1 Comment »

Customers have asked me to recommend a protocol for their vSphere environments more times than I can remember.  The best answer to this question is “stick with what you know”.  By far staying with an existing infrastructure is the best solution.  This leverages your existing skills, minimizes risk, and keeps costs down.  And no protocol can on its own claim to be the undisputed best choice.

But choosing between protocols does imply some design differences, limitations or benefits.  In this article I want to collect some of these items for your consideration.  As I asked my friends and colleagues about this subject I realized no one person could completely enumerate the protocol choice implications.  So, add your comments to the bottom and we will continue to update this article as a living document.

Lastly, this is a comparison of the differences between block and file in a VMware environments.  Differences in physical will not be discussed.  A comparison of different block protocols is left as an exercise to the reader.

Throughput

Throughput is usually limited by the backend storage configuration, not the protocol.  Only when an array can serve data very fast is throughput gated by the link speed.  Obviously 10 Gb is faster than 8 Gb which is faster than 1 Gb.  But here multipathing across many links gives block an advantage.  vSphere’s use of an NFS v3 client makes load balancing tricky, at best.  But multiple block paths can be simply aggregated to deliver a higher maximum throughput than a single Ethernet link.

Latency

Block is nominally faster.  I recall seeing this when I still worked at VMware but I cannot find supporting data now.  My memory is confirmed with a quote from a very old VMware paper: “The data also demonstrates [sic] that…Fibre Channel has the best performance in throughput, latency, and CPU efficiency.”

CPU efficiency

Block can be more efficient but that is not inherent to its “blockiness”.  Protocols managed an HBA use less CPU.  But NFS is more cycle-efficient than SW iSCSI.  Fibre Channel is better than HW iSCSI which is better than NFS which is better than SW iSCSI.  Check page 6 of VMware’s aging vSphere 4 protocol comparison for details.

Simplicity

This is obviously subjective. But can anyone argue that NFS is the king of simplicity?

Robustness/Availability

From vSphere’s perspective block must be considered more available.  vSphere provides no multipathing options for file storage.  Physical design of both Fibre Channel and Ethernet can be improve availability. But regardless of the physical design, vSphere layers more availability options with block’s multipathing options.

Cost

The answer here is tricky.  Ethernet is the cheapest storage interconnect and there are three protocols, two block and one file, that can use it.  But I feel the CPU cost per IO of iSCI to make it prohibitive with mission critical applications.  And Fibre Channel over Ethernet (FCoE) may introduce new requirements besides a cheap NIC and switch that drive up its cost.  So, given its undeniable simplicity and use of any network hardware I will award file the title of cheapest protocol.

Summary

If you look at the above list you’ll see that throughput, latency, and CPU efficiency are really three facets of the same gem: performance.  If we consolidate those three into a single performance field and crown block the winner, then the final score is two for file and two for block.  But because simplicity is so important in early VMware deployments I always recommend file storage for new virtualization deployments.

But to take this article full circle, the operational competency of your VMware and storage admins is much more important than any slight lead in the above list.  Stick with what you know.  When you get really great at both protocols, use block for your mission critical applications and file for the vast majority of your virtual machines.

One Response

NFS can easily be designed to offer throughput across multiple 1GbE links. With 10GbE there’s really not the need to aggregate links anymore.