The general definition of latency is the delay between a stimulus and a response. From a data storage perspective, it’s the time it takes for a data packet to travel from the initiator within the primary server to the target device (initiator/target). Simply put, it’s nothing more than “delay.” Excessive latency can be particularly threatening to a storage networking environment because it creates a bottleneck that causes further delays or prevents data from reaching its final destination. When this occurs, additional packets must be sent, creating even more delays and ultimately network congestion. In today’s data storage environments, latency is more important than ever. Connections to low latency storage such as SSDs and hybrid arrays are on the increase, VM densities are on the rise and attempts to keep up with ever-increasing SLAs are a growing challenge.
Three components—distance, equipment, and protocols—can each contribute to the overall latency of a storage network. Most latency comes from the likes of routers, repeaters and devices that individually examine packets for routing information over larger distances. Other latency factors have to do with the architectural implementation of the individual switch. Our friends at Brocade address some of these differences in this blog. However, for this conversation, let’s focus on protocols and network latencies as they compare within different storage protocols: Fibre Channel, iSCSI and 10GbE Fibre Channel over Ethernet (FCoE) networks.
For the last decade Fibre Channel has been the primary choice for the majority of enterprise storage area network (SAN) infrastructures. Specifically, Fibre Channel has been the default SAN technology for many medium-to-large enterprise data centers running business-critical applications, with high bandwidth, high transaction and low latency requirements.
Despite the great success of Fibre Channel over the last ten years, iSCSI has been slowly gaining traction and has begun to carve out a niche market in small-to-medium-sized businesses. A primary reason for this is iSCSI can be run over existing Ethernet networks, commodity 1Gb Ethernet NICs or LAN on Motherboards (LOMs). Although significantly slower than Fibre Channel, iSCSI has filled the needs of many SMB environments because it is falsely perceived as a technology that lowers the cost-of-entry by utilizing a familiar interface. (Have you ever asked a network administrator about LUN provisioning? This is a conversation for another blog.) With the recent introduction of 10GbE speeds, iSCSI is now being considered for evaluation or deployment in enterprise environments. However, iSCSI imposes additional overhead, and even at 10Gb speeds, has higher latency than 8Gb Fibre Channel.
While there is no denying the success of Fibre Channel and iSCSI, FCoE is a different story altogether. While FCoE is well-suited for supporting convergence between LAN and SAN, it has yet to gain the popularity it deserves. Both iSCSI and FCoE utilize the TCP/IP protocol, which is typically processed by the host CPU, and is very inefficient compared to the Fibre Channel protocol. This added overhead of the IP layer creates inefficiencies that appear as lower bandwidth, lower IOPS, higher latency and increased CPU utilization. While offloading this overhead from the CPU onto an adapter can have some benefits, the extraordinary overhead of 10GbE (10GbE iSCSI and FCoE) is not fully addressed by today’s hardware and software TCP offload engine (TOE) initiatives. So you can see the viability of iSCSI or FCoE displacing Fibre Channel starts to deteriorate.
To improve latency, Fibre Channel was designed to use the Layer 2 network protocol. This design takes much less CPU to encapsulate a payload than is required by IP networks. By encapsulating data in Ethernet frames instead of IP packets you have less overhead. By comparison a TCP/IP frame adds 80 odd bytes per packet, which is lost bandwidth, and a 10GbE connection will deliver roughly only nine or so gigabits-per-second of storage bandwidth. Native 16Gb Gen 5 Fibre Channel is 40 percent faster than a 10GbE network.
Another feature that reduces latencies for Fibre Channel is the use of buffer-to-buffer credits. With buffer-to-buffer credits, you never stop the transfer of data; it simply slows down as the buffer fills. By comparison, FCoE uses TCP for flow control of iSCSI and Priority Flow Control (PFC) for lossless Ethernet. With TCP and PFC flow control, when the target starts running out of buffers, it has to send a message to stop all transmissions from the sender and all packets in transit are lost. Stopping transmission and lost packets are not acceptable for most SAN applications and halting the transfer of information to sequential access storage devices such as tape drives can be detrimental to the archival process.
One thing is certain. Fibre Channel isn’t going away anytime soon. Research shows that deployment is at an all-time high. Recent QLogic and ESG surveys tell us that enterprises rely on Fibre Channel above all other technologies for valuable SAN services, such as reliability and performance. It’s important to note that SANs and LANs are typically the responsibility of different teams within these enterprises. Through my own discussions with end-users, I’ve learned that storage administrators are not willing to entrust production storage traffic to be run over a shared network. Storage administrators tend to trust what they know and distrust what they don't. If you’ve run business-critical information over a Fibre Channel SAN, you’re not likely to trust your data to a slower, less reliable architecture.
The majority of storage administrators still depend upon 8Gb Fibre Channel, with 16Gb Gen 5 Fibre Channel rapidly overtaking new deployments. Data in a recent report from the Dell’Oro Group indicates that Fibre Channel is alive and well. The report revealed that Fibre Channel revenue from directors, switches and adapters surpassed $2.8 billion in 2012 and just four months ago, Brocade announced a record quarter for Fibre Channel switch revenue. Although there’s no way to escape the fundamental laws of physics that govern latency, one can minimize its effects over a Fibre Channel storage network. Fibre Channel is a purpose-built, data center-proven network infrastructure for storage, and it’s the most functional protocol for efficiently transferring block-level storage. More than just a “speed,” Fibre Channel provides back-end network capabilities such as LAN-less backup and high speed data migration, and it efficiently handles the high percentage of random I/O typical in highly virtualized environments. Let’s not forget it also delivers unmatched reliability, simplicity and of course low-latency performance. In spite of all the propaganda, Fibre Channel isn’t going away any time soon.