In our previous two blogs we explored the foundation of the feature responsible for making customers happy and more profitable by ensuring their customers applications perform as expected, while also allowing precious server resources to be utilized to their maximum potential. We describe how this is possible using the QLogic Switch Independent NIC Partitioning (QLogic NPAR) feature available on the 3400 and 8400 series of networking adapters. We discussed at a high level how this feature makes 10GbE bandwidth more manageable and relevant to applications that may not always need 10Gbps of network bandwidth, but may need it occasionally. We walked through an example configuration using QLogic NPAR to demonstrate how dynamic Maximum Bandwidth and Relative Bandwidth Weight parameters can benefit customer applications. In this blog we will compare the QLogic NPAR algorithm to the Linux based SR-IOV Rate Limiting method of statically limiting bandwidth on a per Virtual Function (VF) basis.
How does QLogic NPAR QoS compare to Linux based SR-IOV Rate Limiting?
The first thing to remember is that QLogic NPAR QoS is Operating System independent, so you can have the same level of dynamic QoS if you are running a QLogic 3400 or 8400 Series adapter on Linux, Windows, VMware or any other supported OS. In contrast, SR-IOV based QoS is dependent on the OS that implements it.
SR-IOV Virtual Functions
QLogic NPAR creates multiple PCIe devices called Physical Functions (PFs) that behave like physical adapter ports. There is no SR-IOV algorithm required to support QoS using a QLogic NPAR PF. The operating system is not involved. This is contrasted with an SR-IOV Virtual Function that is dependent on the Operating System to provide tools specific to that operating system to create the Virtual Functions. SR-IOV does offer customers many great features and actually can compliment QLogic NPAR. For example, using QLogic NPAR on a 10GbE port could result in 4 PFs being created. Now each PF will behave just like a separate Ethernet port which means that each PF can support multiple SR-IOV based Virtual Functions. This is what QLogic refers to as concurrent NPAR and SR-IOV functionality. This further demonstrates how a QLogic NPAR PF is really like a physical port. When a VF is assigned to a PF, the PF is the "parent" and the bandwidth available to all VFs will be determined based on how the PF has been configured.
SR-IOV VFs can each be statically rate limited. This means you could map an application to a VF and ensure that that application never gets more than 6Gbps of bandwidth. The problem with this method of Rate Limiting is that the setting is static. If an application that is using that particular VF needs more bandwidth than the current rate limited value, then the setting will need to be changed again. As we described earlier, QLogic NPAR will provide as much or as little bandwidth to PFs as you desire. This is achieved by giving a user two parameters - Maximum Bandwidth (the maximum amount of bandwidth a PF can ever consume) and a Relative Bandwidth Weight (minimum amount of bandwidth a PF can get when there is bandwidth contention with other PFs).
Applying Linux SR-IOV Rate Limits to Virtual Functions
If we take our example with four applications, and now apply it to Virtual Functions in SR-IOV: Application 1 must have access to a maximum of 6Gbps, Application 2 gets 2Gbps and Applications 3 and 4 can have a maximum of 1Gbps each.
In Linux the following four commands would achieve this when SR-IOV is enabled in Linux and the adapter is configured for SR-IOV:
- #ip link set eth3 vf 1 rate 6000
- #ip link set eth3 vf 2 rate 2000
- #ip link set eth3 vf 3 rate 1000
- #ip link set eth3 vf 4 rate 1000
These commands have now set static rate limits on each VF. It is a static setting because the only way to allow a given VF to have more bandwidth based on changing network traffic patterns is to re-execute the command with a different rate parameter.
QLogic NPAR allows for dynamic bandwidth allocation and SR-IOV Rate Limiting is a static configuration requiring user intervention and continuous monitoring by network administrators - time administrators do not have. Using an SR-IOV-only based QoS method would prevent a hungry backup application, from having access to available bandwidth when the network is not congested; available bandwidth would go unutilized and an application would take longer to finish its work. In Part 4 of this series we compare QLogic NPAR to a basic Round-Robin QoS method that is also dependent on SR-IOV, but doesn't utilize Rate Limits.
For More Information
Please see these links for more information on these innovative 10GbE Intelligent Ethernet and Converged Network adapters. An excellent use case discussion for VMware and QLogic NPAR can be found in our Solution Sheet: Switch Independent NIC Partitioning and VMware Virtual Environments. A handy list of Frequently Asked Questions is detailed in our Switch Independent NIC Partitioning document.
Blog Series Links