The InfiniBand architecture defined by IBTA includes Quality of Service and Congestion Control features that are tailored perfectly to the needs of Data Center Networks. Through use of features available today in InfiniBand Architecture based hardware devices, InfiniBand can address QoS requirements in Data Center applications in an efficient and simpler way, better than any other interconnect technology available today.

When starting to play around InfiniBand QoS, you will need to understand two basic concepts.

Service levels (SLs) and Virtual Lanes (VLs).

Basic Concepts

Service Level (SL)
A field in the Local Routing Header (LRH) which defines the requested QoS. It permits a packet to operate at one of 16 SLs. Assignment of SLs is a function of each node’s communication manager (CM) and its negotiation with the SM.

Open

Virtual Lane (VL)

Virtual lanes provide a mechanism for creating multiple channels within a single physical link.

• Each VL uses different buffer

• In the standard there are 16 possible VLs.

  • VLs 0-14 –used for traffic (0-7 are the only VLs that are currently implemented by Mellanox switches)

  • VL 15 –used for management traffic only

Open

SL to VL mapping allows QoS implementation.

OpenSM Parameters

OpenSM configuration file has a set of QoS parameters, that are used to configure SL2VL mapping and VL arbitration.

• qos - admin mode for enabling/disabling QoS: True/False (default false)

Default Global parameters

• qos_max_vls - the maximum number of VLs on the subnet

• qos_high_limit - The limit of high priority component of VL arbitration table. High-limit indicates the number of high-priority packets that can be transmitted without an opportunity to send a low-priority packet. A value of 255 indicates that the limit is unbounded. If the 255 value is used, the low priority VLs may be starved. A value of 0 indicates that only a single packet from the high-priority table may be sent before an opportunity is given to the low-priority table, which basically means that there is no real high priority VL. If the high-priority VLs do not have an available packet for transmission, a packet may be sent from the low-priority VLs.

• qos_vlarb_high - High priority VL arbitration table. This is a list of VL/weight pairs, where each list entry contains:

  • VL number (range 0-14)

  • Weight value (range 0-255) – the number of 64-byte units which may be transmitted from that VL when its turn in the arbitration occurs. A weight of 0 indicates that this entry should be skipped.

• qos_vlarb_low - Low priority VL arbitration table. This is a list of VL/weight pairs, where each list entry contains:

  • VL number (range 0-14)

  • Weight value (range 0-255) – the number of 64-byte units which may be transmitted from that VL when its turn in the arbitration occurs. A weight of 0 indicates that this entry should be skipped.

• qos_sl2vl - SL2VL Mapping table: It is a list of VLs corresponding to SLs 0-15.

  • Position is the SL number

  • Numeric value is the VL number
    
    

For example:

• qos_max_vls 2 - We will have 2 VLs on the subnet.

• qos_high_limit 255 - We have strict priority example. High priority VLs will always be sent before low priority VLs.

• qos_vlarb_high 1:192, 2:128, 3:64 - means that there are 3 high priority VLs while the arbiter will perform weighted round robin among them with the wight of 192, 128, 64 per SLs 1,2 and 3.

• qos_vlarb_low 0:1 - low priority SL 0 can send one 64*64 byte (=4096 byte) unit and after that check if there other high priority SLs.

Note: default InfiniBand MTU is 4KB. A single packet of MTU size, will require 64 units of 64 bytes. weight values for each VL should be multiples of 64.

• qos_sl2vl 0,1,2,3 - 4 SLs are mapped to 4 VLs. you can map more than one SL to the same VL.