Skip to end of metadata
Go to start of metadata

You are viewing an old version of this content. View the current version.

Compare with Current View Version History

« Previous Version 2 Current »

When starting to play around InfiniBand QoS, you will need to understand two basic concepts.

Service levels (SLs) and Virtual Lanes (VLs).

Basic Concepts

Service Level (SL)
A field in the Local Routing Header (LRH) which defines the requested QoS. It permits a packet to operate at one of 16 SLs. Assignment of SLs is a function of each node’s communication manager (CM) and its negotiation with the SM.

Virtual Lane (VL)

Virtual lanes provide a mechanism for creating multiple channels within a single physical link.

  • Each VL uses different buffer

  • In the standard there are 16 possible VLs.

    • VLs 0-14 –used for traffic (0-7 are the only VLs that are currently implemented by Mellanox switches)

    • VL 15 –used for management traffic only

SL to VL mapping allows QoS implementation.

OpenSM Parameters

OpenSM configuration file has a set of QoS parameters, that are used to configure SL2VL mapping and VL arbitration.

  • qos - admin mode for enabling/disabling QoS: True/False (default false)

# Enable QoS setup
qos TRUE

Default Global parameters

  • qos_max_vls - the maximum number of VLs on the subnet

  • qos_high_limit - The limit of high priority component of VL arbitration table. High-limit indicates the number of high-priority packets that can be transmitted without an opportunity to send a low-priority packet. A value of 255 indicates that the limit is unbounded. If the 255 value is used, the low priority VLs may be starved. A value of 0 indicates that only a single packet from the high-priority table may be sent before an opportunity is given to the low-priority table, which basically means that there is no real high priority VL. If the high-priority VLs do not have an available packet for transmission, a packet may be sent from the low-priority VLs.

  • qos_vlarb_high - High priority VL arbitration table. This is a list of VL/weight pairs, where each list entry contains:

    • VL number (range 0-14)

    • Weight value (range 0-255) – the number of 64-byte units which may be transmitted from that VL when its turn in the arbitration occurs. A weight of 0 indicates that this entry should be skipped.

  • qos_vlarb_low - Low priority VL arbitration table. This is a list of VL/weight pairs, where each list entry contains:

    • VL number (range 0-14)

    • Weight value (range 0-255) – the number of 64-byte units which may be transmitted from that VL when its turn in the arbitration occurs. A weight of 0 indicates that this entry should be skipped.

  • qos_sl2vl - SL2VL Mapping table: It is a list of VLs corresponding to SLs 0-15.

    • Position is the SL number

    • Numeric value is the VL number

For example:

  • qos_max_vls 2 - We will have 2 VLs on the subnet.

  • qos_high_limit 255 - We have strict priority example. High priority VLs will always be sent before low priority VLs.

  • qos_vlarb_high 1:192, 2:128, 3:64 - means that there are 3 high priority VLs while the arbiter will perform weighted round robin among them with the wight of 192, 128, 64 per SLs 1,2 and 3.

  • qos_vlarb_low 0:1 - low priority SL 0 can send one 64*64 byte (=4096 byte) unit and after that check if there other high priority SLs.

Note: default InfiniBand MTU is 4KB. A single packet of MTU size, will require 64 units of 64 bytes. weight values for each VL should be multiples of 64.

  • qos_sl2vl 0,1,2,3 - 4 SLs are mapped to 4 VLs. you can map more than one SL to the same VL.

# QoS default options
qos_max_vls 4
qos_high_limit 255
qos_vlarb_high 1:192, 2:128, 3:64
qos_vlarb_low 0:64
qos_sl2vl 0,1,2,3

  • No labels