...
| Code Block |
|---|
routing_engine ar_ftree |
In addition, set the root_guid config file
For example: set in opensm.conf:
| Code Block |
|---|
root_guid_file /etc/opensm/root_guid.cfg |
Create a file:
| Code Block |
|---|
# cat root_guid.cfg
0xb8599f0300fcca6c #thor-qm8700-S1
0xb8599f0300df8faa #thor-qm8790-S2
0xb8599f0300fcc2cc #rome-qm8700-S1 |
Enable AR per SL
By default all SLs are enabled with AR is enabled. To set AR enabled per SL, set the following bitmap parameter:
...
| Code Block |
|---|
ar_sl_mask 0xFFFA # A=0x1010 SL1, SL3 enabled with AR |
UCX Support
FW 12.29.0356 added support for ooo_sl_mask per vport that’s now used by UCX.
With that, UCX has a way to request AR support. It could be done using UCX_IB_AR_ENABLE={ yes | no | try}. The default value is “try”.
UCX_IB_AR_ENABLE=yes strictly selects SL with AR, if there no SLs with AR, it fails UCX
UCX_IB_AR_ENABLE=try selects SL with AR, if there no SLs with AR, it selects the first available (or specified by UCX_IB_SL=<sl>)
UCX_IB_AR_ENABLE=no strictly selects SL without AR, if there no SLs without AR, it fails UCX
Specific SL still could be specified by user through UCX_IB_SL=[0..15]
If the UCX_IB_AR_ENABLE=yes/no was asked on old FW, it fails UCX since UCX is not able to detect ooo_sl_mask.
The feature is part of the UCX v1.10 release.
Validation
smparquery
Check that AR is enabled on a switch (using the switch lid), us the smparquery (supported on OFED 5.x)
...
| Code Block |
|---|
$ sudo smparquery ARInfo -L 136 -C mlx5_1 op = ARInfo, dest = 136, rest = -I- Getting ARInfo from lid=136 -I- ARInfo: AR Status.........................Enabled Is ARN Supported..................Yes Is FRN Supported..................Yes Is FR Supported...................Yes FR Enabled........................Yes RN Xmit Enabled...................Yes AR Sub Groups Active..............0 AR Groups Copy Supported..........15 Direction Num Supported...........4 AR Fallback.......................Enabled AR IS4 mode.......................No AR Glb Group......................Yes AR By SL Cap......................Yes AR By Transport Cap...............Yes AR Dynamic Cap Calc...............Yes AR Group Capability...............1792 AR Group Top......................1 AR Group Table Capability.........1 RN String Width Capability........3 AR Sub Groups Capability..........0x3 AR Version........................2 RN Version........................0 AR By SL Mask Enable..............0 (All SLs enabled) AR SL Mask........................N/A AR By TransportDisable............0x5 AR Ageing Time Value..............0 |
Note that the switch SL map is always enabled for all SLs. The SL mask configuration is done on the adapter.
Get the SL Mask Configuration on the hosts
ibdiagnet gathers the SL Mask on the adapters. look for the OOOSLMASK column in the ibdiagnet2.db_csv output. In this example, SL mask is set to 0xFFFA.
| Code Block |
|---|
cat /var/tmp/ibdiagnet2/ibdiagnet2.db_csv | grep -A 1 OOOSLMask NodeGuid,PortGuid,PortNum, ... ,OOOSLMask,CapMsk2,FECActv,RetransActv 0x248a0703009c01e6,0x248a0703009c01e6,1, ... ,0xfffa,2,3,0 -- NodeGuid,PortGuid,PortNum, ... , ,HDRFECModeEnabled,OOOSLMask 0x248a0703009c01e6,0x248a0703009c01e6,1,,0x0000,0xfffa |
ibdiagnet
Refer to the example here: .