...

Advanced → AMD CBS → NBIO → SMU → DF Cstates = Disabled

Relaxed Ordering

Relaxed Using relaxed ordering is PCI configuration to run a-sync in parallel to get more bandwidth.

Notes:

...

Relaxed ordering API is used by default on perftest since MLNX_OFED version 5.1-0.6.6.0

...

for PCI operations is a key mechanism to get maximum performance when targeting memory attached to AMD 2nd Gen EPYC CPUs. There are two ways to achieve that : using communication libraries which enable relaxed ordering using a new API, or forcing all traffic to use relaxed ordering. The former is preferred, since the latter can break libraries which optimized communication relying on memory being written and visible in a given order in memory.

By default, the PCI_WR_ORDERING firmware parameter setting is set to "per_mkey"

# mlxconfig -d /dev/mst/mt4123_pciconf0 q | grep PCI_WR_ORDERING     
    PCI_WR_ORDERING                     per_mkey(0)

In this configuration, applications can enable relaxed ordering per memory region.

• UCX since v1.9 can use that feature when UCX_IB_PCI_RELAXED_ORDERING is set to "on". The default is auto which is enabled for specific architectures and may change.

# Enable relaxed ordering for PCIe transactions to improve performance on some systems.
#
# syntax:    <on|off|auto>
#
UCX_IB_PCI_RELAXED_ORDERING=auto

...

In case you have Nvidia GPUs, make sure to use NCCL 2.8 or later.

To force all libraries to use relaxed ordering, you can set the following parameter to 1.

...

• Perftests embedded in MLNX_OFED also use relaxed ordering by default since version 5.1-0.6.6.0.

If using UCX is not possible, one can force all traffic to use relaxed ordering :

# mlxconfig -d /dev/mst/mt4123_pciconf0 q | grep ORDER
         PCI_WR_ORDERING                     per_mkey(0)


# mlxconfig -d /dev/mst/mt4123_pciconf0 -s PCI_WR_ORDERING=1
…

# mlxconfig -d /dev/mst/mt4123_pciconf0 q | grep ORDER
         PCI_WR_ORDERING                     force_relax(1)

...

# mlxconfig -d /dev/mst/mt4123_pciconf0.1 q | grep ORDER
         PCI_WR_ORDERING                     per_mkey(0)


# mlxconfig -d /dev/mst/mt4123_pciconf0.1 -s PCI_WR_ORDERING=1
…

# mlxconfig -d /dev/mst/mt4123_pciconf0.1 q | grep ORDER
         PCI_WR_ORDERING                     force_relax(1)

Note : Forcing relaxed ordering will break NCCL up to version 2.7, make sure to upgrade to NCCL 2.8.

Benchmarking

Basic RDMA Testing

...