...
Before starting, please follow AMD Rome Tuning Guide for InfiniBand HPC to set the cluster parameters for high performance. Please use the latest firmware and driver, and find a core close to the adapter on your local Numa, see HowTo Find the local NUMA node in AMD EYPC Servers.
RDMA Testing is important to have before each application or micro-benchmark application testing, as it gives you the low level capabilities of your fabric.
...
| Code Block | ||
|---|---|---|
| ||
$ numactl --physcpubind=80 ib_write_bw -a -d mlx5_2 -i 1 --report_gbits -F -n 10000 -b
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write Bidirectional BW Test
Dual-port : OFF Device : mlx5_2
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0xba QPN 0x0138 PSN 0x210c6 RKey 0x01e1ef VAddr 0x002b7835800000
remote address: LID 0xd1 QPN 0x0139 PSN 0x99d1e6 RKey 0x01dae9 VAddr 0x002abd1dc00000
---------------------------------------------------------------------------------------
#bytes #iterations BW average[Gb/sec]
2 10000 0.132682
4 10000 0.27
8 10000 0.53
16 10000 1.06
32 10000 2.12
64 10000 4.22
128 10000 8.50
256 10000 16.93
512 10000 33.83
1024 10000 66.53
2048 10000 131.87
4096 10000 261.10
8192 10000 357.88
16384 10000 379.22
32768 10000 391.56
65536 10000 390.98
131072 10000 393.42
262144 10000 393.66
524288 10000 393.76
1048576 10000 393.80
2097152 10000 393.82
4194304 10000 393.82
8388608 10000 393.83
--------------------------------------------------------------------------------------- |
Note: All trademarks are property of their respective owners. All information is provided “As-Is” without any kind of warranty. The HPC-AI Advisory Council makes no representation to the accuracy and completeness of the information contained herein. HPC-AI Advisory Council undertakes no duty and assumes no obligation to update or correct any information presented herein.