...
To check the latency of RDMA write -Write please follow those notes:
Make sure to Please use the core local to the HCA, in this example the HDR InfiniBand adapter is local to core 80
More iterations helps to make the output more smooth. In this example, We are using 10000 iterations.
...
To check the latency of RDMA write follow those notes:
Make sure to Please use the core local to the HCA, in this example the HDR InfiniBand adapter is local to core 80
More iterations helps to make the output more smooth. In this example, we are using 10000 iterations.
NPS Configuration should set to 1 (or 2) for HDR for maximum bandwidth.
...
To check the latency of RDMA write follow those notes:
Make sure to Please use the core local to the HCA, in this example the HDR InfiniBand adapter is local to core 80
More iterations helps to make the output more smooth. In this example, we are using 10000 iterations.
NPS Configuration should set to 1 (or 2) for HDR for maximum bandwidth.
...
| Code Block | ||
|---|---|---|
| ||
$ numactl --physcpubind=80 ib_write_bw -a -d mlx5_2 -i 1 --report_gbits -F -n 10000 -b
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write Bidirectional BW Test
Dual-port : OFF Device : mlx5_2
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0xba QPN 0x0138 PSN 0x210c6 RKey 0x01e1ef VAddr 0x002b7835800000
remote address: LID 0xd1 QPN 0x0139 PSN 0x99d1e6 RKey 0x01dae9 VAddr 0x002abd1dc00000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 10000 0.133457 0.132682 8.292601
4 10000 0.27 0.27 8.312058
8 10000 0.53 0.53 8.270148
16 10000 1.07 1.06 8.289442
32 10000 2.13 2.12 8.287524
64 10000 4.24 4.22 8.251245
128 10000 8.53 8.50 8.301570
256 10000 16.99 16.93 8.264573
512 10000 33.94 33.83 8.258309
1024 10000 66.96 66.53 8.120855
2048 10000 132.49 131.87 8.048677
4096 10000 261.92 261.10 7.968046
8192 10000 358.14 357.88 5.460775
16384 10000 379.43 379.22 2.893244
32768 10000 391.65 391.56 1.493699
65536 10000 391.03 390.98 0.745735
131072 10000 393.45 393.42 0.375196
262144 10000 393.68 393.66 0.187714
524288 10000 393.78 393.76 0.093879
1048576 10000 393.85 393.80 0.046945
2097152 10000 393.88 393.82 0.023473
4194304 10000 393.87 393.82 0.011737
8388608 10000 393.85 393.83 0.005869
--------------------------------------------------------------------------------------- |
Note: All trademarks are property of their respective owners. All information is provided “As-Is” without any kind of warranty. The HPC-AI Advisory Council makes no representation to the accuracy and completeness of the information contained herein. HPC-AI Advisory Council undertakes no duty and assumes no obligation to update or correct any information presented herein.
References
...