HowTo Find the local NUMA node in AMD EPYC Servers
- Ophir Maor
Owned by Ophir Maor
References
Basic information
1. Numa
How to check the mapping of the Network adapter to the NUMA node?
Each of the EPYC CPUs comes with 4 NUMA nodes (8 NUMAs on 2 sockets).
Here is an example from our Venus cluster.
$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 1 Core(s) per socket: 32 Socket(s): 2 NUMA node(s): 8 Vendor ID: AuthenticAMD CPU family: 23 Model: 1 Model name: AMD EPYC 7551 32-Core Processor Stepping: 2 CPU MHz: 2000.000 CPU max MHz: 2000.0000 CPU min MHz: 1200.0000 BogoMIPS: 3999.39 Virtualization: AMD-V L1d cache: 32K L1i cache: 64K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 NUMA node2 CPU(s): 16-23 NUMA node3 CPU(s): 24-31 NUMA node4 CPU(s): 32-39 NUMA node5 CPU(s): 40-47 NUMA node6 CPU(s): 48-55 NUMA node7 CPU(s): 56-63
Note that other OEMs may map differently the cores to CPUs. Here is another example from Dell R7415 with one socket equipped with EPYC 7551.
# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian ... Thread(s) per core: 1 Core(s) per socket: 32 Socket(s): 1 NUMA node(s): 4 Vendor ID: AuthenticAMD ... Model name: AMD EPYC 7551 32-Core Processor ... CPU MHz: 1996.203 ... NUMA node0 CPU(s): 0,4,8,12,16,20,24,28 NUMA node1 CPU(s): 1,5,9,13,17,21,25,29 NUMA node2 CPU(s): 2,6,10,14,18,22,26,30 NUMA node3 CPU(s): 3,7,11,15,19,23,27,31 ...
2. Check the adapter you have:
$ sudo lspci | grep Mel 21:00.0 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 21:00.1 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
3. Check with Numa is connected to the network adapter:
There are several options to do it.
Using MST tools
$ ibdev2netdev mlx5_0 port 1 ==> ib0 (Up) mlx5_1 port 1 ==> ib1 (Down) $ sudo mst start ... $ sudo mst status -v MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA ConnectX5(rev:0) /dev/mst/mt4121_pciconf0.1 21:00.1 mlx5_1 net-ib1 2 ConnectX5(rev:0) /dev/mst/mt4121_pciconf0 21:00.0 mlx5_0 net-ib0 2
Using the net device:
$ cat /sys/class/net/ib0/device/numa_node 2
Note: Knowing which numa is connected to the network adapter is important for performance tuning.
Another option is to use lstopo-no-graphics
You can see under numa 2, the two port adapter.
$ lstopo-no-graphics
Machine (256GB total)
Package L#0
NUMANode L#0 (P#0 32GB)
L3 L#0 (8192KB)
L2 L#0 (512KB) + L1d L#0 (32KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 (P#0)
L2 L#1 (512KB) + L1d L#1 (32KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 (P#1)
L2 L#2 (512KB) + L1d L#2 (32KB) + L1i L#2 (64KB) + Core L#2 + PU L#2 (P#2)
L2 L#3 (512KB) + L1d L#3 (32KB) + L1i L#3 (64KB) + Core L#3 + PU L#3 (P#3)
L3 L#1 (8192KB)
L2 L#4 (512KB) + L1d L#4 (32KB) + L1i L#4 (64KB) + Core L#4 + PU L#4 (P#4)
L2 L#5 (512KB) + L1d L#5 (32KB) + L1i L#5 (64KB) + Core L#5 + PU L#5 (P#5)
L2 L#6 (512KB) + L1d L#6 (32KB) + L1i L#6 (64KB) + Core L#6 + PU L#6 (P#6)
L2 L#7 (512KB) + L1d L#7 (32KB) + L1i L#7 (64KB) + Core L#7 + PU L#7 (P#7)
HostBridge L#0
PCIBridge
PCIBridge
PCI 1a03:2000
GPU L#0 "card0"
GPU L#1 "controlD64"
NUMANode L#1 (P#1 32GB)
L3 L#2 (8192KB)
L2 L#8 (512KB) + L1d L#8 (32KB) + L1i L#8 (64KB) + Core L#8 + PU L#8 (P#8)
L2 L#9 (512KB) + L1d L#9 (32KB) + L1i L#9 (64KB) + Core L#9 + PU L#9 (P#9)
L2 L#10 (512KB) + L1d L#10 (32KB) + L1i L#10 (64KB) + Core L#10 + PU L#10 (P#10)
L2 L#11 (512KB) + L1d L#11 (32KB) + L1i L#11 (64KB) + Core L#11 + PU L#11 (P#11)
L3 L#3 (8192KB)
L2 L#12 (512KB) + L1d L#12 (32KB) + L1i L#12 (64KB) + Core L#12 + PU L#12 (P#12)
L2 L#13 (512KB) + L1d L#13 (32KB) + L1i L#13 (64KB) + Core L#13 + PU L#13 (P#13)
L2 L#14 (512KB) + L1d L#14 (32KB) + L1i L#14 (64KB) + Core L#14 + PU L#14 (P#14)
L2 L#15 (512KB) + L1d L#15 (32KB) + L1i L#15 (64KB) + Core L#15 + PU L#15 (P#15)
HostBridge L#3
PCIBridge
PCI 8086:1521
Net L#2 "eth0"
PCI 8086:1521
Net L#3 "eno2"
PCI 8086:1521
Net L#4 "eno3"
PCI 8086:1521
Net L#5 "eno4"
PCIBridge
PCI 1022:7901
Block(Disk) L#6 "sda"
NUMANode L#2 (P#2 32GB)
L3 L#4 (8192KB)
L2 L#16 (512KB) + L1d L#16 (32KB) + L1i L#16 (64KB) + Core L#16 + PU L#16 (P#16)
L2 L#17 (512KB) + L1d L#17 (32KB) + L1i L#17 (64KB) + Core L#17 + PU L#17 (P#17)
L2 L#18 (512KB) + L1d L#18 (32KB) + L1i L#18 (64KB) + Core L#18 + PU L#18 (P#18)
L2 L#19 (512KB) + L1d L#19 (32KB) + L1i L#19 (64KB) + Core L#19 + PU L#19 (P#19)
L3 L#5 (8192KB)
L2 L#20 (512KB) + L1d L#20 (32KB) + L1i L#20 (64KB) + Core L#20 + PU L#20 (P#20)
L2 L#21 (512KB) + L1d L#21 (32KB) + L1i L#21 (64KB) + Core L#21 + PU L#21 (P#21)
L2 L#22 (512KB) + L1d L#22 (32KB) + L1i L#22 (64KB) + Core L#22 + PU L#22 (P#22)
L2 L#23 (512KB) + L1d L#23 (32KB) + L1i L#23 (64KB) + Core L#23 + PU L#23 (P#23)
HostBridge L#6
PCIBridge
PCI 15b3:1019
Net L#7 "ib0_mlx5"
OpenFabrics L#8 "mlx5_0"
PCI 15b3:1019
Net L#9 "ib0"
OpenFabrics L#10 "mlx5_1"
NUMANode L#3 (P#3 32GB)
L3 L#6 (8192KB)