Device information

Owned by Ophir Maor
Last updated: Jan 27, 2020
31 min read

Here are few options to get your InfiniBand HCA device information

 

 

ibv_devinfo

Print information about RDMA devices available for use from userspace.

$ ibv_devinfo hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 20.26.4012 node_guid: 9803:9b03:0067:a614 sys_image_guid: 9803:9b03:0067:a614 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000228 phys_port_cnt: 1 Device ports: port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 38 port_lmc: 0x00 link_layer: InfiniBand

ibv_devinfo -v

Print verbos information about RDMA devices available for use from userspace.

$ ibv_devinfo -v hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 20.26.4012 node_guid: 9803:9b03:0067:a614 sys_image_guid: 9803:9b03:0067:a614 vendor_id: 0x02c9 vendor_part_id: 4123 hw_ver: 0x0 board_id: MT_0000000228 phys_port_cnt: 1 max_mr_size: 0xffffffffffffffff page_size_cap: 0xfffffffffffff000 max_qp: 262144 max_qp_wr: 32768 device_cap_flags: 0xe17e1c36 BAD_PKEY_CNTR BAD_QKEY_CNTR AUTO_PATH_MIG CHANGE_PHY_PORT PORT_ACTIVE_EVENT SYS_IMAGE_GUID RC_RNR_NAK_GEN XRC Unknown flags: 0xe16e0000 device_cap_exp_flags: 0x5648F8F100000000 EXP_DC_TRANSPORT EXP_CROSS_CHANNEL EXP_MR_ALLOCATE EXT_ATOMICS EXT_SEND NOP EXP_UMR EXP_ODP EXP_RX_CSUM_TCP_UDP_PKT EXP_RX_CSUM_IP_PKT EXP_DC_INFO EXP_MASKED_ATOMICS EXP_RX_TCP_UDP_PKT_TYPE EXP_PHYSICAL_RANGE_MR EXP_UMR_FIXED_SIZE EXP_PACKET_BASED_CREDIT_MODE Unknown flags: 0x200000000000 max_sge: 30 max_sge_rd: 30 max_cq: 16777216 max_cqe: 4194303 max_mr: 16777216 max_pd: 16777216 max_qp_rd_atom: 16 max_ee_rd_atom: 0 max_res_rd_atom: 4194304 max_qp_init_rd_atom: 16 max_ee_init_rd_atom: 0 atomic_cap: ATOMIC_HCA (1) log atomic arg sizes (mask) 0x8 masked_log_atomic_arg_sizes (mask) 0x3c masked_log_atomic_arg_sizes_network_endianness (mask) 0x34 max fetch and add bit boundary 64 log max atomic inline 5 max_ee: 0 max_rdd: 0 max_mw: 16777216 max_raw_ipv6_qp: 0 max_raw_ethy_qp: 0 max_mcast_grp: 2097152 max_mcast_qp_attach: 240 max_total_mcast_qp_attach: 503316480 max_ah: 2147483647 max_fmr: 0 max_srq: 8388608 max_srq_wr: 32767 max_srq_sge: 31 max_pkeys: 128 local_ca_ack_delay: 16 hca_core_clock: 156250 max_klm_list_size: 65536 max_send_wqe_inline_klms: 20 max_umr_recursion_depth: 4 max_umr_stride_dimension: 1 general_odp_caps: ODP_SUPPORT ODP_SUPPORT_IMPLICIT max_size: 0xFFFFFFFFFFFFFFFF rc_odp_caps: SUPPORT_SEND SUPPORT_RECV SUPPORT_WRITE SUPPORT_READ SUPPORT_SRQ_RECV uc_odp_caps: NO SUPPORT ud_odp_caps: SUPPORT_SEND dc_odp_caps: SUPPORT_SEND SUPPORT_WRITE SUPPORT_READ xrc_odp_caps: NO SUPPORT raw_eth_odp_caps: NO SUPPORT max_dct: 262144 max_device_ctx: 1020 Multi-Packet RQ supported Supported for objects type: IBV_EXP_MP_RQ_SUP_TYPE_SRQ_TM IBV_EXP_MP_RQ_SUP_TYPE_WQ_RQ Supported payload shifts: 2 bytes Log number of strides for single WQE: 3 - 16 Log number of bytes in single stride: 6 - 13 rx_pad_end_addr_align: 64 tso_caps: max_tso: 0 packet_pacing_caps: qp_rate_limit_min: 0kbps qp_rate_limit_max: 0kbps ooo_caps: ooo_rc_caps = 0x1 ooo_xrc_caps = 0x1 ooo_dc_caps = 0x1 ooo_ud_caps = 0x0 SUPPORT_RC_RW_DATA_PLACEMENT SUPPORT_XRC_RW_DATA_PLACEMENT SUPPORT_DC_RW_DATA_PLACEMENT sw_parsing_caps: supported_qp: max_rndv_hdr_size: 0x40 max_num_tags: 0x7f max_ops: 0x8000 max_sge: 0x1 capability_flags: IBV_EXP_TM_CAP_RC IBV_EXP_TM_CAP_DC tunnel_offloads_caps: UMR fixed size: max entity size: 2147483648 Device ports: port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 1 port_lid: 38 port_lmc: 0x00 link_layer: InfiniBand max_msg_sz: 0x40000000 port_cap_flags: 0x2251e848 max_vl_num: 4 (3) bad_pkey_cntr: 0x0 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 128 gid_tbl_len: 8 subnet_timeout: 18 init_type_reply: 0 active_width: 4X (2) active_speed: 50.0 Gbps (64) phys_state: LINK_UP (5) GID[ 0]: fe80:0000:0000:0000:9803:9b03:0067:a614

 

ibstatus

ibstatus is a script which displays basic information obtained from the local IB driver. Output includes LID, SMLID, port state, link width active, and port physical state.

$ ibstatus Infiniband device 'mlx5_0' port 1 status: default gid: fe80:0000:0000:0000:9803:9b03:0067:a614 base lid: 0x26 sm lid: 0x1 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (4X HDR) link_layer: InfiniBand

 

ibstat

ibstat is a binary which displays basic information obtained from the local IB driver. Output includes LID, SMLID, port state, link width active, and port physical state.

It is similar to the ibstatus utility but implemented as a binary rather than a script. It has options to list CAs and/or ports and displays more information than ibstatus.

$ ibstat CA 'mlx5_0' CA type: MT4123 Number of ports: 1 Firmware version: 20.26.4012 Hardware version: 0 Node GUID: 0x98039b030067a614 System image GUID: 0x98039b030067a614 Port 1: State: Active Physical state: LinkUp Rate: 200 Base lid: 38 LMC: 0 SM lid: 1 Capability mask: 0x2651e848 Port GUID: 0x98039b030067a614 Link layer: InfiniBand

 

ibv_devices

Get the device list on the node.

$ ibv_devices device node GUID ------ ---------------- mlx5_3 98039b030067a619 mlx5_2 98039b030067a618 mlx5_1 98039b030067a615 mlx5_0 98039b030067a614

Â