Export of RoCEv2 Base Transport Header (BTH) Information Using IP Flow Information Export (IPFIX)

Remote Direct Memory Access (RDMA) is a network technology that allows a computer to read from or write to the memory of another computer directly, without involving the operating system. This zero-copy and kernel-bypass feature greatly reduces CPU overhead and communication latency. InfiniBand and RDMA over Converged Ethernet (RoCE) are two mainstream RDMA implementations that bypass the operating system kernel and achieve zero-copy data transfer. RoCE technology has become a key component of high-performance data center networks, especially in low-latency, high-throughput scenarios such as artificial intelligence training, distributed storage, and financial transactions. RDMA over Converged Ethernet version 2 (RoCEv2) runs on top of UDP (port 4791) and inherits the transport layer protocol of the InfiniBand Architecture (IBA). The existing IPFIX standard lacks the ability to monitor specific fields of RoCEv2, which limits the ability of network operators to perform in-depth analysis, troubleshooting, and performance optimization of RDMA traffic. To close this gap, this document defines a new set of Information Elements (IEs) to carry RoCEv2 BTH key fields.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The following terms are used as defined in :

IPFIX
IPFIX Information Elements

The following terms are used in this document:

QP: Queue Pair, a communication endpoint in RDMA architecture.
BTH: Base Transport Header, the RDMA transport header containing QP information.
P_Key: Partition Key.
PSN: Packet Sequence Number.

This section defines new IPFIX IEs for exporting RoCEv2 BTH key fields. The RoCEv2 BTH OpCode field, which indicates the type of the InfiniBand Architecture data packet. The RoCEv2 BTH Partition Key field, which identifies the logical partition to which the data packet belongs. The RoCEv2 BTH Destination Queue Pair (QP) field, indicating the work QP number at the destination. The RoCEv2 BTH Source Queue Pair (QP) field (if present), indicating the work QP number at the Source. The RoCEv2 BTH Packet Sequence Number (PSN) field, which is used to detect whether data packets are lost or duplicated, ensuring the reliability and orderliness of transmission. The SE, M, Pad, and TVer fields of The RoCEv2 BTH. The size of this Information Element is 1 octet. The F/R, and B/R fields of The RoCEv2 BTH. The size of this Information Element is 1 octet. The A field of The RoCEv2 BTH. The size of this Information Element is 1 octet.

The exporter needs to parse RoCEv2 BTH information, which may affect the exporter's performance. Implementers SHOULD consider measures to mitigate this impact, such as sampling rate limiting or hardware acceleration.

The Security Considerations for IPFIX apply to this document as well. When exporting RDMA BTH information across security domains, to prevent information security risks caused by the leakage of sensitive network topology details such as QP numbers, it is recommended to implement comprehensive protection measures, such as using the encrypted transmission options of the IPFIX framework (such as DTLS ) to ensure the confidentiality and integrity of data during transmission. To defend against Denial-of-Service (DOS) attacks that may be caused by maliciously crafted RoCEv2 packets and to prevent the exporter from being overloaded by processing a large amount of invalid traffic, it is recommended to adopt certain protection strategies, such as configuring reasonable rate limiting policies to prevent the monitoring system from being overwhelmed by massive data packets and to ensure its stable operation.

This document specifies new IPFIX IEs to enable export of RoCEv2 BTH key fields along with other flow information. This document requests IANA to add these IPFIX IEs to the "IPFIX Information Elements" registry available at . Table 1 lists the new IPFIX IEs for RoCEv2 BTH:

Name:: rdmaOpCode

Element ID:: TBD1

Description:: The RoCEv2 BTH OpCode field, which indicates the type of the InfiniBand Architecture data packet.

Abstract Data Type:: unsigned8

Data Type Semantics:: identifier

Status:: current

Reference:: [this document]

Name:: rdmaPartitionKey

Element ID:: TBD2

Description:: The RoCEv2 BTH Partition Key field, which identifies the logical partition to which the data packet belongs.

Abstract Data Type:: unsigned16

Data Type Semantics:: identifier

Status:: current

Reference:: [this document]

Name:: rdmaDestinationQP

Element ID:: TBD3

Description:: The RoCEv2 BTH Destination Queue Pair (QP) field, indicating the work QP number at the destination. The actual effective bits are 24 bits, stored in the lower 24 bits of the 32-bit field, and the higher 8 bits should be 0.

Abstract Data Type:: unsigned32

Data Type Semantics:: identifier

Status:: current

Reference:: [this document]

Name:: rdmaSourceQP

Element ID:: TBD4

Description:: The RoCEv2 BTH Source Queue Pair (QP) field (if present), indicating the work QP number at the Source. The actual effective bits are 24 bits, stored in the lower 24 bits of the 32-bit field, and the higher 8 bits should be 0.

Abstract Data Type:: unsigned32

Data Type Semantics:: identifier

Status:: current

Reference:: [this document]

Name:: rdmaPacketSequenceNumber

Element ID:: TBD5

Description:: The RoCEv2 BTH Packet Sequence Number (PSN) field, which is used to detect whether data packets are lost or duplicated, ensuring the reliability and orderliness of transmission. The actual effective bits are 24 bits, stored in the lower 24 bits of the 32-bit field, and the higher 8 bits should be 0.

Abstract Data Type:: unsigned32

Data Type Semantics:: default

Status:: current

Reference:: [this document]

Name:: rdmaBTHFlags1

Element ID:: TBD6

Description:: The SE, M, Pad, and TVer fields of The RoCEv2 BTH. The size of this Information Element is 1 octet. 0 1 2 3 4 5 6 7 +--+-+-+-+-+-+-+-+ |SE|M|Pad| TVer | +--+-+-+-+-+-+-+-+ Bits 0: Solicited Event (SE) field. Bits 1: Migration Request (M) field. Bits 2-3: Pad Count (Pad) field. Bits 4-7: Transport Header Version (TVer) field.

Abstract Data Type:: unsigned8

Data Type Semantics:: default

Status:: current

Reference:: [this document]

Name:: rdmaBTHFlags2

Element ID:: TBD7

Description:: The F/R, and B/R fields of The RoCEv2 BTH. The size of this Information Element is 1 octet. 0 1 2 3 4 5 6 7 +---+---+---+---+---+---+---+---+ |F/R|B/R| Resv | +---+---+---+---+---+---+---+---+ Bits 0: Forward Explicit Congestion Notification (FECN)/Res1 (F/R) field. Bits 1: Backward Explicit Congestion Notification (BECN)/Res1 (B/R) field. Bits 2-7: Reserved field.

Abstract Data Type:: unsigned8

Data Type Semantics:: flags

Status:: current

Reference:: [this document]

Name:: rdmaBTHFlags3

Element ID:: TBD8

Description:: The A field of The RoCEv2 BTH. The size of this Information Element is 1 octet. 0 1 2 3 4 5 6 7 +-+-+-+-+-+-+-+-+ |A| Resv | +-+-+-+-+-+-+-+-+ Bits 0: Acknowledge Request (A) field. Bits 1-7: Reserved field.

Abstract Data Type:: unsigned8

Data Type Semantics:: flags

Status:: current

Reference:: [this document]