Proactive Flow Control Point Detection in WAN

Introduction establishes requirements for sub-millisecond to millisecond fast notification delivery for network events (e.g., failure, congestion or state change) in modern network applications. An example where this fast network notification problem arises is flow control in wide-area networks (WANs). With the growth of intelligent computing services, scenarios such as disaggregated computing and real-time inference require the lossless transmission of large volumes of bursty traffic over WANs. To achieve lossless data transmission over WAN, there're some recent works aiming to deploy flow control mechanisms in WAN to avoid packet loss in case of congestion, e.g, discusses how to deploy PFC(Priority-based Flow Control, ) in WAN based on SRv6 data plane, and proposes the method of precise/fine-grained flow control to achieve flow control at the network slice level. As in , the fast network notification recipient nodes include adjacent routers/switches, non-adjacent routers/switches, ingress nodes, end hosts, and controllers. To deploy flow control mechanism in WAN, the node facing congestion needs to generate a flow control message and sends it to the upstream point which is able to perform the flow control action (e.g, stop sending the corresponding traffic or reducing the sending rate). In this case, the recipient node may be the adjacent router/switch, non-adjacent router/switch or the ingress node. To control the sending of the flow control message, one of the following mechanisms may be used:

Multicast. Although standard PFC propagates congestion information via Ethernet multicast frames, multicast-based mechanism is not preferred in WAN since it cannot accurately reach upstream nodes, potentially leading to incorrect flow suppression and impacting unrelated services.
Centralized configuration. The controller, with the awareness of all the node and the path information in the network, can configure the information of the upstream flow control point on each node that may generate the flow control message. But this methods will bring extra burden for the controller in large scale networks.
Distributed decision. The congested node decides the upstream node itself. The difficulty is that the congested node needs to be aware of the necessary information to make the proper decision.

Based on the above considerations, this document proposes a proactive detection mechanism for the distributed decision mechanism, letting the congested node to know precisely which upstream point should the flow control message be sent to. Besides flow control in WAN, this mechanism potentially can support other types of network fast notification use cases if the distributed decision mechanism is required. The detailed flow control mechanism itself on the recipient node is out of the scope of this document.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Flow Control in WAN

Flow Control in WAN +----+ --> +----+ --> +----+ --> +----+ | R1 | | R2 | | R3 | | R4 | | R5 | +----+ +----+ +----+ +----+ +----+ ]]> As shown in Figure 1, data center A and data center B are connected via a path(R1-R2-R3-R4-R5) in WAN. R1,R2,R4 and R5 are able to perform the function of flow control, and R3 is a legacy device which doesn't support any flow control technology. When congestion occurs at R4, R4 generates a flow control message (e.g, the PFC pause frame defined in , the congestion notification message defined in ) to the nearest upstream stream node that is able to perform flow control, i.e, node R2. R2 receives the notification and performs the flow control based on the content of the notification message and local capacity. If R2 cannot handle the congestion, a flow control message is forwarded further upstream to R1.

Proactive Detection Mechanism Requirements For the distributed decision mechanism, what is needed is a lightweight, data-plane method that allows each notification message generating node to be aware of the identity of the nearest upstream point capable of handling a specific class of fast notifications. This discovery should be:

Proactive: information is available before the event occurs, so that the notification can be sent immediately when needed.
Same-path: follows the same path as the data traffic, ensuring that the discovered node is topologically relevant.
State-efficient: only the nodes that may generate notifications need to maintain the discovered information.

Proactive Control Point Detection Method The basic concept of the proactive control point detection method in this document is to send a control point detection packet along the packet forwarding path. When receiving the control detection packet, the node that is capable of flow control updates the packet with its own information(e.g, the interface address or the corresponding SRv6 ajacency SID of the interface), so the detection packet will always carry the information of the nearest upstream node that's capable of flow control and the node receiving the detection packet would store this information and use it as the destination of the flow control message when congestion occurs.

Proactive Control Point Detection Packet Format A new Hop-by-Hop option (Section 4.3 of ) type is defined in this document to carry the fields above for control point detection. Its format is shown in Figure 2.

Control Point Detection Option

Option Type: TBA for the control point detection option.
Opt Data Len: 8-bit unsigned integer indicates the length of the option Data field of this option, in octets.
Message Type: 8-bit identifier of the type of the message. The type of Flow Control Detection message is TBD1.
Upstream Type: indicates the type of the upstream point that is able perform corresponding control for the traffic. When set to 1, the upstream Point Info carries a 128-bit IPv6 interface address, when set to 2, the upstream Point Info carries a 128-bit SRv6 adj-SID.
Upstream Point Info: Variable-length field carrying the corresponding upstream point information based on the value of the Upstream Type.
Context Type: The Context Type field is an 8-bit bitmap that specifies which contexts are included in the Context field of the packet. Each bit in this field corresponds to a specific context. When a bit is set to 1, it indicates the presence of the corresponding context.
Context: Optional, Context-Type-specific information.

The packet fields defined above can be carried in-band or out-band as long as the packet is forwarded along the same path of the normal traffic flow.

Processing Procedures As in Figure 1, the traffic of network slice 1 is forwarded along an SRv6 path, the corresponding SID list is <SID-R12,SID-R23,SID-R45>, whereas SID-R12 is the adjacency SID of R1 for the adjacency between R1 and R2, SID-R23 and SID-R45 are also the corresponding adj-SID on R2 and R4. And precise flow control is enabled in the network to control the congestion at the network slice level. R1,R2,R4 and R5 are able to perform flow control at the network slice level, and R3 is a legacy device which doesn't support any flow control technology. To detect the flow control point along the path of slice 1, R1 sends a flow control detection packet in band with the traffic of slice 1, since R1 is capable of flow control, R1 puts SID-R12 into the Upstream Point Info. When R2 receives the packet, it stores the mapping between slice 1 and SID-R12, and updates the Upstream Point Info with its own information, i.e, SID-R23. Since R3 doesn't recognize the flow control detection packet, it just forwards the packet based on the SID-list and slice-ID of the packet. When R4 receives the packet, it stores the mapping between slice 1 and SID-R23, and updates the Upstream Point Info with its own information, i.e, SID-R45. When R5 receives the packet, it stores the mapping between slice 1 and SID-R45, and since R5 is the endpoint, it stops processing the packet further. When congestion occurs at R4 in slice 1, R4 would generate a flow control message for slice 1, and based on the local information, R4 finds the information of upstream flow control point, i.e, SID-R23, and uses it as the destination of the flow control message. When R2 receives the flow control message with DA set as local adj-SID SID-R23, R2 perform the flow control for slice 1 on the port related with SID-R23 based on the context of the flow control message. If R2 is not able to control the congestion and generates a flow control message further, R2 would send the message with DA set to SID-R12 based on the local information.

Considerations for Other Usecases The proactive control point detection mechanism described in this document is not limited to flow control in WAN. It provides a generic recipient discovery primitive for fast network notification scenarios for the distributed decision method, especially when intended recipient node is a non-adjacent node on the forwarding path. Future version of this document will discuss more about other fast network notification usecases.

Security Considerations The security considerations with IPv6 Hop-by-Hop Options header are described in ,, , , . This document introduces a new IPv6 Hop-by-Hop option which is either processed in the fast path or ignored by network nodes, thus it does not introduce additional security issues.

IANA Considerations This document requests IANA to assign a new option type from "Destination Options and Hop-by-Hop Options" registry .