Internet DRAFT - draft-lyu-rtgwg-coordinated-cm
draft-lyu-rtgwg-coordinated-cm
RTGWG Y. Lyu
Internet-Draft Y. Zhang
Intended status: Standards Track M. Liu
Expires: 22 April 2024 Huawei
20 October 2023
Coordinated Congestion Management
draft-lyu-rtgwg-coordinated-cm-00
Abstract
AI fabric is sensitive to bandwidth. Congestion management,
including congestion control and load balancing, is a main method to
fully utilize network resource. However, current congestion
management mechanism are not coordinated, which leads to throughput
decreasing. This document provides a scheme for coordinating
different congestion management mechanisms. It describes the design
principle, behaviors of network switches and hosts in the scheme, and
gives an example to show end-to-end procedure.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 22 April 2024.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
Lyu, et al. Expires 22 April 2024 [Page 1]
Internet-Draft CCM October 2023
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Requirements Language . . . . . . . . . . . . . . . . . . . . 4
4. Existing congestion management . . . . . . . . . . . . . . . 4
5. Design principle of coordinated congestion management . . . . 5
6. Coordinated congestion management scheme . . . . . . . . . . 6
6.1. Coordination tag . . . . . . . . . . . . . . . . . . . . 6
6.2. Notification message . . . . . . . . . . . . . . . . . . 6
6.3. Behavior of network switches . . . . . . . . . . . . . . 7
6.3.1. Identify congestion type . . . . . . . . . . . . . . 7
6.3.2. Notify CC congestion . . . . . . . . . . . . . . . . 7
6.3.3. Notify upstream point to perform AR . . . . . . . . . 8
6.3.4. Perform congestion control . . . . . . . . . . . . . 8
6.3.5. Perform adaptive routing . . . . . . . . . . . . . . 8
6.4. Behavior of source hosts . . . . . . . . . . . . . . . . 9
7. An example of end-to-end procedure . . . . . . . . . . . . . 9
8. Security Considerations . . . . . . . . . . . . . . . . . . . 11
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 11
10.1. Normative References . . . . . . . . . . . . . . . . . . 11
10.2. Informative References . . . . . . . . . . . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12
1. Introduction
ML/AI has been progressing rapidly over the last decade. ML/AI model
compute, which is measured in FLOPs, are constantly increasing. It
is imperative to employ distributed parallel training to train such
large models in AI cluster.
The communication in AI cluster is bandwidth sensitive. Analyzing
data parallelism and model parallelism which are the 2 acceleration
methods in AI training, it shows an on-off type of burst traffic
pattern with huge traffic amount in each iteration.
Therefore, it is important that AI fabric should provide high
effective bandwidth, so to shorten communication time and improve
computation efficiency. Effective bandwidth indicates fully
utilization of link bandwidth to achieve high throughput. Congestion
management is the key technology, including congestion control
mechanisms and load balancing mechanisms.
Lyu, et al. Expires 22 April 2024 [Page 2]
Internet-Draft CCM October 2023
This document discusses the uncoordinated mechanisms in current
congestion management. That leads to throughput issues which are
particularly harmful in AI fabric. A scheme for coordinating
different congestion management mechanisms is provided in this
document, which can be effectively and widely deployed in AI fabric.
2. Terminology
* ML: Machine Learning
* AI: Artificial Intelligence
* FLOPs: Floating-Point Operations
* ECN: Explicit Congestion Notification
* AR: Adaptive Routing
* DCQCN: Data center QCN [DCQCN]
* CNP: Congestion Notification Packet
* PLB: Protective Load Balancing [PLB]
* CC: Congestion Control
* ECMP: Equal-cost multi-path routing
* Incast congestion: the congesiton is caused by mutiple sources
sending traffic to the same destination simultaneously.
* Incast flow: the flow causing incast congestion
* Incast traffic: packets in incast flows
* CC congestion: the congestion is casued by incast or by high-speed
port sending traffic to low-speed port
* CC flow: the flow causing CC congestion
* CC traffic: packets in CC flows
Lyu, et al. Expires 22 April 2024 [Page 3]
Internet-Draft CCM October 2023
3. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
4. Existing congestion management
Congestion is usually caused by in-cast traffic and/or imbalanced
network load. Incast traffic is the traffic from multiple source
hosts, but towards to the same destination host. Commonly used
solutions include congestion control algorithms that control sending
rates and load balancing algorithms that adjust paths for traffic.
* The congestion control algorithm, such as DCQCN [DCQCN], Timely
[Timely], identifies network congestion by network status, like
queue length of switch port, end-to-end delay RTT, etc., then
adjust the sending rate at the sender to alleviate congestion.
How to quickly flatten down the rate curve to avoid packet loss
and how to recover the rate for less throughput reduction are
essential to congestion control mechanism in AI fabric.
* Adaptive routing is a way for load balancing. According to
network status, network switch dynamically change traffic path of
a flow in order to fully utilize network resource. Network status
could be indicated by local link status, downstream link status
etc. How to locate the proper new traffic path without back-and-
forth path switching is critical in AI fabric, because each path
switching may increase the systeme complexity, like re-ordering.
Currently, congestion control mechanism and adaptive routing work
independently, without coordination. That results in negative impact
on system performance. For example, when congestion caused by
imbalanced load on network occurs on a switch, both DCQCN and
adaptive routing are activated. ECN in data packets is marked
causing the CNP to be sent back to sender. Thus, sender slows down
the sending rate of the congested flow. Meanwhile, the switch
changes the path for congested flow, traversing the new incoming
packets to a light-loaded path. The result is that the congested
flow is forwarded on the light-loaded path at a low rate. Then,
DCQCN needs some time to recover the sending rate at the new path.
It reduces effective bandwidth and seriously impact computation
efficiency in AI training. Another example, if the congestion is
caused by in-cast traffic, congestion control should be enough.
Additional adaptive routing adjustments not only fail to mitigate
congestion, but may also introduce more out-of-order issue.
Lyu, et al. Expires 22 April 2024 [Page 4]
Internet-Draft CCM October 2023
It is shown that current congestion management cannot efficiently
handle congestion issue in AI fabric. Uncoordinated behaviors reduce
effective network bandwidth which is essential for AI workload.
5. Design principle of coordinated congestion management
Coordinated congestion management is designed to coordinate
congestion control and adaptive routing. Design principle is shown
as below.
* Avoid unnecessary sending rate reduction
AI fabric is bandwidth sensitive. High throughput is extremely
important. Multipath is needed to make full use of network
bandwidth. Slowing down the sending rate while there are still
available paths for traffic will be a waste of network resource,
thereby increasing communication time in AI cluster and reducing
AI training performance.
* Fully use multipath while reducing invalid path switching
While searching for light-loaded paths for load balancing, new
paths should be located quickly and accurately. The new path
should not be restricted to local paths but extends the search to
available paths upstream. Invalid path switching should be
avoided. Invalid path switching includes switching in-cast
traffic as no matter how to switch the traffic path, it will final
get congested on the last hop.
* Reuse current CC algorithm and AR algorithm
There are already a variety of CC algorithm and AR algorithms.
Those can still be used in the congestion management coordination
scheme. The scheme enables CC and AR be triggered coordinately,
adjusting sending rate or switching path depending on different
reasons of congestion.
* Applicable to various topologies
Most AI fabrics use CLOS or FATTREE topologies, but there are also
new studies considering the use of direct topologies, such as
torus, dragonfly, dragonfly+. Some of existing solutions for CC
and AR coordination, e.g PLB [PLB], relies on ECMP which can only
be used in topologies with equal cost paths like CLOS. For those
topologies without equal cost paths, like dragonfly+, such
solutions do not work. The coordination scheme should be
applicable to different topologies.
Lyu, et al. Expires 22 April 2024 [Page 5]
Internet-Draft CCM October 2023
6. Coordinated congestion management scheme
The key to the coordinated congestion management is to identify CC
traffic and non-CC traffic, thereby they are treated differently in
network when congestion occurs. CC flow recognized by network is
notified to the source host and the subsequent packets of the CC flow
are tagged by the source host. This indicates the network switch to
perform CC mechanism on the flow instead of AR. For non-CC traffic,
the network switch first performs AR. Only when AR mechansim cannot
find light-loaded path for switching, the traffic turns to be CC
traffic and CC will be run to alleviate congestion.
Coordinated congestion management requires interaction between
network switches and source hosts, and adds a new tag to data packets
for the coordination. The following sections explain the detail of
the scheme.
6.1. Coordination tag
Coordination tag is inserted into data packets. The tag contains CC
indicator and AR indicator.
* CC indicator: indicates if the packet belongs to a flow which
needs congestion control, such as incast flow .
* AR indicator: indicates the location of upstream AR point where
adaptive routing can be performed. The AR point can be a network
switch or a source host. AR indicator can be an ID, an IP address
or other information which guides how to send a message to the AR
point.
The tag can use in-band telemetry scheme to carry in data packet. A
new method CSIG [I-D.draft-ravi-ippm-csig] may provide another
possibility.
6.2. Notification message
There are 3 types of notification.
* Type 1: congestion control required
Example: Type 1 message is sent from incast congetion switch to
incast flow source host, notifying the source host to tag (set CC
indicator) the packets in the incast flow.
Lyu, et al. Expires 22 April 2024 [Page 6]
Internet-Draft CCM October 2023
* Type 2: congestion control released
Example: When incast congestion is eliminated, the switch sends
type 2 message to corresponding hosts, notfifying the source hosts
to untag CC indicator in the subsequent packets of the
corresponding flow.
* Type 3: upstream AR required
Example: If the switch determins to perform AR upstream, type 3
message is sent to the upstream AR point. The upstream AR point
can be one-hop neighbour of the switch or a point multi-hop away.
The notification message includes source IP, destination IP,
notification type and flow key. Source IP is the ip address of the
switch which sends the notification. Destination IP is the ip
address of the destination which will handle the notification
message. Notification type is one of the above 3 types. Flow key is
the information of the flow to be handled, such as 5-tuple
information.
6.3. Behavior of network switches
6.3.1. Identify congestion type
When congestion is detected, network switch judge whether it is CC
congestion or non-CC congestion. CC congestion includes incast
congestion and congestion caused by high-speed port sending traffic
to low-speed port.
If congestion occurs at the switch egress port, and the switch is the
last-hop switch to destination host, it is determined that the
congestion is incast congestion. The flows causing incast congestion
are identified as incast flow.
There may have other methods to identify congestion type. This
document does not make limitation on that.
6.3.2. Notify CC congestion
When CC congestion is determined by the network switch, it generates
type 1 notification messages for each identified CC flow, and sends
the notification messages to source hosts of the flows. When CC
congestion is eliminated, the switch sends type 2 notification
messages to the source hosts.
Lyu, et al. Expires 22 April 2024 [Page 7]
Internet-Draft CCM October 2023
6.3.3. Notify upstream point to perform AR
When it is determined to perform AR, but network switch cannot do it
locally and AR indicator in the data packet shows availability to do
AR upstream, a type 3 notification message is sent to upstream point
according to AR indicator.
6.3.4. Perform congestion control
Network switch performs congestion control in below cases.
* It is identified as CC congestion.
* It is identified as non-CC congestion, but adaptive routing cannot
be used because there is no available new path for traffic
switching either locally or upstream.
This document does not limit which CC mechanism is performed.
6.3.5. Perform adaptive routing
Network switch performs adaptive routing in below cases.
* The flow is non-CC traffic. CC indicator in data packet is used
to determine if it is CC traffic or non-CC traffic.
* Type 3 notification message is received. According to flow
information in the notification, new path is selected for the
subsequent packets of the flow.
In order to enable upstream AR, it is required to update AR indicator
in data packets hop by hop. When a data packet arrives at the
network switches,
* if there are several local light-loaded paths available for AR on
the switch, the switch updates AR indicator in the data packet to
itself, such as its own ID. Then the switch selects the
appropriate local path to send the data packet. This document
does not define algorithm of local path selection. It depends on
routing strategy on the network switch.
* If there is only one local light-loaded path available for AR,
network switch can only select that path for traffic. AR
indicator in the data packet will not be updated.
* If there is no local light-loaded path, network switch gets
upstream AR availability by reading AR indicator in the data
packet. If AR indicator indicates upstream point can perform AR,
Lyu, et al. Expires 22 April 2024 [Page 8]
Internet-Draft CCM October 2023
network switch generates type 3 notification message and sends it
directly to the corresponding upstream point. Otherwise, network
switch triggers congestion control mechanism, such as set ECN in
data packet.
6.4. Behavior of source hosts
When receiving type 1 notification message, source host sets CC
indicator of the subsequent packets for the corresponding flow.
When receiving type 2 notificiation message, source host unset CC
indicator of the subsequent packets for the corresponding flow.
When receiving type 3 notification message, source host performs AR
on the subsequent packets for the corresponding flow.
When receiving congestion control signals and the CC indicator is
set, source host performs CC on the flow.
7. An example of end-to-end procedure
Network topology is shown in Figure 1. This is a 4 layer fattree
topology. There are n computing racks and m switching racks.
Computing racks have source hosts, layer 1 switches and layer 2
switches. Swithcing racks contain layer 3 and layer 4 switches.
Lyu, et al. Expires 22 April 2024 [Page 9]
Internet-Draft CCM October 2023
Switching Rack 1 Switching Rack m
+---------------+ +---------------+
|L4-1-1...L4-1-e| |L4-m-1...L4-m-e|
| | \ / | | | | \ / | |
| | \ / | | | | \ / | |
| | \/ | | | | \/ | |
| | /\ | |...| | /\ | |
| | / \ | | | | / \ | |
| | / \ | | | | / \ | |
|L3-1-1...L3-1-d| |L3-m-1...L3-m-d|
+--+-----------\ +-/----------+--+
| \ / |
| \ / |
| ...... \/ ...... |
| /\ |
| / \ |
| / \ |
+--+-----------/ \----------+---+
|L2-1-1...L1-1-c| |L2-n-1...L2-n-c|
| | \ / | | | | \ / | |
| | \ / | | | | \ / | |
| | \/ | | | | \/ | |
| | /\ | |... | | /\ | |
| | / \ | | | | / \ | |
| | / \ | | | | / \ | |
|L1-1-1...L1-1-b| |L1-n-1...L1-n-b|
| + + | | + + |
| H-1-1... H-1-a| | H-n-1... H-n-a|
+---------------+ +---------------+
Computing Rack 1 Computing Rack n
Figure 1: Network Topology
* Host H-1-1 in computing rack 1sends out a data packet P1 belonging
to flow F1 to H-n-1 in computing rack n. The value of CC
indicator in the packet tag is not set indicating this packet is
in a non-incast flow. The AR indicator in the packet tag does not
point to any available AR point.
* P1 arrives at switch L1-1-1 in computing rack 1. L1-1-1 has
multiple light-loaded paths for AR. Path from L1-1-1 to L2-1-1 is
selected for P1. AR indicator in P1 tag is updated to L1-1-1.
* P1 arrives at switch L2-1-1. L2-1-1 also has multiple light-
loaded paths for AR. Path from L2-1-1 to L3-1-1 is selected for
P1. AR indicator in P1 tag is updated to L2-1-1.
Lyu, et al. Expires 22 April 2024 [Page 10]
Internet-Draft CCM October 2023
* P1 arrives at switch L3-1-1. L3-1-1 only has one light-loaded
paths. The only path from L3-1-1 to L4-1-1 is selected for P1.
AR indicator in P1 tag keeps to be L2-1-1.
* P1 arrives at switch L4-1-1. L4-1-1 is congested and no local
path available for performing AR. By reading AR indicator in P1,
L4-1-1 sends an type 3 notification to L2-1.
* After receiving AR notification, L2-1-1 switches path from
L2-1-1->L3-1-1 to L2-1-1->L3-m-1 for the new incoming packets of
flow F1.
* After a while, L1-n-1 is congested due to incast. The flow F1 is
identified as incast flow. L1-n-1 sends type 1 notification to
H-1-1.
* By receiving the type 1notification, H-1-1 sets CC indicator of
the subsequent packets of F1 indicating the packets are in a
incast flow. Thus those packets will not be performed AR.
Sending rate of F1 will also be reduced according to congestion
control algorithm.
8. Security Considerations
TBD.
9. IANA Considerations
TBD.
10. References
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.
10.2. Informative References
[I-D.draft-ravi-ippm-csig]
Ravi, A., Dukkipati, N., Mehta, N., and J. Kumar,
"Congestion Signaling (CSIG)", Work in Progress, Internet-
Lyu, et al. Expires 22 April 2024 [Page 11]
Internet-Draft CCM October 2023
Draft, draft-ravi-ippm-csig-00, 31 August 2023,
<https://datatracker.ietf.org/doc/html/draft-ravi-ippm-
csig-00>.
[DCQCN] "Congestion Control for Large-Scale RDMA Deployments",
August 2015,
<https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/
p523.pdf>.
[Timely] "TIMELY: RTT-based Congestion Control for the Datacenter",
August 2015,
<https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/
p537.pdf>.
[PLB] "PLB: Congestion Signals are Simple and Effective for
Network Load Balancing", August 2022,
<https://dl.acm.org/doi/pdf/10.1145/3544216.3544226>.
Authors' Addresses
Yunping(Lily) Lyu
Huawei
Email: lvyunping@huawei.com
Yuhan Zhang
Huawei
Email: zhangyuhan6@huawei.com
Mengzhu Liu
Huawei
Email: liumengzhu@huawei.com
Lyu, et al. Expires 22 April 2024 [Page 12]
