Internet DRAFT - draft-wh-rtgwg-application-aware-dc-network

draft-wh-rtgwg-application-aware-dc-network







Network Working Group                                            H. Wang
Internet-Draft                                                    Huawei
Intended status: Standards Track                                  K. Yao
Expires: 2 September 2024                                   China Mobile
                                                                  W. Pan
                                                                H. Huang
                                                                  Huawei
                                                            1 March 2024


Application-aware Data Center Network (APDN) Use Cases and Requirements
             draft-wh-rtgwg-application-aware-dc-network-02

Abstract

   The deployment of large-scale AI services within data centers
   introduces significant challenges to established technologies,
   including load balancing and congestion control.  Additionally, the
   adoption of cutting-edge network technologies, such as in-network
   computing, is on the rise within AI-centric data centers.  These
   advanced network-assisted application acceleration technologies
   necessitate the flexible exchange of cross-layer interaction
   information between end-hosts and network nodes.

   The Application-aware Data Center Network (APDN) leverages the
   Application-aware Networking (APN) framework for application side to
   furnish the data center network with detailed application-aware
   information.  This approach facilitates the rapid advancement of
   network-application co-design technologies.  This document delves
   into the use cases of APDNs and outlines the associated requirements,
   setting the stage for enhanced performance and efficiency in data
   center operations tailored to the demands of AI services.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."




Wang, et al.            Expires 2 September 2024                [Page 1]

Internet-Draft                    APDN                        March 2024


   This Internet-Draft will expire on 2 September 2024.

Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   4
     1.2.  Requirements Language . . . . . . . . . . . . . . . . . .   4
   2.  Use Case and Requirements for Application-aware Data Center
           Network . . . . . . . . . . . . . . . . . . . . . . . . .   4
     2.1.  Fine-grained packet scheduling for load balancing . . . .   4
     2.2.  Enhancing Distributed Machine Learning Training with
           In-Network Computing  . . . . . . . . . . . . . . . . . .   6
     2.3.  Enhanced Congestion Control with Precise Feedback
           Mechanisms  . . . . . . . . . . . . . . . . . . . . . . .   8
   3.  Encapsulation . . . . . . . . . . . . . . . . . . . . . . . .   9
   4.  Security Considerations . . . . . . . . . . . . . . . . . . .   9
   5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   9
   6.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
     6.1.  Normative References  . . . . . . . . . . . . . . . . . .   9
     6.2.  Informative References  . . . . . . . . . . . . . . . . .  10
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  11
   Contributors  . . . . . . . . . . . . . . . . . . . . . . . . . .  11
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  11

1.  Introduction

   The advent of large AI models like AlphaGo and ChatGPT4 has
   positioned distributed training for AI large models as a pivotal
   operation within large-scale data centers.  To enhance the efficiency
   of training these substantial models, a significant number of
   computing units—such as thousands of GPUs operating in tandem—are
   deployed for parallel processing, aiming to minimize job completion
   time (JCT).  This setup necessitates frequent and bandwidth-heavy
   communications among concurrent computing nodes, introducing a novel



Wang, et al.            Expires 2 September 2024                [Page 2]

Internet-Draft                    APDN                        March 2024


   multi-party communication mode that demands heightened throughput
   performance, load balancing proficiency, and congestion management
   capabilities from the data center network.

   Traditionally, data center technology primarily views the network as
   a mere conduit for data transmission for upper-layer applications,
   offering basic connectivity services.  Yet, the scenario of large AI
   model training is increasingly incorporating network-assisted
   technologies, such as offloading parts of the computation to the
   network.  This approach seeks to boost AI job efficiency through the
   joint optimization of network communication and computing
   applications.  In many current instances of network assistance,
   operators tailor and implement proprietary protocols on a limited
   scale, leading to a lack of widespread interoperability.

   However, as AI data centers grow and diversify in offering cloud
   services for various AI tasks, emerging data center network
   technologies must account for serving different transports and
   applications.  Building large-scale data centers now involves not
   just ensuring device interoperability but also facilitating
   interaction between network devices and end-host services.

   This document illustrates use cases that requires application-aware
   information between network nodes and applications.  Current ways of
   conveying information are limited by the extensibility of packet
   headers, where only coarse-grained information can be transmitted
   between the network and the host through a limited space (for
   example, one-bit ECN [RFC3168] or DSCP in IP layer).

   The Application-aware Networking (APN) framework
   [I-D.li-apn-framework] delineates how application-aware information,
   including APN identification (ID) and/or parameters (e.g., network
   performance requirements), is encapsulated by network edge devices.
   This information is then carried in packets across an APN domain to
   support service provisioning, enable fine-grained traffic steering,
   and adjust network resources.  An extension of the APN framework
   caters to the application side [I-D.li-rtgwg-apn-app-side-framework],
   allowing APN domain resources to be allocated to applications that
   encapsulate the APN attribute in packets.

   This document delves into the application side of the APN framework
   to foster enriched interaction between hosts and networks within the
   data center, outlining several use cases and the corresponding
   requirements for Application-aware Data center Network (APDN).







Wang, et al.            Expires 2 September 2024                [Page 3]

Internet-Draft                    APDN                        March 2024


1.1.  Terminology

   APDN: APplication-aware Data center Network

   SQN: SeQuence Number

   TOR: Top Of Rack switch

   PFC: Priority-based Flow Control

   NIC: Network Interface Card

   ECMP: Equal-Cost Multi-Path routing

   AI: Artificial Intelligence

   JCT: Job Completion Time

   PS: Parameter Server

   INC: In-Network Computing

   APN: APplication-aware Network

1.2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  Use Case and Requirements for Application-aware Data Center Network

2.1.  Fine-grained packet scheduling for load balancing

   Traditional data centers utilize the per-flow Equal-Cost Multi-Path
   (ECMP) method to distribute traffic evenly across several paths.
   These centers, primarily focused on cloud computing, handle a vast
   number of data flows.  Despite the large quantity, these flows are
   predominantly small and short-lived, allowing the ECMP method to
   facilitate a nearly uniform traffic distribution across multiple
   pathways.








Wang, et al.            Expires 2 September 2024                [Page 4]

Internet-Draft                    APDN                        March 2024


   Contrastingly, the communication dynamics shift markedly during the
   training of large AI models.  This process demands unprecedented
   bandwidth levels, where a singular data flow between multiple
   machines could potentially max out the upstream bandwidth of a
   server’s egress Network Interface Controller (NIC), with single data
   flow throughputs approaching or exceeding 100GB x X.

   Applying traditional per-flow ECMP strategies, such as hash-based or
   round-robin algorithms, often results in the concurrent allocation of
   large ("elephant") flows to a single pathway.  This can lead to
   severe congestion, notably when two simultaneous 100Gb/s flows vie
   for the same 100Gb/s bandwidth, significantly impacting the
   completion time for AI jobs.

   To mitigate these issues, there's a pivotal shift towards
   implementing a fine-grained, per-packet ECMP strategy.  This approach
   ensures the distribution of all packets from a single flow across
   multiple paths, enhancing balance and preventing congestion.
   However, due to the varying delays (propagation and switching) across
   these paths, such a strategy may result in significant packet
   disorder upon arrival at the destination, thereby degrading the
   performance of both transport and application layers.

   A viable solution is the resequencing of out-of-order packets at the
   egress Top-of-Rack (ToR) switch, employing per-packet ECMP.  This
   assumes multipath transmission extends from ingress to egress ToRs,
   with the reordering principle ensuring that the packet departure
   sequence from the last ToR mirrors the arrival sequence at the first
   ToR.

   Achieving packet reordering at the egress ToR necessitates a clear
   indication of packet arrival sequences at the ingress ToR.  Current
   protocols do not directly mark sequence numbers (SQNs) at the
   Ethernet and IP layers.

   *  Presently, SQNs are encapsulated within transport layers (e.g.,
      TCP, QUIC, RoCEv2) or application protocols.  Relying on these
      SQNs for packet reordering requires network devices to interpret a
      vast array of transport/application layer information.

   *  SQNs at the transport/application layer are allocated per flow,
      with each having distinct sequence number spaces and initial
      values.  These cannot directly represent the packet arrival
      sequence at the initial ToR.  Although assigning a specific
      reordering queue to each flow at the egress ToR and reordering
      based on upper-layer SQNs is conceivable, the associated hardware
      resource demands are significant.




Wang, et al.            Expires 2 September 2024                [Page 5]

Internet-Draft                    APDN                        March 2024


   *  Direct modification of upper-layer SQNs by network devices to
      reflect ToR-ToR pairwise SQNs compromises end-to-end transmission
      reliability.

   Consequently, a mechanism to convey specific order information across
   the multipath forwarding domain, from the initial to the final device
   with reordering capabilities, is essential.

   The Application-aware Networking (APN) framework is proposed to
   transport critical ordering information.  In this context, it records
   the sequence number of packets as they arrive at the ingress ToR
   (each ToR-ToR pair having a unique, incremental SQN), facilitating
   packet reordering by the egress ToR based on this data.

   Requirements:

   *  [REQ1-1] The APN framework SHOULD tag each packet with an SQN
      alongside the APN ID to enable reordering.  The ingress ToR SHOULD
      assign and log an SQN for each packet based on its arrival
      sequence, with SQN granularity adaptable to ToR-ToR, port-port, or
      queue-queue levels.

   *  [REQ1-2] The APN-encapsulated SQN MUST remain unaltered within the
      multipathing domain and may be removed at the egress device.

   *  [REQ1-3] The APN framework SHOULD convey necessary queue
      information (i.e., the sorting queue ID) to support fine-grained
      reordering.  The queue ID SHOULD match the granularity of SQN
      assignments.  Additionally, the APN framework COULD transport path
      details to expedite the differentiation between out-of-order
      packets and packet loss.

2.2.  Enhancing Distributed Machine Learning Training with In-Network
      Computing

   Distributed machine learning training frequently employs the
   AllReduce communication mode[mpi-doc] for efficient cross-accelerator
   data transfer.  This method is pivotal in scenarios involving data
   and model parallelism, where parallel execution across multiple
   processors necessitates the exchange of intermediate results, such as
   gradient data, as a core component of the communication process.

   The Parameter Server (PS) architecture[atp], which centralizes
   gradient data aggregation through a server from multiple clients and
   redistributes the aggregated results, often faces incast congestion
   challenges due to simultaneous large-volume data transmissions to the
   server.




Wang, et al.            Expires 2 September 2024                [Page 6]

Internet-Draft                    APDN                        March 2024


   In-network computing (INC) introduces a paradigm shift by delegating
   the server's processing tasks to network switches.  Utilizing network
   devices equipped with high-capacity switching and computational
   abilities (for basic arithmetic operations) as surrogate parameter
   servers for gradient aggregation enables the consolidation of
   multiple data streams into a singular network stream.  This approach
   not only alleviates server-side incast congestion but also leverages
   the superior speed of on-switch computing (e.g., ASICs) over
   traditional server-based processing (e.g., CPUs), offering a boon to
   distributed computing applications.

   As outlined in [I-D.draft-lou-rtgwg-sinc], the realization of INC
   requires network devices to comprehend the computing tasks dictated
   by applications, including the accurate parsing of relevant data
   units and the coordination of synchronization signals across diverse
   data sources.

   Present implementations like ATP[atp] and NetReduce[netreduce]
   necessitate that switches interpret upper-layer protocols and
   application-specific logic, which remains tailored to particular
   applications due to the absence of standardized transport or
   application protocols for INC.  To accommodate a broad spectrum of
   INC applications, network devices must exhibit versatility across
   various protocol formats.

   Moreover, while end users may encrypt payloads for security, they
   might be inclined to expose certain non-sensitive data to benefit
   from accelerated INC operations.  However, the current protocol
   landscape does not facilitate easy access to necessary INC data
   without decrypting the entire payload, posing interoperability
   challenges between applications and INC functionalities.

   The Application-aware Networking (APN) framework emerges as a
   solution, capable of conveying essential information for INC tasks
   and their associated data segments, thereby enabling the offloading
   of specific computational tasks to the network.

   _Requirements_:

   *  [REQ2-1] The APN framework MUST include identifiers to
      differentiate among INC tasks.

   *  [REQ2-2] The APN framework MUST accommodate the transport of
      application data in varied formats and lengths, such as gradient
      data for INC, along with the specified operations.






Wang, et al.            Expires 2 September 2024                [Page 7]

Internet-Draft                    APDN                        March 2024


   *  [REQ2-3] To augment INC efficiency, the APN framework SHOULD
      transmit additional application-aware information to support
      computational processes without undermining end-to-end transport
      reliability.

   *  [REQ2-4] The APN framework MUST have the capability to convey
      comprehensive INC outcomes and document the computational status
      within data packets.

2.3.  Enhanced Congestion Control with Precise Feedback Mechanisms

   Data center environments encompass various congestion scenarios,
   notably:

   *  The prevalent use of multi-accelerator collaborative AI model
      training, employing AllReduce and All2All communication patterns
      (Section 2.2), often leads to server-side incast congestion as
      multiple clients simultaneously transmit substantial volumes of
      gradient data.

   *  Diverse load balancing methodologies across different flows can
      induce overload conditions on specific links.

   *  The inherent randomness of service access within data centers
      frequently triggers traffic bursts, extending queue lengths and
      precipitating congestion.

   To mitigate these challenges, the industry has developed an array of
   congestion control algorithms tailored for data center networks.
   ECN-based congestion control mechanisms, such as DCTCP[RFC8257] and
   DCQCN[dcqcn], leverage ECN marks based on switch buffer occupancy
   levels to signal congestion.

   However, these approaches are constrained by the use of a singular
   1-bit mark within packet headers to denote congestion, limiting the
   scope of conveyed congestion details due to header space
   restrictions.  Alternative strategies, such as HPCC++
   [I-D.draft-miao-ccwg-hpcc], adopt in-band telemetry to cumulatively
   append congestion data at each hop, increasing packet length and
   bandwidth consumption.











Wang, et al.            Expires 2 September 2024                [Page 8]

Internet-Draft                    APDN                        March 2024


   A compromise solution, AECN[I-D.draft-shi-ippm-advanced-ecn],
   endeavors to encapsulate critical congestion indicators along the
   path while minimizing data overhead through hop-by-hop aggregation,
   including queue delay and congested hop counts.  This model allows
   end-hosts to specify the congestion metrics of interest, with network
   devices incrementally compiling this data en route.  APN frameworks
   can facilitate this nuanced exchange, enabling tailored congestion
   data accumulation.

   _Requirements_:

   *  [REQ3-1] The APN framework MUST empower data senders to specify
      the congestion metrics they wish to gather.

   *  [REQ3-2] The APN framework MUST enable network nodes to log and
      update selected measurements accordingly.  This may encompass
      metrics such as port queue lengths, link monitoring rates, PFC
      frame counts, probed RTTs, and variability, among others.
      Additionally, the APN MAY tag each measurement with its collector,
      assisting in the identification of potential congestion points.

3.  Encapsulation

   The encapsulation of application-aware information proposed by use
   cases of APDN in the APN Header [I-D.draft-li-apn-header] will be
   defined in the future version of the draft.

4.  Security Considerations

   TBD.

5.  IANA Considerations

   This document has no IANA actions.

6.  References

6.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.




Wang, et al.            Expires 2 September 2024                [Page 9]

Internet-Draft                    APDN                        March 2024


6.2.  Informative References

   [mpi-doc]  "Message-Passing Interface Standard", August 2023,
              <https://www.mpi-forum.org/docs/mpi-4.1>.

   [dcqcn]    "Congestion Control for Large-Scale RDMA Deployments",
              n.d.,
              <https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/
              p523.pdf>.

   [netreduce]
              "NetReduce - RDMA-Compatible In-Network Reduction for
              Distributed DNN Training Acceleration", n.d.,
              <https://arxiv.org/abs/2009.09736>.

   [atp]      "ATP - In-network Aggregation for Multi-tenant Learning",
              n.d.,
              <https://www.usenix.org/conference/nsdi21/presentation/
              lao>.

   [I-D.li-apn-framework]
              Li, Z., Peng, S., Voyer, D., Li, C., Liu, P., Cao, C., and
              G. S. Mishra, "Application-aware Networking (APN)
              Framework", Work in Progress, Internet-Draft, draft-li-
              apn-framework-07, 3 April 2023,
              <https://datatracker.ietf.org/doc/html/draft-li-apn-
              framework-07>.

   [I-D.li-rtgwg-apn-app-side-framework]
              Li, Z. and S. Peng, "Extension of Application-aware
              Networking (APN) Framework for Application Side", Work in
              Progress, Internet-Draft, draft-li-rtgwg-apn-app-side-
              framework-00, 22 October 2023,
              <https://datatracker.ietf.org/doc/html/draft-li-rtgwg-apn-
              app-side-framework-00>.

   [I-D.draft-lou-rtgwg-sinc]
              Lou, Z., Iannone, L., Li, Y., Zhangcuimin, and K. Yao,
              "Signaling In-Network Computing operations (SINC)", Work
              in Progress, Internet-Draft, draft-lou-rtgwg-sinc-01, 15
              September 2023, <https://datatracker.ietf.org/doc/html/
              draft-lou-rtgwg-sinc-01>.

   [RFC8257]  Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L.,
              and G. Judd, "Data Center TCP (DCTCP): TCP Congestion
              Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257,
              October 2017, <https://www.rfc-editor.org/rfc/rfc8257>.




Wang, et al.            Expires 2 September 2024               [Page 10]

Internet-Draft                    APDN                        March 2024


   [I-D.draft-miao-ccwg-hpcc]
              Miao, R., Anubolu, S., Pan, R., Lee, J., Gafni, B.,
              Tantsura, J., Alemania, A., and Y. Shpigelman, "HPCC++:
              Enhanced High Precision Congestion Control", Work in
              Progress, Internet-Draft, draft-miao-ccwg-hpcc-02, 29
              February 2024, <https://datatracker.ietf.org/doc/html/
              draft-miao-ccwg-hpcc-02>.

   [I-D.draft-shi-ippm-advanced-ecn]
              Shi, H., Zhou, T., and Z. Li, "Advanced Explicit
              Congestion Notification", Work in Progress, Internet-
              Draft, draft-shi-ippm-advanced-ecn-00, 11 December 2023,
              <https://datatracker.ietf.org/doc/html/draft-shi-ippm-
              advanced-ecn-00>.

   [I-D.draft-li-apn-header]
              Li, Z., Peng, S., and S. Zhang, "Application-aware
              Networking (APN) Header", Work in Progress, Internet-
              Draft, draft-li-apn-header-04, 12 April 2023,
              <https://datatracker.ietf.org/doc/html/draft-li-apn-
              header-04>.

Acknowledgements

Contributors

Authors' Addresses

   Haibo Wang
   Huawei
   Email: rainsword.wang@huawei.com


   Kehan Yao
   China Mobile
   Email: yaokehan@chinamobile.com


   Wei Pan
   Huawei
   Email: tarzan.pan@huawei.com


   Hongyi  Huang
   Huawei
   Email: hongyi.huang@huawei.com





Wang, et al.            Expires 2 September 2024               [Page 11]