Internet DRAFT - draft-yao-cats-ps-usecases

draft-yao-cats-ps-usecases







cats                                                              K. Yao
Internet-Draft                                              China Mobile
Intended status: Informational                                D. Trossen
Expires: 1 January 2024                              Huawei Technologies
                                                            M. Boucadair
                                                                  Orange
                                                           LM. Contreras
                                                              Telefonica
                                                                  H. Shi
                                                                   Y. Li
                                                     Huawei Technologies
                                                                S. Zhang
                                                            China Unicom
                                                            30 June 2023


 Computing-Aware Traffic Steering (CATS) Problem Statement, Use Cases,
                            and Requirements
                     draft-yao-cats-ps-usecases-03

Abstract

   Distributed computing is a tool that service providers can use to
   achieve better service response time and optimized energy
   consumption.  In such a distributed computing environment, providing
   services by utilizing computing resources hosted in various computing
   facilities aids support of services such as computationally intensive
   and delay sensitive services.  Ideally, compute services are balanced
   across servers and network resources to enable higher throughput and
   lower response times.  To achieve this, the choice of server and
   network resources should consider metrics that are oriented towards
   compute capabilities and resources instead of simply dispatching the
   service requests in a static way or optimizing solely on connectivity
   metrics.  The process of selecting servers or service instance
   locations, and of directing traffic to them on chosen network
   resources is called "Computing-Aware Traffic Steering" (CATS).

   This document provides the problem statement and the typical
   scenarios for CATS, which shows the necessity of considering more
   factors when steering traffic to the appropriate computing resource
   to best meet the customer's expectations and deliver the requested
   service.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.




Yao, et al.              Expires 1 January 2024                 [Page 1]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 1 January 2024.

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Definition of Terms . . . . . . . . . . . . . . . . . . . . .   4
   3.  Problem Statement . . . . . . . . . . . . . . . . . . . . . .   5
     3.1.  Multi-deployment of Edge Sites and Service  . . . . . . .   5
     3.2.  Traffic Steering among Edges Sites and Service
           Instances . . . . . . . . . . . . . . . . . . . . . . . .   6
   4.  Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . .   9
     4.1.  Computing-Aware AR or VR  . . . . . . . . . . . . . . . .  10
     4.2.  Computing-Aware Intelligent Transportation  . . . . . . .  13
     4.3.  Computing-Aware Digital Twin  . . . . . . . . . . . . . .  14
     4.4.  Computing-Aware SD-WAN  . . . . . . . . . . . . . . . . .  15
   5.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .  17
     5.1.  Support dynamic and effective selection among multiple
           serivce instances . . . . . . . . . . . . . . . . . . . .  17
     5.2.  Support Agreement on Metric Representation  . . . . . . .  17
     5.3.  Support Moderate Metric Distributing  . . . . . . . . . .  18
     5.4.  Support Flexible Use of Metrics . . . . . . . . . . . . .  18
     5.5.  Support Session and Service Continuity  . . . . . . . . .  19
     5.6.  Preserve Communication Confidentiality  . . . . . . . . .  21
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  21



Yao, et al.              Expires 1 January 2024                 [Page 2]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  21
   8.  Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  22
   9.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  22
   10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  22
     10.1.  Normative References . . . . . . . . . . . . . . . . . .  22
     10.2.  Informative References . . . . . . . . . . . . . . . . .  23
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  23

1.  Introduction

   Network and computing convergence has been evolving in the Internet
   for considerable time.  With Content Delivery Networks (CDNs)
   'frontloading' access to many services, over-the-top service
   provisioning has become a driving force for many services, such as
   video, storage and many others.  Network operators have extended
   their capabilities by complementing their network infrastructure by
   developing CDN capabilities, particularly in edge sites.  In
   addition, more computing resource are deployed at these edge sites as
   well.

   The reason of the fast development of this converged network/compute
   infrastructure is user demand.  On the one hand, users want the best
   experience, e.g., expressed in low latency and high reliability, for
   new emerging applications such as high-definition video, Augmented
   Reality(AR)/Virtual Reality(VR), live broadcast and so on.  On the
   other hand, users want stable experience when moving to different
   areas.

   Generally, edge computing aims to provide better response times and
   transfer rates compared to cloud computing, by moving the computing
   towards the edge of a network.  There are millions of home gateways,
   thousands of base stations, and hundreds of central offices in a city
   that could serve as compute-capable nodes to deliver a service.  Note
   that not all of these nodes would be considered as edge nodes in some
   views of the network, but they can all provide computing resources to
   enable a service.

   That brings about the key problem of deploying and scheduling traffic
   to the most suitable computing resource in order to meet the users'
   service demand.

   Depending on the locations of computing resource and their capacity,
   different amounts of resource from different locations can be used to
   deliver a service.  At peak hours, computing resources closest to a
   client might not be sufficient to handle all the incoming service
   requests.  Longer response times or even dropping of requests could
   be experienced by users.  Increasing the computing resources hosted
   at each location to the potential maximum capacity is neither



Yao, et al.              Expires 1 January 2024                 [Page 3]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   feasible nor economically viable in many cases.  Offloading
   computation intensive processing to the user devices would place huge
   pressure on local resources such as the battery, and the needed data
   set (for the computation) that may not exist on the user device
   because of the size of data pool or due to data governance reasons.

   Service providers often have their own server sites, many of which
   have been enhanced to support computing services.  A service instance
   deployed at a single site might not provide sufficient capacity to
   fully guarantee the quality of service required by a customer.
   Instead, the same service can be deployed at multiple sites for
   better availability and scalability.  Furthermore, it is desirable to
   balance the load across all service instances to improve throughput.
   For this, traffic needs to be steered to the 'best' service instance
   according to information that may include current computing load,
   where the notion of 'best' may highly depend on the application
   demands.

   A particular example is the popular and pervasive 5G Mobile Edge
   Computing(MEC) service.  In 5G MEC, the Uplink Classifier(UL-CL)
   functionality of User Plane Functions(UPFs) are deployed close to
   edge sites, which are capable of effectively classifying & switching
   uplink traffic to the suitable computing-resources that might be
   located either in local-area Data Network(DN), operators' DN, or even
   3rd-party's DN.  Through possibly using some 'intelligent' criteria,
   this could warrant the selection of resources with either low, high-
   computational power or all-involved requirements.

   This document describes sample usage scenarios that drive CATS
   requirements and will help to identify candidate solution
   architectures and solutions.

2.  Definition of Terms

   This document makes use of the following terms:

   Service:  An offering provided by a service provider, similar to the
     notion of a 'service function' in [RFC7665], which may or may not
     be of composite nature but appears in the problem space of CATS as
     a single service to which traffic needs to be steered.

   Service instance:  A run-time environment (e.g., a server or a
     process on a server) that makes a service available.  A particular
     service could be made available at multiple service instances at
     the same or different locations.

   Service identifier:  Used to uniquely identify a service, at the same




Yao, et al.              Expires 1 January 2024                 [Page 4]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


     time identifying the whole set of service instances that each
     represent the same service behavior, no matter where those service
     instances are running.

   Even though this document is not a protocol specification, it makes
   use of upper case key words to define requirements unambiguously.
   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Problem Statement

3.1.  Multi-deployment of Edge Sites and Service

   Since edge computing aims at a closer computing service based on the
   shorter network path, there will be more than one edge site with the
   same application in the city/province/state, a number of
   representative cities have deployed multi-edge sites and the typical
   applications, and there are more edge sites to be deployed in the
   future.  Before deploying edge sites, there are some factors that
   need to be considered, such as:

   o The exsiting infrastructure capacities, which could be updated to
   edge sites, e.g. operators' machine room.

   o The amount and frequency of computing resource that is needed.

   o The network resource status linked to computing resource.

   To improve the effectiveness of service deployment, the problem of
   how to choose the optimal edge node on which to deploy services needs
   to be solved.  [I-D.contreras-alto-service-edge] introduces
   considerations for how to deploy applications or functions to the
   edge, such as the type of instance, optional storage extension,
   optional hardware acceleration characteristics, and the compute
   flavor of CPU/GPU, etc.  More network and service factors may also be
   considered, such as:

   o Network and computing resource topology: The overall consideration
   of network access, connectivity, path protection or redundancy, and
   the location and overall distribution of computing resources in the
   network, and the relative position within the network topology.







Yao, et al.              Expires 1 January 2024                 [Page 5]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   o Location: The number of users, the differentiation of service
   types, and the number of connections requested by users, etc.  For
   edge nodes located in populous area with a large number of users and
   service requests, service duplication could be deployed more than in
   other areas.

   o Capacity of multiple edge nodes: Not only the capacity of a single
   node, but also the total number of requests that can be processed by
   the resource pool composed of multiple nodes.

   o Service category: For example, whether the service is a multi-user
   interaction, such as video conferencing, or games, or whether it just
   resource acquisition, such as video viewing.  ALTO [RFC7285] can help
   to obtain one or more of the above pieces of information, so as to
   provide suggestions or formulate principles and strategies for
   service deployment.

   This information could be collected periodically, and could record
   the total consumption of computing resources, or the total number of
   sessions accessed.  This would indicate whether additional service
   instances need to be deployed.  Unlike the scheduling of service
   requests, service deployment should follow the principle of proximity
   to place new service instances near to customer sites that will
   request them.  If the resources are insufficient to support new
   instances, the operator can be informed to increase the hardware
   resources.

   In general, the choice of where to locate service instances and when
   to create new ones in order to provide the right levels of resource
   to support user demands is important in building a network that
   supports computing services.  However, those aspects are out of scope
   for CATS and are left for consideration in another document.

3.2.  Traffic Steering among Edges Sites and Service Instances

   This section describes how existing edge computing systems do not
   provide all of the support needed for real-time or near-real-time
   services, and how it is necessary to steer traffic to different sites
   considering mobility of people, different time slots, events, server
   loads, and network capabilities, etc.

   In edge computing, the computing resources and network resources are
   considered when deploying edge sites and services.  Traffic is
   steered to an edge site that is "closest" or to one of a few "close"
   sites using load-balancing.  But the "closest" site is not always the
   "best" as the status of computing resources and of the network may
   vary as follows:




Yao, et al.              Expires 1 January 2024                 [Page 6]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   o Closest site may not have enough resource, the load may dynamically
   change.

   o Closest site may not have related resource, heterogeneous hardware
   in different sites.

   o The network path to the closest site might not provide the
   necessary network characteristics, such as low latency or high
   throughput.

   To address these issues some enhancements are needed to steer traffic
   to sites that can support the requested services.

   We assume that clients access one or more services with an objective
   to meet a desired user experience.  Each participating service may be
   realized at one or more places in the network (called, service
   instances).  Such service instances are instantiated and deployed as
   part of the overall service deployment process, e.g., using existing
   orchestration frameworks, within so-called edge sites, which in turn
   are reachable through a network infrastructure via an edge router.

   When a client issues a service request for a required service, the
   request is steered to one of the available service instances.  Each
   service instance may act as a client towards another service, thereby
   seeing its own outbound traffic steered to a suitable service
   instance of the request service and so on, achieving service
   composition and chaining as a result.

   The aforementioned selection of one of candidate service instances is
   done using traffic steering methods, where the steering decision may
   take into account pre-planned policies (assignment of certain clients
   to certain service instances), realize shortest-path to the 'closest'
   service instance, or utilize more complex and possibly dynamic metric
   information, such as load of service instances, latency experienced
   or similar, for a more dynamic selection of a suitable service
   instance.

   It is important to note that clients may move.  This means that the
   service instance that was "best" at one moment might no longer be
   best when a new service request is issued.  This creates a (physical)
   dynamicity that will need to be catered for in addition to the
   changes in server and network load.









Yao, et al.              Expires 1 January 2024                 [Page 7]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   Figure 1 shows a common way to deploy edge sites in the metro.  There
   is an edge data center for metro area which has high computing
   resource and provides the service to more User Equipments(UEs) at the
   working time.  Because more office buildings are in the metro area.
   And there are also some remote edge sites which have limited
   computing resource and provide the service to the UEs closed to them.

   Applications to meet service demands could be deployed in both the
   edge data center in metro area and the remote edge sites.  In this
   case, the service request and the resource are matched well.  Some
   potential traffic steering may be needed just for special service
   request or some small scheduling demand.

        +----------------+    +---+                  +------------+
      +----------------+ |- - |UE1|                +------------+ |
      | +-----------+  | |    +---+             +--|    Edge    | |
      | |Edge server|  | |    +---+       +- - -|PE|            | |
      | +-----------+  | |- - |UE2|       |     +--|   Site 1   |-+
      | +-----------+  | |    +---+                +------------+
      | |Edge server|  | |     ...        |            |
      | +-----------+  | +--+         Potential      +---+ +---+
      | +-----------+  | |PE|- - - - - - -+          |UEa| |UEb|
      | |Edge server|  | +--+         Steering       +---+ +---+
      | +-----------+  | |    +---+       |                  |
      | +-----------+  | |- - |UE3|                  +------------+
      | |  ... ...  |  | |    +---+       |        +------------+ |
      | +-----------+  | |     ...              +--|    Edge    | |
      |                | |    +---+       +- - -|PE|            | |
      |Edge data center|-+- - |UEn|             +--|   Site 2   |-+
      +----------------+      +---+                +------------+
      High computing resource              Limited computing resource
      and more UE at metro area            and less UE at remote area

                 Figure 1: Common Deployment of Edge Sites

   Figure 2 shows that during non-working hours, for example at weekend
   or daily night, more UEs move to the remote area that are close to
   their house or for some weekend events.  So there will be more
   service request at remote but with limited computing resource, while
   the rich computing resource might not be used with less UE in the
   metro area.  It is possible for many people to request services at
   the remote area, but with the limited computing resource, moreover,
   as the people move from the metro area to the remote area, the edge
   sites that serve common services will also change, so it may be
   necessary to steer some traffic back to the metro data center.






Yao, et al.              Expires 1 January 2024                 [Page 8]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


        +----------------+                           +------------+
      +----------------+ |                         +------------+ |
      | +-----------+  | |  Steering traffic    +--|    Edge    | |
      | |Edge server|  | |          +-----------|PE|            | |
      | +-----------+  | |    +---+ |           +--|   Site 1   |-+
      | +-----------+  | |- - |UEa| |    +----+----+-+----------+
      | |Edge server|  | |    +---+ |    |           |           |
      | +-----------+  | +--+       |  +---+ +---+ +---+ +---+ +---+
      | +-----------+  | |PE|-------+  |UE1| |UE2| |UE3| |...| |UEn|
      | |Edge server|  | +--+       |  +---+ +---+ +---+ +---+ +---+
      | +-----------+  | |    +---+ |          |           |
      | +-----------+  | |- - |UEb| |          +-----+-----+------+
      | |  ... ...  |  | |    +---+ |              +------------+ |
      | +-----------+  | |          |           +--|    Edge    | |
      |                | |          +-----------|PE|            | |
      |Edge data center|-+  Steering traffic    +--|   Site 2   |-+
      +----------------+                           +------------+
      High computing resource              Limited computing resource
      and less UE at metro area            and more UE at remote area

                Figure 2: Steering Traffic among Edge Sites

   There will also be the common variable of network and computing
   resources, for someone who is not moving but get a poor latency
   sometime.  Because of other UEs moving, a large number of request for
   temporary events such as vocal concert, shopping festival and so on,
   and there will also be the normal change of the network and computing
   resource status.  So for some fixed UEs, it is also expected to steer
   the traffic to appropriate sites dynamicity.

   Those problems indicate that traffic needs to be steered among
   different edge sites, because of the mobility of the UE and the
   common variable of network and computing resources.  Moreover, some
   use cases in the following section require both low latency and high
   computing resource usage or specific computing hardware capabilities
   (such as local GPU); hence joint optimization of network and
   computing resource is needed to guarantee the Quality of
   Experience(QoE).

4.  Use Cases

   This section presents a non-exhaustive set of use cases which would
   benefit from the dynamic selection of service instances and the
   steering of traffic to those service instances.







Yao, et al.              Expires 1 January 2024                 [Page 9]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


4.1.  Computing-Aware AR or VR

   Cloud VR/AR services are used in some exhibitions, scenic spots, and
   celebration ceremonies.  In the future, they might be used in more
   applications, such as industrial internet, medical industry, and meta
   verse.

   Cloud VR/AR introduces the concept of cloud computing to the
   rendering of audiovisual assets in such applications.  Here, the edge
   cloud helps encode/decode and render content.  The end device usually
   only uploads posture or control information to the edge and then VR/
   AR contents are rendered in the edge cloud.  The video and audio
   outputs generated from the edge cloud are encoded, compressed, and
   transmitted back to the end device or further transmitted to central
   data center via high bandwidth networks.

   Edge sites may use CPU or GPU for encode/decode.  GPU usually has
   better performance but CPU is simpler and more straightforward to use
   as well as possibly more widespread in deployment.  Available
   remaining resources determines if a service instance can be started.
   The instance's CPU, GPU and memory utilization has a high impact on
   the processing delay on encoding, decoding and rendering.  At the
   same time, the network path quality to the edge site is a key for
   user experience of quality of audio/ video and input command response
   times.

   A Cloud VR service, such as a mobile gaming service, brings
   challenging requirements to both network and computing so that the
   edge node to serve a service request has to be carefully selected to
   make sure it has sufficient computing resource and good network path.
   For example, for an entry-level cloud VR (panoramic 8K 2D video) with
   110-degree Field of View (FOV) transmission, the typical network
   requirements are bandwidth 40Mbps, 20ms for motion-to-photon latency,
   packet loss rate is 2.4E-5; the typical computing requirements are 8K
   H.265 real-time decoding, 2K H.264 real-time encoding.  We can
   further divide the 20ms latency budget into:

   (i) sensor sampling delay(client), which is considered imperceptible
   by users is less than 1.5ms including an extra 0.5ms for
   digitalization and end device processing.

   (ii) display refresh delay(client), which take 7.9ms based on the
   144Hz display refreshing rate and 1ms extra delay to light up.

   (iii) image/frame rendering delay(server), which could be reduced to
   5.5ms.





Yao, et al.              Expires 1 January 2024                [Page 10]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   (iv) round trip network delay(network), which should be bounded to
   20-1.5-5.5-7.9 = 5.1ms.

   So the the budgets for server(computing) delay and network delay are
   almost equivalent, which make sense to consider both of the delay for
   computing and network.  And it can't meet the total delay
   requirements or find the best choice by either optimize the network
   or computing resource.

   Based on the analysis, here are some further assumption as figure 3
   shows, the client could request any service instance among 3 edge
   sites.  The delay of client could be same, and the differences of
   different edge sites and corresponding network path has different
   delays:

   o Edge site 1: The computing delay=4ms based on a light load, and the
   corresponding network delay=9ms based on a heavy traffic.

   o Edge site 2: The computing delay=10ms based on a heavy load, and
   the corresponding network delay=4ms based on a light traffic.

   o Edge site 3: The edge site 3's computing delay=5ms based on a
   normal load, and the corresponding network delay=5ms based on a
   normal traffic.

   In this case, we can't get a optimal network and computing total
   delay if choose the resource only based on either of computing or
   network status:

   o If choosing the edge site based on the best computing delay it will
   be the edge site 1, the E2E delay=22.4ms.

   o If choosing the edge site based on the best network delay it will
   be the edge site 2, the E2E delay=23.4ms.

   o If choosing the edge site based on both of the status it will be
   the edge site 3, the E2E delay=19.4ms.

   So, the best choice to ensure the E2E delay is edge site 3, which is
   19.4ms and is less than 20ms.  The differences of the E2E delay is
   only 3~4ms among the three, but some of them will meet the
   application demand while some doesn't.









Yao, et al.              Expires 1 January 2024                [Page 11]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   The conclusion is that it requires to dynamically steer traffic to
   the appropriate edge to meet the E2E delay requirements considering
   both network and computing resource status.  Moreover, the computing
   resources have a big difference in different edges, and the "closest
   site" may be good for latency but lacks GPU support and should
   therefore not be chosen.

        Light Load          Heavy Load           Normal load
      +------------+      +------------+       +------------+
      |    Edge    |      |    Edge    |       |    Edge    |
      |   Site 1   |      |   Site 2   |       |   Site 3   |
      +-----+------+      +------+-----+       +------+-----+
   computing|delay(4ms)          |           computing|delay(5ms)
            |           computing|delay(10ms)         |
       +----+-----+        +-----+----+         +-----+----+
       |  Egress  |        |  Egress  |         |  Egress  |
       | Router 1 |        | Router 2 |         | Router 3 |
       +----+-----+        +-----+----+         +-----+----+
     newtork|delay(9ms)   newtork|delay(4ms)   newtork|delay(5ms)
            |                    |                    |
            |           +--------+--------+           |
            +-----------|  Infrastructure |-----------+
                        +--------+--------+
                                 |
                            +----+----+
                            | Ingress |
            +---------------|  Router |--------------+
            |               +----+----+              |
            |                    |                   |
         +--+--+              +--+---+           +---+--+
       +------+|            +------+ |         +------+ |
       |Client|+            |Client|-+         |Client|-+
       +------+             +------+           +------+
                      client delay=1.5+7.9=9.4ms

                     Figure 3: Computing-Aware AR or VR

   Furthermore, specific techniques may be employed to divide the
   overall rendering into base assets that are common across a number of
   clients participating in the service, while the client-specific input
   data is being utilized to render additional assets.  When being
   delivered to the client, those two assets are being combined into the
   overall content being consumed by the client.  The requirements for
   sending the client input data as well as the requests for the base
   assets may be different in terms of which service instances may serve
   the request, where base assets may be served from any nearby service
   instance (since those base assets may be served without requiring
   cross-request state being maintained), while the client-specific



Yao, et al.              Expires 1 January 2024                [Page 12]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   input data is being processed by a stateful service instance that
   changes, if at all, only slowly over time due to the stickiness of
   the service that is being created by the client-specific data.  Other
   splits of rendering and input tasks can be found in[TR22.874] for
   further reading.

   When it comes to the service instances themselves, those may be
   instantiated on-demand, e.g., driven by network or client demand
   metrics, while resources may also be released, e.g., after an idle
   timeout, to free up resources for other services.  Depending on the
   utilized node technologies, the lifetime of such "function as a
   service" may range from many minutes down to millisecond scale.
   Therefore computing resources across participating edges exhibit a
   distributed (in terms of locations) as well as dynamic (in terms of
   resource availability) nature.  In order to achieve a satisfying
   service quality to end users, a service request will need to be sent
   to and served by an edge with sufficient computing resource and a
   good network path.

4.2.  Computing-Aware Intelligent Transportation

   For the convenience of transportation, more video capture devices are
   required to be deployed as urban infrastructure, and the better video
   quality is also required to facilitate the content analysis.
   Therefore, the transmission capacity of the network will need to be
   further increased, and the collected video data need to be further
   processed, such as for pedestrian face recognition, vehicle moving
   track recognition, and prediction.  This, in turn, also impacts the
   requirements for the video processing capacity of computing nodes.

   In auxiliary driving scenarios, to help overcome the non-line-of-
   sight problem due to blind spot or obstacles, the edge node can
   collect comprehensive road and traffic information around the vehicle
   location and perform data processing, and then vehicles with high
   security risk can be warned accordingly, improving driving safety in
   complicated road conditions, like at intersections.  This scenario is
   also called "Electronic Horizon", as explained in[HORITA].  For
   instance, video image information captured by, e.g., an in-car,
   camera is transmitted to the nearest edge node for processing.  The
   notion of sending the request to the "nearest" edge node is important
   for being able to collate the video information of "nearby" cars,
   using, for instance, relative location information.  Furthermore,
   data privacy may lead to the requirement to process the data as close
   to the source as possible to limit data spread across too many
   network components in the network.






Yao, et al.              Expires 1 January 2024                [Page 13]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   Nevertheless, load at specific "closest" nodes may greatly vary,
   leading to the possibility for the closest edge node becoming
   overloaded, leading to a higher response time and therefore a delay
   in responding to the auxiliary driving request with the possibility
   of traffic delays or even traffic accidents occurring as a result.
   Hence, in such cases, delay-insensitive services such as in-vehicle
   entertainment should be dispatched to other light loaded nodes
   instead of local edge nodes, so that the delay-sensitive service is
   preferentially processed locally to ensure the service availability
   and user experience.

   In video recognition scenarios, when the number of waiting people and
   vehicles increases, more computing resources are needed to process
   the video content.  For rush hour traffic congestion and weekend
   personnel flow from the edge of a city to the city center, efficient
   network and computing capacity scheduling is also required.  Those
   would cause the overload of the nearest edge sites if there is no
   extra method used, and some of the service request flow might be
   steered to others edge site except the nearest one.

4.3.  Computing-Aware Digital Twin

   A number of industry associations, such as the Industrial Digital
   Twin Association or the Digital Twin Consortium
   (https://www.digitaltwinconsortium.org/), have been founded to
   promote the concept of the Digital Twin (DT) for a number of use case
   areas, such as smart cities, transportation, industrial control,
   among others.  The core concept of the DT is the "administrative
   shell" [Industry4.0], which serves as a digital representation of the
   information and technical functionality pertaining to the "assets"
   (such as an industrial machinery, a transportation vehicle, an object
   in a smart city or others) that is intended to be managed,
   controlled, and actuated.


















Yao, et al.              Expires 1 January 2024                [Page 14]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   As an example for industrial control, the programmable logic
   controller (PLC) may be virtualized and the functionality aggregated
   across a number of physical assets into a single administrative shell
   for the purpose of managing those assets.  PLCs may be virtualized in
   order to move the PLC capabilities from the physical assets to the
   edge cloud.  Several PLC instances may exist to enable load balancing
   and fail-over capabilities, while also enabling physical mobility of
   the asset and the connection to a suitable "nearby" PLC instance.
   With this, traffic dynamicity may be similar to that observed in the
   connected car scenario in the previous sub-section.  Crucial here is
   high availability and bounded latency since a failure of the
   (overall) PLC functionality may lead to a production line stop, while
   boundary violations of the latency may lead to loosing
   synchronization with other processes and, ultimately, to production
   faults, tool failures or similar.

   Particular attention in Digital Twin scenarios is given to the
   problem of data storage.  Here, decentralization, not only driven by
   the scenario (such as outlined in the connected car scenario for
   cases of localized reasoning over data originating from driving
   vehicles) but also through proposed platform solutions, such as those
   in [GAIA-X], plays an important role.  With decentralization,
   endpoint relations between client and (storage) service instances may
   frequently change as a result.

4.4.  Computing-Aware SD-WAN

   SD-WAN provides organizations or enterprises with centralized control
   over multiple sites which are network endpoints including branch
   offices, headquarters, data centers, clouds, and more.  A enterprise
   may deploy their services and applications in different locations to
   achieve optimal performance.  The traffic sent by a host will take
   the shortest WAN path to the closest server.  However, the closet
   server may not be the best choice with lowest cost of network and
   computing resources for the host.  If the path computation element
   can consider the computing dimension information in path computation,
   the best path with lowest cost can be provided.

   The computing related information can be the number of vCPUs of the
   VM running the application/services, CPU utilization rate, usage of
   memory, etc.

   The SD-WAN can be aware of the computing resource of applications
   deployed in the multiple sites and can perform the routing policy
   according to the information is defined as the computing-aware SD-
   WAN.





Yao, et al.              Expires 1 January 2024                [Page 15]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   Many enterprises are performing the cloud migration to migrate the
   applications from data centers to the clouds, including public,
   private, and hybrid clouds.  The clouds resources can be from the
   same provider or multiple cloud providers which have some benefits
   including disaster recovery, load balancing, avoiding vendor lock-in.

   In such cloudification deployments SD-WAN provides enterprises with
   centralized control over Customer-Premises Equipments(CPEs) in branch
   offices and the cloudified CPEs(vCPEs) in the clouds.The CPEs connect
   the clients in branch offices and the application servers in clouds.
   The same application server in different clouds is called an
   application instance.  Different application instances have different
   computing resource.

   SD-WAN is aware of the computing resource of applications deployed in
   the clouds by vCPEs, and selects the application instance for the
   client to visit according to computing power and the network state of
   WAN.

   Figure 4 below illustrates Computing-aware SD-WAN for Enterprise
   Cloudification.

                                                       +---------------+
      +-------+                      +----------+      |    Cloud1     |
      |Client1|            /---------|   WAN1   |------|  vCPE1  APP1  |
      +-------+           /          +----------+      +---------------+
        +-------+        +-------+
        |Client2| ------ |  CPE  |
        +-------+        +-------+                     +---------------+
      +-------+           \          +----------+      |    Cloud2     |
      |Client3|            \---------|   WAN2   |------|  vCPE2  APP1  |
      +-------+                      +----------+      +---------------+

       Figure 4: Illustration of Computing-aware SD-WAN for Enterprise
                            Cloudification

   The current computing load status of the application APP1 in cloud1
   and cloud2 is as follows: each application uses 6 vCPUs.  The load of
   application in cloud1 is 50%. The load of application in cloud2 is
   20%. The computing resource of APP1 are collected by vCPE1 and vCPE2
   respectively.  Client1 and Client2 are visiting APP1 in cloud1.  WAN1
   and WAN2 have the same network states.  Considering lightly loaded
   application SD-WAN selects APP1 in cloud2 for the client3 in branch
   office.  The traffic of client3 follows the path: Client3 -> CPE ->
   WAN1 -> Cloud2 vCPE1 -> Cloud2 APP1






Yao, et al.              Expires 1 January 2024                [Page 16]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


5.  Requirements

   In the following, we outline the requirements for the CATS system to
   overcome the observed problems in the realization of the use cases
   above.

5.1.  Support dynamic and effective selection among multiple serivce
      instances

   The basic requirement of CATS is to support the dynamic access to
   different service instances residing in multiple computing sites and
   then being aware of their status, which is also the fundamental model
   to enable the traffic steering and to further optimize the network
   and computing services.  A unique service identifier is used by all
   the service instances for a specific service no matter which edge
   site an instance may attach to.  The mapping of this service
   identifier to a network locator makes sure the data packet CATS
   potentially reach any of the service instances deployed in various
   edge sites.

   Moreover, according to CATS use cases, some applications require E2E
   low latency, which warrants a quick mapping of the service identifier
   to the network locator.  This leads to naturally the in-band methods,
   involving the consideration of using metrics that are oriented
   towards compute capabilities and resources, and their corelation with
   services.  Therefore, a desirable system

   R1: MUST provide a discovery and resolving methodology for the
   mapping of a service identifier to a specific address.

   R2: MUST provide an mapping methods for further quickly selecting the
   service instance.

5.2.  Support Agreement on Metric Representation

   Computing metrics can have many different semantics, particularly for
   being service- specific.  Even the notion of a "computing load"
   metric could be represented in many different ways.  Such
   representation may entail information on the semantics of the metric
   or it may be purely one or more semantic- free numerals.  Agreement
   of the chosen representation among all service and network elements
   participating in the service instance selection decision is
   important.  Therefore, a desirable system

   R3: MUST agree on using metrics that are oriented towards compute
   capabilities and resources and their representation among service
   elements in the participating edges.




Yao, et al.              Expires 1 January 2024                [Page 17]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   R4: MUST include network metrics.

5.3.  Support Moderate Metric Distributing

   Network path costs in the current routing system usually do not
   change very frequently.  Network traffic engineering metrics (such as
   available bandwidth) may change more frequently as traffic demands
   fluctuate, but distribution of these changes is normally damped so
   that only significant changes cause routing protocol messages.

   However, metrics that are oriented towards compute capabilities and
   resources in general can be highly dynamic, e.g., changing rapidly
   with the number of sessions, the CPU/GPU utilization and the memory
   consumption, etc.  It has to be determined at what interval or based
   on what events such information needs to be distributed.  Overly
   frequent distribution with more accurate synchronization may result
   in unnecessary overhead in terms of signaling.

   Moreover, depending on the service related decision logic, one or
   more metrics need to be conveyed in a CATS domain.  The problem to be
   addressed here may be the frequency of such conveyance, thanks to the
   comprehensive load that a signaling process may add to the overall
   network traffic.  While existing routing protocols may serve as a
   baseline for signaling metrics, other means to convey the metrics can
   equally be considered and even be realized.  Specifically, a
   desirable system

   R5 MUST provide mechanisms to distribute the metrics

   R6 MUST realize means for rate control for distributing of metrics

5.4.  Support Flexible Use of Metrics

   Considering computing resources assigned to a service instance on a
   server, which might be related to some critical metrics like the
   processing delay, is crucial in addition to the network delay in some
   cases.  Therefore, the CATS components might use both the network and
   computing metrics for service instance selection.  For this reason:

   R7: a computing semantic model SHOULD be defined for the mapping
   selection.

   We recognize that different network nodes, e.g., routers, switches,
   etc., may have diversified capabilities even in the same routing
   domain, let alone in different administrative domains.  So, metrics
   that are oriented towards compute capabilities and resources that
   have been adopted by some nodes may not be supported by others,
   either due to technical reasons, administrative reasons, or something



Yao, et al.              Expires 1 January 2024                [Page 18]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   else.  There exist scenarios in which a node supporting service-
   specific metrics might prefer some type of metrics to
   others[TR22.874].  Of course, specific metrics might not be utilized
   at all in other scenarios.  Hence:

   R8: there MUST exist flexibility in term of metrics definition and
   utilization for the selection of service instance.

   Therefore, a desirable system

   R9: MUST set up metric information that can be understood by CATS
   components.

   R10: MUST use network and computing metrics in a flexible way that
   includes a default action for the interoperation of network nodes
   which may or may not support the specific metrics.

5.5.  Support Session and Service Continuity

   In the CATS system, a service may be provided by one or more service
   instances that would be deployed at different locations in the
   network.  Each instance provides equivalent service functionality to
   their respective clients.  The decision logic of the instance
   selection are subject to the normal packet level communication and
   packets are forwarded based on the operating status of both network
   and computing resources.  This resource status will likely change
   over time, leading to individual packets potentially being sent to
   different network locations, possibly segmenting individual service
   transactions and breaking service-level semantics.  Moreover, when a
   client moves, the access point might change and successively lead to
   the same result of the change of service instance.  If execution
   changes from one (e.g., virtualized) service instance to another,
   state/context needs transfer to another.  Such required transfer of
   state/context makes it desirable to have session persistence (or
   instance affinity) as the default, removing the need for explicit
   context transfer, while also supporting an explicit state/context
   transfer (e.g., when metrics change significantly).  So in those
   situations:

   R11: session as well as service continuity MUST be maintained.

   The nature of this continuity is highly dependent on the nature of
   the specific service, which could be seen as a 'instance affinity' to
   represent the relationship.  The minimal affinity of a single request
   represents a stateless service, where each service request may be
   responded to without any state being held at the service instance for
   fulfilling the request.




Yao, et al.              Expires 1 January 2024                [Page 19]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   Providing any necessary information/state in-band as part of the
   service request, e.g., in the form of a multi-form body in an HTTP
   request or through the URL provided as part of the request, is one
   way to achieve such stateless nature.

   Alternatively, the affinity to a particular service instance may span
   more than one request, as in the AR/VR use case, where previous
   client input is needed to render subsequent frames.

   However, a client, e.g., a mobile UE, may have many applications
   running.  If all, or majority, of the applications request the CATS-
   based services, then the runtime states that need to be created and
   accordingly maintained would require high granularity.  In the
   extreme scenario, this granular requirement could reach the level of
   per-UE per-APP per-(sub)flow with regard to a service instance.
   Evidently, these fine-granular runtime states can potentially place a
   heavy burden on network devices if they have to dynamically create
   and maintain them.  On the other hand, it is not appropriate either
   to place the state-keeping task on clients themselves.

   Besides, there might be the case that UE moves to a new (access)
   network or the service instance is migrated to another cloud, which
   cause the unreachable or inconvenient of the original service
   instance.  So the UE and service instance mobility also need to be
   considered.

   Therefore, a desirable system

   R12: MUST maintain "instance affinity" which MAY span one or more
   service requests, i.e., all the packets from the same application-
   level flow MUST go to the same service instance unless the original
   service instance is unreachable

   R13: MUST avoid keeping fine runtime-state granularity in network
   nodes for providing session and service continuity.

   R14: MUST provide mechanisms to minimize client side states in order
   to achieve the instance affinity.

   R15: SHOULD support the UE and service instance mobility.











Yao, et al.              Expires 1 January 2024                [Page 20]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


5.6.  Preserve Communication Confidentiality

   Exposing the information of computing resources to the network may
   lead to the leakage of computing domain and application privacy.  In
   order to prevent it, it need to consider the methods to process the
   sensitive information related to computing domain.  For instance,
   using general anonymous methods, including hiding the key information
   representing the identification of devices, or using an index to
   represent the service level of computing resources, or using
   customized information exposure strategies according to specific
   application requirements or network scheduling requirements.  At the
   same time, when anonymity is achieved, it is also necessary to
   consider whether the computing information exposed in the network can
   help make full use of traffic steering.  Therefore, a CATS system

   R16: MUST preserve the confidentiality of the communication relation
   between user and service provider by minimizing the exposure of user-
   relevant information according to user needs.

6.  Security Considerations

   CATS decision making process is deeply related to computing and
   network status as well as some service information.  Some security
   issues need to be considered when designing CATS system.

   Service data sometimes needs to be moved among different edge sites
   to maintain service consistency and availability.  Therefore:

   R17: service data MUST be protected from interception.

   The act of making compute requests may reveal the nature of user's
   activities, so that:

   R18: the nature of user's activities SHOULD be hidden as much as
   possible.

   The behavior of the network can be adversely affected by modifying or
   interfering with advertisements of computing resource availability.
   Such attacks could deprive users' of the services they desires, and
   might be used to divert traffic to interception points.  Therefore,

   R19: secure advertisements are REQUIRED to prevent rogue nodes from
   participating in the network.

7.  IANA Considerations

   This document makes no requests for IANA action.




Yao, et al.              Expires 1 January 2024                [Page 21]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


8.  Contributors

   The following people have substantially contributed to this document:

           Peter Willis
           pjw7904@rjt.edu

           Philip Eardley
           philip.eardley@googlemail.com

           Tianji Jiang
           China Mobile
           tianjijiang@chinamobile.com

           Markus Amend
           Deutsche Telekom
           Markus.Amend@telekom.de

           Guangping Huang
           ZTE
           huang.guangping@zte.com.cn

9.  Acknowledgements

   The author would like to thank Adrian Farrel, Peng Liu, Luigi
   IANNONE, Christian Jacquenet and Yuexia Fu for their valuable
   suggestions to this document.

10.  References

10.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC7285]  Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S.,
              Previdi, S., Roome, W., Shalunov, S., and R. Woundy,
              "Application-Layer Traffic Optimization (ALTO) Protocol",
              RFC 7285, DOI 10.17487/RFC7285, September 2014,
              <https://www.rfc-editor.org/info/rfc7285>.

   [RFC7665]  Halpern, J., Ed. and C. Pignataro, Ed., "Service Function
              Chaining (SFC) Architecture", RFC 7665,
              DOI 10.17487/RFC7665, October 2015,
              <https://www.rfc-editor.org/info/rfc7665>.




Yao, et al.              Expires 1 January 2024                [Page 22]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

10.2.  Informative References

   [I-D.contreras-alto-service-edge]
              Contreras, L. M., Randriamasy, S., Ros-Giralt, J., Perez,
              D. A. L., and C. E. Rothenberg, "Use of ALTO for
              Determining Service Edge", Work in Progress, Internet-
              Draft, draft-contreras-alto-service-edge-07, 13 March
              2023, <https://datatracker.ietf.org/doc/html/draft-
              contreras-alto-service-edge-07>.

   [TR22.874] 3GPP, "Study on traffic characteristics and performance
              requirements for AI/ML model transfer in 5GS (Release
              18)", 2021.

   [HORITA]   Horita, Y., "Extended electronic horizon for automated
              driving", Proceedings of 14th International Conference on
              ITS Telecommunications (ITST)", 2015.

   [Industry4.0]
              Industry4.0, "Details of the Asset Administration Shell,
              Part 1 & Part 2", 2020.

   [GAIA-X]   Gaia-X, ""GAIA-X: A Federated Data Infrastructure for
              Europe"", 2021.

Authors' Addresses

   Kehan Yao
   China Mobile
   Email: yaokehan@chinamobile.com


   Dirk Trossen
   Huawei Technologies
   Email: dirk.trossen@huawei.com


   Mohamed Boucadair
   Orange
   Email: mohamed.boucadair@orange.com


   Luis M. Contreras
   Telefonica



Yao, et al.              Expires 1 January 2024                [Page 23]

Internet-Draft   Computing-Aware Traffic Steering (CATS)       June 2023


   Email: luismiguel.contrerasmurillo@telefonica.com


   Hang Shi
   Huawei Technologies
   Email: shihang9@huawei.com


   Yizhou Li
   Huawei Technologies
   Email: liyizhou@huawei.com


   Shuai Zhang
   China Unicom
   Email: zhangs366@chinaunicom.cn



































Yao, et al.              Expires 1 January 2024                [Page 24]