Internet DRAFT - draft-dunbar-ippm-5g-edge-compute-ip-layer-metrics
draft-dunbar-ippm-5g-edge-compute-ip-layer-metrics
Network Working Group L. Dunbar
Internet Draft H. Song
J. Kaippallimalil
Intended status: Standard Futurewei
Expires: May 16, 2021
February 3, 2022
IP Layer Metrics for 5G Edge Computing Service
draft-dunbar-ippm-5g-edge-compute-ip-layer-metrics-02
Abstract
This draft describes the IP Layer metrics and methods to
measure the Edge Computing Servers running status and
environment for IP networks to select the optimal Edge
Computing server location in 5G Edge Computing (EC)
environment. Those measurements are for IP network to
dynamically optimize the forwarding of 5G edge computing
service without any knowledge above IP layer.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. This document may not be
modified, and derivative works of it may not be created,
except to publish it as an RFC and to translate it into
languages other than English.
Internet-Drafts are working documents of the Internet
Engineering Task Force (IETF), its areas, and its working
groups. Note that other groups may also distribute working
documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of
six months and may be updated, replaced, or obsoleted by
other documents at any time. It is inappropriate to use
Internet-Drafts as reference material or to cite them other
than as "work in progress."
xxx, et al. Expires May 16, 2021 [Page 1]
IP Layer Metrics for 5G Edge Computing Services
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed
at http://www.ietf.org/shadow.html
This Internet-Draft will expire on April 7, 2021.
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as
the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date
of publication of this document. Please review these
documents carefully, as they describe your rights and
restrictions with respect to this document. Code Components
extracted from this document must include Simplified BSD
License text as described in Section 4.e of the Trust Legal
Provisions and are provided without warranty as described in
the Simplified BSD License.
Table of Contents
1. Introduction..............................................3
1.1. 5G Edge Computing Background.........................3
1.2. Problem 1: Selecting 5G Edge Application Location....4
1.3. Problem 2: UE mobility creates unbalanced anycast
distribution..............................................5
2. Conventions used in this document.........................7
3. IP-Layer Metrics Definitions for 5G EC services...........9
3.1. IP-Layer Application ID..............................9
3.2. IP-Layer metric for App Server Load Measurement......9
3.3. Capacity Index in the overall cost..................12
3.4. Site Preference Index in the overall cost...........12
3.5. RTT to an ANYCAST Address in 5G EC..................13
4. Algorithm in Selecting the optimal Target Location.......14
5. Scope of IP Layer Metrics Advertisement..................14
6. Manageability Considerations.............................15
7. Security Considerations..................................16
8. IANA Considerations......................................16
Dunbar, et al. Expires May 16, 2021 [Page 2]
IP Layer Metrics for 5G Edge Computing Services
9. References...............................................16
9.1. Normative References................................16
9.2. Informative References..............................16
10. Acknowledgments.........................................17
1. Introduction
1.1. 5G Edge Computing Background
In 5G Edge Computing environment [3GPP-EdgeComputing], one
Application can have multiple Application Servers hosted in
different Edge Computing data centers that are close in
proximity. Those Edge Computing (mini) data centers are
usually very close to, or co-located with, 5G base stations,
with the goal to minimize latency and optimize the user
experience.
When a UE (User Equipment) initiates application packets
using the destination address from a DNS reply or from its
own cache, the packets from the UE are carried in a PDU
session through 5G Core [5GC] to the 5G UPF-PSA (User Plan
Function - PDU Session Anchor). The UPF-PSA decapsulate the
5G GTP outer header and forwards the packets from the UEs to
the Ingress router of the Edge Computing (EC) Local Data
Network (LDN). The LDN for 5G EC, which is the IP Networks
from 5GC perspective, is responsible for forwarding the
packets to the intended destinations.
Routers in the local IP network should be able to select the
"best" or "closest" application server location out of many.
However, simply using distance alone as a metric may not be
sufficient as there may be many locations in close proximity.
Moreover, one of the main aims of locating application
servers so close to the user is to provide lower latency.
When a user moves and attaches from another access router
(UPF), the local IP network should be able to continue
routing to the established application server. As a user
keeps moving further away, a closer application server maybe
able to serve the user better. Network measurements,
including latency of various paths are provided to the
application domain to assist in reselection. These problems
are outlined in sections 1.2, and 1.3.
Dunbar, et al. Expires May 16, 2021 [Page 3]
IP Layer Metrics for 5G Edge Computing Services
1.2. Problem 1: Selecting 5G Edge Application Location
Having multiple locations closer to UEs to host one
Application server can greatly improve the user experience.
But selecting an optimal location for the application traffic
from a UE may not be that simple.
Using DNS to reply with the address of the Application Server
location closest to the requesting UE can encounter issues
like:
- UE can cache results indefinitely, when the UE moves to a
5G cell site very far away, the cached address may still
be used, which can incur large network delay.
- The Application Server at a specific location whose
address replied by the DNS might be heavily loaded
causing slow or no response, when there are available low
utilized Application Server, for the same application, at
different locations very close in proximity.
- No inherent leverage of proximity information present in
the network (routing) layer, resulting in loss of
performance
- Local DNS resolver become the unit of traffic management
Increasingly, Anycast is used extensively by various
application providers and CDNs because ANYCAST makes it
possible to dynamically load balance across locations that
host the Application server based on network conditions.
Application server location selection using Anycast address
leverages the proximity information present in the network
(routing) layer and eliminates the single point of failure
and bottleneck at the DNS resolvers and application layer
load balancers. Another benefit of using ANYCAST address is
removing the dependency on UEs that use their cached
destination IP addresses for extended period.
But selection of an ANYCAST location purely based on the
network condition can encounter issue of the location
selected by network routing information being overutilized
while there are available underutilized locations close by.
Dunbar, et al. Expires May 16, 2021 [Page 4]
IP Layer Metrics for 5G Edge Computing Services
1.3. Problem 2: UE mobility creates unbalanced anycast
distribution
Another problem of using ANYCAST address for multiple
locations of an Application Server in 5G environment is that
UEs' frequent moving from one 5G site to another. The
frequent move of UEs can make it difficult to plan where
Application server should be hosted. When a large number of
UEs using a particular Application congregate together
unpredictably, the ANYCAST location selected based on routing
distance can be heavily utilized, while the same Application
Server at other locations close-by are underutilized.
Dunbar, et al. Expires May 16, 2021 [Page 5]
IP Layer Metrics for 5G Edge Computing Services
+--+
|UE|---\+---------+ +------------------+
+--+ | 5G | +-----------+ | S1: aa08::4450 |
+--+ | Site A +----+ +----+ |
|UE|----| | Ra | | R1 | S2: aa08::4460 |
+--+ | +----+ +----+ |
+---+ | | | | | S3: aa08::4470 |
|UE1|---/+---------+ | | +------------------+
+---+ |IP Network | L-DN1
|(3GPP N6) |
| | | +------------------+
| | | | S1: aa08::4450 |
| | +----+ |
| | | R3 | S2: aa08::4460 |
v | +----+ |
| | | S3: aa08::4470 |
| | +------------------+
| | L-DN3
+--+ | |
|UE|---\+---------+ | | +------------------+
+--+ | 5G | | | | S1: aa08::4450 |
+--+ | Site B +----+ +----+ |
|UE|----| | Rb | | R2 | S2: aa08::4460 |
+--+ | +----+ +----+ |
+--+ | | +-----------+ | S3: aa08::4470 |
|UE|---/+---------+ +------------------+
+--+ L-DN2
Figure 1: multiple ANYCAST instances in different edge DCs
This document describes the measurements at IP Layer that can
reflect the application server running status and environment at
the specific locations. This document also describes the method
of incorporating those measurements with IP routing cost to come
up with a more optimal criteria in selecting ANYCAST locations.
Note: for the ease of description, the Edge Application Server
and Application Server are used interchangeably throughout this
document.
Dunbar, et al. Expires May 16, 2021 [Page 6]
IP Layer Metrics for 5G Edge Computing Services
2. Conventions used in this document
A-ER: Egress Router to an Application Server Instance,
[A-ER] is used to describe the last router that
the application instance is attached. For 5G EC
environment, the A-ER can be the gateway router
to the Edge Computing Data Center.
ANYCAST Instance: refer to the Application Server Gateway at
a specific location which is reachable by the
ANYCAST address.
Application Server: An application server is a physical or
virtual server that host the software system for
the application.
Application Server Location: Represent a cluster of servers
at one location serving the same Application. One
application may have a Layer 7 Load balancer,
whose address(es) are reachable from external IP
network, in front of a set of application
servers. From IP network perspective, this whole
group of servers are considered as the
Application server at the location.
EC: Edge Computing
Edge Application Server: used interchangeably with
Application Server throughout this document.
Edge Computing Hosting Environment: An environment, such as
psychical or virtual machines, providing support
required for Edge Application Server's execution.
NOTE: The above terminologies are the same as
those used in 3GPP TR 23.758
Dunbar, et al. Expires May 16, 2021 [Page 7]
IP Layer Metrics for 5G Edge Computing Services
Edge DC: Edge Data Center, which provides the Edge Hosting
Environment. It might be co-located with 5G Base
Station and not only host 5G core functions, but
also host frequently used Edge server instances.
gNB next generation Node B
Instance: the term "Instance" if used in the context of
Application Server, is referring to one location
of an application server; if used in the ANYCAST
context, is referring to one location of the
Application server with the same ANYCAST address.
L-DN: Local Data Network
PSA: PDU Session Anchor (UPF)
RTT: Round Trip Time
RTT-ANYCAST: A list of Round trip times to a group of
routers that have the ANYCAST instances directly
attached.
SSC: Session and Service Continuity
UE: User Equipment
UPF: User Plane Function
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
RECOMMENDED", "MAY", and "OPTIONAL" in this document are to
be interpreted as described in BCP 14 [RFC2119] [RFC8174]
when, and only when, they appear in all capitals, as shown
here.
Dunbar, et al. Expires May 16, 2021 [Page 8]
IP Layer Metrics for 5G Edge Computing Services
3. IP-Layer Metrics Definitions for 5G EC services
3.1. IP-Layer Application ID
From network perspective, an application server has an
Identifier, or IP Layer Application Server ID, which is
usually an ANYCAST address that can represent multiple
locations that host the application server.
3.2. IP-Layer metric for App Server Load Measurement
There are many network techniques and protocols to optimize
forwarding and ensuring QoS for applications, such as
DSCP/DiffServ, Traffic Engineered (TE) solutions, Segment
Routing, etc. But the reality is that most application
servers don't expose their internal logics to network
operators. Their communications are generally encrypted. Most
of them do not even respond to PING or ICMP messages
initiated by routers or network gears.
The proposed IP Layer Metrics and algorithms enable the IP
networks to dynamically optimize the forwarding of 5G edge
computing service without any knowledge above IP layer.
In a way, the proposed IP Layer Metrics and algorithm enable
the IP networks to be more aware of Application behavior
without dependency on getting information from Applications
themselves. Without knowledge of application internal
logics, network layer or IP Layer can monitor the traffic
patterns to/from the Application Server at each location to
gauge the running status of the application server at the
location.
First, the network needs to discover which router(s) has the
application server attached. Those routers are called
Application Server Egress Router, or A-ER for short. A-ER is
usually the Gateway Router to an Edge Computing Data Center.
To discover if a (Gateway) router is the A-ER for a specific
Edge Application Server, the (Gateway) router can
periodically send reverse ARP (IPv4) or Neighbor Discovery
scan with the address of the Application Server to discover
if the Application Servers are hosted in its edge computing
data center. If yes, the router or routers are identified as
the A-ER for the Application Server. For one Application
Server, there can be many A-ERs at different EC Data Centers.
Dunbar, et al. Expires May 16, 2021 [Page 9]
IP Layer Metrics for 5G Edge Computing Services
For an Application Server at a specific location, which is
identified by the address of the application server at the IP
layer, the A-ER can measure the amount of traffic destined
towards the address & the amount of the traffic from the
specific address, such as:
- Total number of packets to the attached App Server
(ToPackets);
- Total number of packets from the attached App Server
(FromPackets);
- Total number of Bytes to the attached App Server
(ToBytes);
- Total number of bytes from the attached App Server
(FromBytes);
The actual load measurement to the App Server attached to an
A-ER can be based on one of the metrics above or including
all four metrics with different weights applied to each, such
as:
LoadIndex =
w1*ToPackets+w2*FromPackes+w3*ToBytes+w4*FromBytes
Where 0<= wi <=1 and w1+ w2+ w3+ w4 = 1.
The weights of each metric contributing to the load index of
the App Server attached to an A-ER can be configured or
learned by self-adjusting based on user feedbacks.
The raw measurement is useful when the A-ER routers cannot be
configured with a consistent algorithm to compute the
aggregated load index and the raw measurements are needed by
a central analytic system.
The A-ER can advertise either the aggregated Load Index or
the raw measurements periodically, by BGP UPDATE messages or
OSPF/ISIS Link statement Advertisements, to a group of
routers that have traffic destined towards the ANYCAST
addresses of those application servers.
Dunbar, et al. Expires May 16, 2021 [Page 10]
IP Layer Metrics for 5G Edge Computing Services
Even though it would be better to have applications or their
controllers directly reporting their own workload running
status to network, it is not easy to have third party
applications to provide information to network operators in
addition to that applications servers can be running anywhere
and.
The IP-Layer Load Measurements provides an intelligent
estimate of the application server running status at a
specific location without requiring cooperation from third
party Applications or Application controllers.
Dunbar, et al. Expires May 16, 2021 [Page 11]
IP Layer Metrics for 5G Edge Computing Services
3.3. Capacity Index in the overall cost
Given that different Edge Computing DCs may have different
capacity, the following metrics need to be included to gauge
an application Server's running status at a specific
location:
- Capacity Index:
Capacity Index is used to differentiate the running
environment of the application server. Some data centers
can have hundreds, or thousands, of servers behind an
Application Server's App Layer Load Balancer that is
reachable from external world. Other data centers can have
very small number of servers for the application server.
"Capacity Index", which is a numeric number, is used to
represent the capacity of the application server in a
specific location.
"Capacity Index" can be a configured value indicating the
capacity of a specific Application Server at a specific
location, e.g. an Edge Computing DC. The Capacity Index is
Application Server specific, meaning at one location, one
Application Server can have the Capacity Index to be 10 and
another server can be 2.
If the Application Server capacity configuration is not
available, a network analytics tool can use the historic
measurements as the basis to estimate the site capacity. If
an Application Server at a specific site has high volume
for extended period historically, the site capacity can be
considered as higher than the other site with historic low
volume. This is under the assumption that application
controllers monitor utilization of the application servers
at different locations. If an application server has
prolonged over-utilized servers at some locations, the
application controller will trigger manual intervention to
increase the computing powers at those locations. However,
the manual intervention cycles can be in weeks/months. That
is why the IP layer metrics and algorithms that can change
flows direction in minutes become very essential.
3.4. Site Preference Index in the overall cost
Dunbar, et al. Expires May 16, 2021 [Page 12]
IP Layer Metrics for 5G Edge Computing Services
As described in [IPv6-StickyService] and [ISPF-EXT-EC], an
EC sticky service needs to connect a UE to the application
server that has been serving the UE before the UE moves to
a new 5G Site, unless there is failure to that location.
To achieve the goal of sticking a flow from one specific UE
to a specific site, a "site Preference Index" is created.
The value of the Site Preference Index can be manipulated
for packets of some flows to be steered towards an
application server location farther away in routing
distance. The "Site Preference Index" enables some sites to
be more preferred for handling the UE traffic to a
application server than others.
3.5. RTT to an ANYCAST Address in 5G EC
ANYCAST used in 5G Edge computing environment is slightly
different from the typical ANYCAST address being deployed.
Typical ANYCAST address is used to represent instances in
vast different geographical locations, such as different
continents. ANCAST address for "app.net" for Asia lead
packets to a server instance of "app.net" hosted in Asia.
Therefore, the RTT for "app.net" in Asia, is a single value
that represent the round time trip to the server in Asia
that host the "app.net".
5G Edge Computing environment can have one Application
server hosted in multiple Edge Computing DCs close in
proximity. Routers, i.e. the ingress router to 5G LDN
(Local Data Network), can forward packets for the ANYCAST
address of "app.net" to different egress routers that have
"app.net" servers attached.
If "app.net" is hosted in four different 5G Edge Computing
Data Centers. All those DCs have the same ANYCAST address
for the "app.net". The RTT to "app.net" ANYCAST address
need to be a group of values (instead of one RTT value to a
unicast address). The RTT group value should include the A-
ER router's specific unicast address (e.g. the loopback
address) to which the Application Server is attached.
RTT to "app.net" ANYCAST Address is represented as:
List of {Egress Router address, RTT value}
This list is called "RTT-ANYCAST".
Dunbar, et al. Expires May 16, 2021 [Page 13]
IP Layer Metrics for 5G Edge Computing Services
In order to better optimize the ANYCAST traffic, each
router adjacent to 5G PSA needs to periodically measure RTT
to a list of A-ER routers that advertise the ANYCAST
address. The RTT to egress router at Site-i is considered
as the RTT to the ANYCAST instance at the Site-i.
4. Algorithm in Selecting the optimal Target Location
The goal of the algorithm is to equalize the traffic among
multiple locations of the same ANYCAST address.
The main benefit of using ANYCAST is to leverage the IP-
layer information to equalize the traffic among multiple
locations of the same Application Server, usually
identified by one or a group of ANYCAST addresses.
For 5G Edge Computing environment, the ingress router to
each LDN needs to be notified of the Load Index and
Capacity Index of the Application Servers at different EC
site to make the intelligent decision on where to forward
the traffic from UEs for an application.
The Algorithm needs to take the following attributes into
consideration:
- Load Measurement Index [Section 3.2],
- capacity index [Section 3.3],
- Preference Index [Section 3.4], and
- network delay [Section 3.5].
Here is an algorithm for a router, e.g. the router directly
attached to the 5G PSA, to compare the cost to reach the
Application Server between the Site-i or Site-j:
Load-i * CP-j Pref-j * Delay-i
Cost-i=min(w *(----------------) + (1-w) *(------------------))
Load-j * CP-i Pref-i * Delay-j
Load-i: Load Index at Site-i, it is the weighted
combination of the total packets and bytes sent to and
received from the Application Server at Site-i during a
fixed time period.
Dunbar, et al. Expires May 16, 2021 [Page 14]
IP Layer Metrics for 5G Edge Computing Services
CP-i (Capacity-i) (higher value means higher capacity):
capacity index at the site i.
Delay-i: Network latency measurement (RTT) to the A-ER
that has the Application Server attached at the site-i.
Pref-i (Preference Index: higher value means higher
preference): Network Preference index for the site-I.
w: Weight for load and site information, which is a value
between 0 and 1. If smaller than 0.5, Network latency and
the site Preference have more influence; otherwise, Server
load and its capacity have more influence.
5. Scope of IP Layer Metrics Advertisement
Each Application Server might be used by a small group of
UEs. Therefore, it is not necessary for A-ER router to
advertise the IP layer metrics to all other routers in the 5G
LDN. Likewise, each EC Data Center may only host a small
number of application servers.
"Application Bound Group Routers" is used to refer a group of
routers that are interested in a group of specific ANYCAST
addresses. The IP Layer Metrics for an Application Server
should be advertised among the routers in the "Application
bound Group Routers".
BGP RT Constrained Distribution [RFC4684] can be used to form
the "Application Bound Group Routers".
Since there are much more Application Servers than the number
of routers in 5G LDN, a more practical way to form the
"Application Bound Group of Routers" is for each ingress
router to query a network controller upon receiving the first
packet to a specific ANYCAST address to be included in the
"Application Bound Group Routers". There should be a timer
associated with Ingress router, as the UE that uses the
Application Server might move away. Upon timer expires, the
Ingress Router is removed from the "Application Bound Group
of Routers".
6. Manageability Considerations
To be added.
Dunbar, et al. Expires May 16, 2021 [Page 15]
IP Layer Metrics for 5G Edge Computing Services
7. Security Considerations
To be added.
8. IANA Considerations
To be added.
9. References
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private
networks (VPNs)", Feb 2006.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in
RFC 2119 Key Words", BCP 14, RFC 8174, DOI
10.17487/RFC8174, May 2017, <https://www.rfc-
editor.org/info/rfc8174>.
[RFC8200] s. Deering R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", July 2017
9.2. Informative References
[3GPP-EdgeComputing] 3GPP TR 23.748, "3rd Generation
Partnership Project; Technical Specification Group
Services and System Aspects; Study on enhancement
of support for Edge Computing in 5G Core network
(5GC)", Release 17 work in progress, Aug 2020.
[RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation
Subsequent Address Family Identifier (SAFI) and the
BGP Tunnel Encapsulation Attribute", April 2009.
Dunbar, et al. Expires May 16, 2021 [Page 16]
IP Layer Metrics for 5G Edge Computing Services
[BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension
for SDWAN Overlay Networks", draft-dunbar-idr-bgp-
sdwan-overlay-ext-03, work-in-progress, Nov 2018.
[SDWAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K.
Majumdar, "BGP UPDATE for SDWAN Edge Discovery",
draft-dunbar-idr-sdwan-edge-discovery-00, work-in-
progress, July 2020.
[Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation
Attribute", draft-ietf-idr-tunnel-encaps-10, Aug
2018.
10. Acknowledgments
Acknowledgements to Donald Eastlake for their review and
contributions.
This document was prepared using 2-Word-v2.0.template.dot.
Dunbar, et al. Expires May 16, 2021 [Page 17]
IP Layer Metrics for 5G Edge Computing Services
Authors' Addresses
Linda Dunbar
Futurewei
Email: ldunbar@futurewei.com
HaoYu Song
Futurewei
Email: haoyu.song@futurewei.com
John Kaippallimalil
Futurewei
Email: john.kaippallimalil@futurewei.com
Dunbar, et al. Expires May 16, 2021 [Page 18]
