Internet DRAFT - draft-zzhang-dmm-5gdn-end-marker
draft-zzhang-dmm-5gdn-end-marker
dmm Z. Zhang
Internet-Draft Juniper Networks
Intended status: Standards Track M. Liebsch
Expires: 9 August 2024 NEC Lab
T. Jiang
China Mobile
6 February 2024
End Marker in 5G Data Network
draft-zzhang-dmm-5gdn-end-marker-01
Abstract
In a mobile network, when a mobile device (referred to as User
Equipment or UE) moves from one Access Node (source AN) to another
(target AN), the anchoring node that connects the UE to the Data
Network sends an End Marker to the source AN after it starts sending
UE traffic towards the target AN. The target AN buffers the data
until it receives the End Marker forwarded by the source AN. This is
to preserve the order of packets during the handover between ANs.
The anchoring node is referred to as User Plane Function (UPF) in 5G
and it is a Network Function of the mobile network. The UPFs are
traditionally centrally deployed but are more and more distributed
closer to the edge. With distributed UPFs, handover becomes
necessary among UPFs, and the End Marker mechanism may need to be
extended to Data Network devices that are not mobile network
functions. This document discusses the problem and proposes a
solution based on ICMP messages if packet ordering needs to be
preserved during handover between UPFs.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 9 August 2024.
Zhang, et al. Expires 9 August 2024 [Page 1]
Internet-Draft 5G-DN End Marker February 2024
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. End Marker with Centralized UPF . . . . . . . . . . . . . 2
1.2. End Marker with Distributed UPF/ANUP . . . . . . . . . . 4
1.2.1. Traffic via a Hub Router . . . . . . . . . . . . . . 5
1.2.2. Traffic from Anywhere . . . . . . . . . . . . . . . . 5
1.2.3. Remote UE Routes on Source UPF . . . . . . . . . . . 6
1.2.4. Is End Marker Mechanism Really Needed? . . . . . . . 6
2. Specification . . . . . . . . . . . . . . . . . . . . . . . . 7
3. Security Considerations . . . . . . . . . . . . . . . . . . . 7
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7
6. Informative References . . . . . . . . . . . . . . . . . . . 7
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8
1. Introduction
In a mobile network like 5G [_3GPP-23.501], when a mobile device
(referred to as User Equipment or UE) moves from one Access Node
(source AN) to another (target AN), the anchoring node User Plane
Function (UPF) that connects the UE to the Data Network sends an End
Marker to the source AN after it starts sending UE traffic towards
the target AN. The target AN buffers the data until it receives the
End Marker forwarded by the source AN. This is to preserve the order
of packets during the handover between ANs.
1.1. End Marker with Centralized UPF
The End Marker mechanism in case of centralized UPF is described with
the following diagram.
Zhang, et al. Expires 9 August 2024 [Page 2]
Internet-Draft 5G-DN End Marker February 2024
Internet
/
/N6
UPF
/ \
/ \
| N3 |
| |
| |
AN1 AN2
| -> |
\ /
UE1
UE1 is initially connected to AN1. There is an N3 tunnel between AN1
and UPF for the UE. The UE1-AN1 connection and the AN1-UPF N3 tunnel
together provide a Packet Data Unit Session (PDU Session) that
connects UE1 to the Data Network (DN, which is Internet in this case)
via the UPF. The UPF does IP routing between the DN and UE (among
other things).
When UE1 moves from AN1 (source) to AN2 (target), a new N3 tunnel
between AN2 and UPF is established as instructed by 5G control plane
functions. For Down Link (DL) traffic from the Internet, UPF starts
sending towards AN2 via the new N3 tunnel. There could be inflight
packets towards AN1, which will be redirected by AN1 to AN2. The
redirected packets were sent by the UPF before it switches over to
the new tunnel, yet they could arrive on AN2 after the packets that
the UPF sends over the new tunnel to AN2.
To preserve the packet ordering during this handover, the following
sequence of actions are taken:
* The UPF sends an End Marker to the source AN1 over the old tunnel
after it sends the last packet over the old tunnel and switches to
the new tunnel for future packets.
* The source AN1 redirects inflight packets to the target AN2, who
forwards the redirected traffic to UE1, while buffering data
arriving on the new tunnel.
* The source AN1 redirects the received End Marker (which comes
after the inflight packets) to the target AN2, who will now start
sending buffered and new arrival data to UE1.
Zhang, et al. Expires 9 August 2024 [Page 3]
Internet-Draft 5G-DN End Marker February 2024
1.2. End Marker with Distributed UPF/ANUP
With distributed UPF, the diagram becomes the following:
Hub PE --- Internet
/ \
| DN |
| |
PE1 PE2
| |
UPF1 UPF2
| |
| N3 |
AN1 AN2
| -> |
\ /
UE1
UPF1 and UPF2 are close to the ANs. The DN is an IP VPN (referred to
as DNVPN) provided by PE1/PE2 and a hub PE (where the previous
central UPF was) connected to the Internet. The UPFs are VPN CEs.
[I-D.zzhang-dmm-mup-evolution] proposes an Integrated AN/UPF (ANUP)
concept for co-located AN and UPFs. Whether AN and UPFs are separate
or integrated, the problem discussion and proposed solution in this
document apply.
As described in [I-D.zzhang-dmm-mup-evolution], when UE1 moves from
AN1/UPF1 to AN2/UPF2, its IP address may be retained if that is
desired. A UPF has UE specific routes/state for its local UEs so
that DL traffic can be forwarded accordingly. The hub PE has UE
specific routes for all UEs that have persistent addresses so that DL
traffic may be sent to the appropriate UPFs. While some PEs may also
have UE specific routes for UEs attached to other UPFs, some may just
have a default route to send all Up Link (UL) traffic (traffic not
destined to its local UEs) towards the hub PE.
This is achieved by standard hub-and-spoke VPN mechanisms: some
routes are advertised with Route Targets that allow them to be
installed everywhere, while some other routes are advertised with
different Route Targets that allow them to be installed only on hub
routers.
Zhang, et al. Expires 9 August 2024 [Page 4]
Internet-Draft 5G-DN End Marker February 2024
1.2.1. Traffic via a Hub Router
For DL traffic from the Internet via the hub router, re-ordering may
happen when UE1 moves from AN1/UPF1 to AN2/UPF2 - after the hub PE
updates its UE1 route to send traffic to UPF2, there may be inflight
UE1 packets towards PE/UPF1 but then rerouted to PE2/UPF2 (either
directly or via the hub PE) in the DN, and they may arrive after the
traffic that the hub sends to UPF2 directly.
Notice that during this process, UPF1 will not trigger an End Marker
to AN1 because there is no N3 tunnel changing on UPF1 as UE1 is now
moving to gNB2/UPF2 and UPF1 does not use N3 tunneling to gNB2/UPF2.
Instead, the hub PE needs to trigger UPF1 to send an End Marker to
AN1 after the hub router changes its UE1 route from PE1 to PE2.
Because PEs are just DN devices not 5G function, an ICMP message can
be sent instead by the hub PE towards UPF1's address in the DN. The
ICMP message is of a new type/code and carries the UE route so that
UPF1 can match the PDU session and send End Marker to AN1. In
certain cases, e.g., with [I-D.mhkk-dmm-srv6mup-architecture], the
hub PE may know the PDU session information associated with a UE
route and it can include the session information in the ICMP message.
Note that, if PE1 also maintains UE routes received from other PEs,
it prefers its own received from UPF1. This is typical IPVPN
behavior that local routes are preferred. This ensures that inflight
traffic (before the ICMP trigger for End Marker) from the hub PE is
sent to UPF1 instead of routed to PE2/UPF2. Otherwise, reordering
may happen (inflight traffic before the End Markter trigger arrives
after the traffic that is sent to UPF2 after the trigger).
After UPF1 sends End Marker, it withdraws the UE1 route from PE1.
1.2.2. Traffic from Anywhere
In the previous section, only traffic from the hub PE is considered.
However, with distributed UPFs, DL traffic may be from anywhere
directly without going through a hub. For example, there could be a
UPF3/PE3 connecting to a local DN site and traffic from that site via
PE3 or UE traffic via UPF3 to UE1 also needs to preserve order when
UE1 moves to UPF2/AN2. If PE3 has UE1 specific route (previously
through PE1/UPF1 then through PE2/UPF2 directly instead of just a
default route via the hub PE), then UPF3 also needs to trigger UPF1
to send End Marker.
In this case, the source UPF1 needs to know how many triggers it
needs to receive before sending the End Marker. The number is
dependent on how many PEs are configured with the Route Target that
Zhang, et al. Expires 9 August 2024 [Page 5]
Internet-Draft 5G-DN End Marker February 2024
the UE1 route is advertised with (those PEs will install the UE1
router and send End Marker trigger when their UE1 route changes from
PE1/UPF1 to PE2/UPF2). Since the trigger could be lost, and some PEs
may be down or slow in sending the trigger, UPF1 needs to send End
Marker to its AN when one of the following conditions are met:
* The expected number of triggers are received, or,
* A preset timer expires
Alternatively, the source UPF1 can send the End Marker upon receipt
of a master trigger from a controller.
1.2.3. Remote UE Routes on Source UPF
In both Section 1.2.1 and Section 1.2.2 scenarios, the source PE1 may
have the specific UE1 route that is advertised from the target PE2.
After the hub PE or PE3 switches to the target PE2/UPF2, inflight
traffic sent before the switch, arriving on PE1 and then following
the UE1 route on PE1, may arrive on UPF2 later than the traffic sent
to UPF2 directly after the switch, if on PE1 the UE1 route through
PE2/UPF2 is preferred over the UE1 route through UPF1.
With VPN the CE routes through a PE-CE interface are typically
preferred over those learned from another PE - so this should work
just fine
1.2.4. Is End Marker Mechanism Really Needed?
One may wonder if the it is really necessary to extend the End Marker
mechanism to the DN, given the following considerations:
* Complexity especially when traffic is not just from a hub PE.
* With higher speed of 5G/6G, buffering data may become more and
more difficult.
* IP network does not guarantee packet ordering. While routers do
try to send traffic of the same flow out of the same ECMP branch,
topology change will likely cause reordering and routers do not
buffer data for ordering purposes.
Zhang, et al. Expires 9 August 2024 [Page 6]
Internet-Draft 5G-DN End Marker February 2024
On the other hand, mobile network is unique in that mobility handover
happens more frequently than topology changes in traditional IP
networks, and in the discussions related to session continuity in the
context of distributed UPF and ANUP, preserving packet ordering is
often brought up (at least rhetorically). Therefore, this document
does discuss the problem and propose a solution in case it is indeed
needed.
2. Specification
Normative specification will be provided in future revisions, if it
is agreed that End Marker mechanism is indeed needed in the DN.
3. Security Considerations
To be provided.
4. IANA Considerations
To be added.
5. Acknowledgements
The authors thank Constantine Polychronopoulos, Arda Akman and Jari
Mutikainen for their review, comments and suggestions to make this
document and solution more complete.
6. Informative References
[I-D.mhkk-dmm-srv6mup-architecture]
Matsushima, S., Horiba, K., Khan, A., Kawakami, Y.,
Murakami, T., Patel, K., Kohno, M., Kamata, T., Camarillo,
P., Horn, J., Voyer, D., Zadok, S., Meilik, I., Agrawal,
A., and K. Perumal, "Mobile User Plane Architecture using
Segment Routing for Distributed Mobility Management", Work
in Progress, Internet-Draft, draft-mhkk-dmm-srv6mup-
architecture-06, 23 October 2023,
<https://datatracker.ietf.org/doc/html/draft-mhkk-dmm-
srv6mup-architecture-06>.
[I-D.zzhang-dmm-mup-evolution]
Zhang, Z. J., Patel, K., Contreras, L. M., Islam, K.,
Mutikainen, J., Jiang, T., Jalil, L., Sejati, O. P., and
S. Zadok, "Mobile User Plane Evolution", Work in Progress,
Internet-Draft, draft-zzhang-dmm-mup-evolution-06, 10 July
2023, <https://datatracker.ietf.org/doc/html/draft-zzhang-
dmm-mup-evolution-06>.
Zhang, et al. Expires 9 August 2024 [Page 7]
Internet-Draft 5G-DN End Marker February 2024
[_3GPP-23.501]
"System architecture for the 5G System (5GS), V18.2.1",
June 2023.
Authors' Addresses
Zhaohui Zhang
Juniper Networks
Email: zzhang@juniper.net
Marco Liebsch
NEC Lab
Email: Marco.Liebsch@neclab.eu
Tianji Jiang
China Mobile
Email: tianjijiang@chinamobile.com
Zhang, et al. Expires 9 August 2024 [Page 8]
