| INTAREA Working Group | S. Bryant |
| Internet-Draft | U. Chunduri |
| Intended status: Informational | T. Eckert |
| Expires: September 10, 2020 | A. Clemm |
| Futurewei Technologies Inc. | |
| March 09, 2020 |
Forwarding Layer Problem Statement
draft-bryant-arch-fwd-layer-ps-00
This document considers the new use cases for IP together with the network capabilities and services that will be needed to address those use cases. It then looks at the underlying packet requirements and considers the changing deployment models and the issues with existing packet designs that need to be addressed. It concludes by looking at some parameters of a solution.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 10, 2020.
Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
There is an emerging set of new requirements that exceed the network and transport services of the current Internet, which only delivers “best effort” service. While many controlled or private networks include further services, such as other DiffServ QoS in addition to best effort and traffic engineering with bandwidth guarantees, the solutions used today only support walled gardens and are thus not available to application service providers and consumers across the Internet.
The purpose of this document is to look at current, evolving and future use cases and to examine the shortcomings that the existing network and transport layer protocols a well as their associated control plane need to overcome to meet these needs.
The IETF is the body responsible for the long term evolution of the IP protocol suit, but is missing a work track to discuss the long-term Internet network architecture evolution. In particular it lacks a programme for the long term evolution of IP itself.
Approximately 30 years ago, the IETF started a process to revolutionize the IPv4 [RFC0791] Internet Protocol. In this process, researchers, industry, and service providers got together, and brought up a number of new proposals, and worked toward a successor to IPv4, which became IPv6 [RFC2460] and later [RFC8200].
30 years later, there is heavy resistance to anything more than minor incremental evolutions to IPv6. There are a number of reasons for this ranging from opinions that all future IP needs can be met through minor incremental evolutions to fears that major proposals for innovation at the IP would be an unwelcome disrupter to the current business of the vendors or the service providers.
The authors take no position on the scale of the problem or the difficulties of deploying any solutions at scale in the Internet. What we seek to do is to establish the scope and nature of the problem. A decision on which aspects of the problem are economically tractable is out of scope of this text, but technologies to support monetization are not.
As a problem statement, this documents goal is to not propose or promote specific solutions to the problems raised. Instead it uses references to not Internet adopted, but proposed or existing solutions only as example evidence that the described problem can actually be solved.
Because the document does not propose specific solutions, it also does not attempt to structure the problem description in a way that would identify sub-set of problems to be resolved by specific solution components.
The purpose of this text is thus to stimulate discussion on the emerging needs of the forwarding layer and to start the process of determining how they are best satisfied within the IETF protocol suite.
The term “forwarding layer” is used in this document because none of the standard terms encompass the parts of the network stack that need attention to address the needs of the applications that are foreseen.
It is possible that development work will need to reach down to layer 2.5 in order to ensure that packets are handled correctly down to the physical layer. The MAC layer is quite sophisticated and includes its own switching function so we need to be sure that the good work done in the network layer is not undone lower down the stack. Equally it is possible that development work will need to reach into the transport layer to address new approaches to congestion and to ensure that the network layer understands the requirements placed on it by the application. An open mind is needed on the boundaries of the layers as they exist today when analyzing the consequential network changes needed to support the evolving application space.
In the network layer itself, this document is only concerned with the forwarding component, not path selection or the other components of routing.
Thus we use the term forwarding layer to describe the scope of the stack that this document addresses.
This section summarizes the use case areas that have been observed by the authors and are considered relevant to the following analysis of gaps.
This section is structured into sub-sections discussing either group of use cases directly or the work of specific groups that are identifying use cases and that may also work on identifying issues and or proposed architectures or solutions for them.
Subsections are ordered from what might be considered to be the most near-term use cases to the potentially most far reaching ones.
Audio/Video streaming for production, entertainment, surveillance and other purposes, and interactive audio/video are the most ubiquitous applications on the Internet and private IP networks after web-services. They have grown primarily through an evolution of the applications to work with the constraints of todays Internet and adopting pre-existing infrastructure such as content caches: best-effort streaming with adaptive video, no service guaranteees for most services, and co-location of caches with large user communities. In environments where more than best-effort services for these applications are required and deployment of current technologies to support them is feasible, it is done. Examples include DiffServ or even on or offpath bandwidth reservations in controlled networks.
Networked AR/VR is a very near term set of use cases, where solution models are very much attempting to use and expand existing solution approaches for video network streaming but where the limits of above current best practices are also amplified by the larger bandwidth requirements and stricter latency and jitter requirements of AR/VR.
To ensure a good user experience, for live Virtual Reality (VR), a much higher resolution than 8K video is required. In addition to the high bandwidth requirements of VR, there needs to be a supporting transmission network to provide a communications path with bounded low latency as well. This stringent VR latency requirement is a challenge to existing networks.
In cellular networks, even though the the air interface link latency needed is significantly reduced e.g. with New Radio (5GNR), the end-to-end (E2E) requirements for live VR is harder to meet. This is because of the fixed L2/IP/MPLS networks in front/mid/backhaul components, and because of the best effort nature of the packet delivery systems in these networks.
The 5G and beyond 5G (B5G) services are not meant to be limited to the 5G-NR (new-radio). In fact for those services relating to uRLLC, mMTC and eMBB packet networks have evolve along with the radio technologies. While 5G-NR protocol stack has evolved to provide per-frame reliability and latency guarantees, the IP/MPLS transport network by and large remains best-effort. It is no longer possible to solve network problems simply by increasing the capacity. The expectations 5G devices have of 5G networks, can not be met without improving IP/MPLS based backhaul networks. For example, the 5G based systems involve machine to machine communications, generally using command-based smaller payloads. In this case the overheads of packet headers and overlays become apparent when computing latency budget of such packets.
The IETF has produced a large body of work on the deterministic needs of network applications [RFC8578]. These range from refinements and expansions of above summarized Audio/Video and AR/VR use cases over gaming into many more “industrial” use cases. Industrial use cases generally involve industrial controllers for high-precision machinery and equipment, such as robotic arms, centrifuges, or manufacturing equipment for the assembly of electronic components.
These use cases have in common that they require delivery of packets with very precise and “deterministic” performance characteristics, as the controlled equipment and the control loops involved have very exact timing requirements and are not tolerant of any latency variations, as otherwise control loop issues and other undesired effects may occur.
Specifically, the use cases involve curtailing maximum latency that could be incurred. However, deterministic networking, by itself, does not appear to be sufficient to meet all of the emerging needs.
The ITU-T has been running a Focus Group (FG) Network-2030 [FGNETWORK2030] to analyze the needs of networks in the period post 2030. This work started in July 2018 and has been an open process with contribution by a cross-section of the networking industry. Because this is non-IETF work, this section summarizes the currently finalied key findings of the ITU-T Focus Group Network-2030 to make it easier for the reader to better undersstand the work. Note that this work is still ongoing and additional findings may be published.
The Focus Group Network 2030 considered a number of use cases that it was postulated would need to be addressed in the 2030 time-frame and the technology gaps that need to be bridged in order to address these needs. It then considered a number of new network services that would be needed to support these services.
An ongoing piece of work on the architecture of the network post 2030 has not yet been completed at the time of writing and is only partially discussed in this document.
The reader is referred to [WP], [NET2030SubG2], [UC] for information beyond that provided in this summary.
ITU-T FG NET2030 Sub-group Sub-G1 (Sub-G1) considered a number of use cases that it considered to be representative of the network needs post 2030. These needs are legitimate needs in their own right, but as is always the case act as poster-children for new applications that will inevitable conceived in the light of the new network capabilities that we postulate to be necessary.
Further information on these use cases is provided in Section 10, and in the ITU documents [UC] and [WP].
Note to the reader: Unlike ITU-T Study Groups which are restricted to members, ITU-T Focus Groups are open to anyone without payment. At the time of writing, ITU-T Focus Group Network-2030 material that is not available for anonymous download, is accessible for free by joining the Study Group.
In order to support the use cases presented in Section 2, a number of new network services will be needed. Likewise, a number of additional more general network capabilities will becoming increasingly important. Neither services nor capabilities are sufficiently supported to the degree that will be required by Internet technology in use today.
This section describes these services and capabilities at a high level. It builds on a corresponding analysis that was conducted at ITU-T FG-NET2030; readers are referred Section 11 for further detail and, of course, to output produced by that group [NET2030SubG2] for a more complete explanation of their considerations.
[NET2030SubG2] identifies a number of network services that will be needed to support many of the new use cases. These network services are divided into two categories:
The following are a set of Foundational Services :
These are described in more detain in Section 11.1.
[NET2030SubG2] identifies also a number of network capabilities that will become increasingly important going forward, in addition to the support for any particular services.
A number of those need to be taken into consideration from the very beginning when thinking about how future data-planes need to evolve. These capabilities are described in more detail in Section 11.2.
The current Internet is essentially of best-effort system, but future applications require high-precision KPIs on throughput, latency and packet loss for industrial manufacturing, control, automation, and machine-to-machine communications.
With upcoming Cellular technologies (5G/B5G) there is a need for Service Providers to expand the type of customers for metropolitan size networks to address their better than best-effort traffic needs.
DetNet has been proposed to support this, however:
The ratio of useful data in the payload to overhead has a direct financial impact on communication links; these links are of finite capacity and hence have a finite cost-per-unit-data that can be calculated. The capacity used to transport information as compared to the overhead which is unavailable for use by a customer, but required to transmit is often expresses as a good-put efficiency and can be related to cost to transmit payload data.
On the other hand, in various non-constrained environments where various network layer functionalities are desired, there are different set of requirements. For example:
Developments in IPv6 [I-D.filsfils-spring-srv6-network-programming] formalize a trend that has been happening for a long time: the morphing of network layer addresses into forwarding identifiers (FI). However, constraining FIs to a fixed size ill serves the development of the forwarding layer. There are clear cases as illustrated above where it would be useful to have shorter network layer addresses. Equally we can see that there will be future cases where 128 bits may be insufficient to specify a forwarding operation. The requirement is thus to formally introduce the concept of forwarding identifiers in place of network layer addresses, and use a forwarding identifier construct that supports multiple semantics and multiple, possibly fully variable, lengths.
Network operators crave facilities that let them better understand and fine tune detailed network behavior, which are hard to retrofit with current IP/IPv6.
The rise of machine learning has led to the expectation of being able to better optimize networks This in turn leads to the increase of network telemetry as a source of data to base these systems on. In-Situ OAM (IOAM) [I-D.ietf-ippm-ioam-data] represents one of the latest developments in that space, allowing the data plane to piggy-back telemetry data onto individual packets in order to diagnose and fine-tune service levels such as latency or jitter. However, there are several issues with this approach:
While useful, IOAM exposes the limits of what add-on solutions can provide. Solutions that provide visibility at the level of flows or that provide automatic verification of Service Level Objectives are missing entirely.
It needs to be also recognized that it will not be sufficient for solutions to support new services and capabilities one at a time and independently from one another. Instead, solutions need to be holistic and be able to support new services and capabilities in integrated fashion and simultaneously.
For example, better-than-best-effort, operational visibility, and efficient packet design should go together, without leading to additional integration problems ore requiring users to to make a choice.
This is in contrast to the current piecemeal approach, in which solutions for any one particular problem may well be developed but emerge one at a time, resulting in fragmented solutions that are may be hard to integrate.
Service requirements from networks and security implications vastly differ in various deployment models as categorized below.
This is the traditional service provider deployment where various network services (VPN, security, Bandwidth..) are offered to the endpoints of the communication and other providers. Such capabilities are purchased through contract with the service provider and are typically expensive.
These networks predominantly use MPLS technology though native IP (IPv4/IPv6) with GRE and IPv6 with routing extension headers with SRv6 are being deployed recently.
..................................
+---+ . +---+ Single +---+ . +---+
|CE1|---|PE1|---.. Provider ..---|PE2|---|CE2|
+---+ . +---+ Network +---+ . +---+
..................................
Figure 1: An Edge-2-Edge Network
In this case there is a single provider network in which E2E offerings and host session are initiated and terminated with in the single provider network.
There are two sub cases:
a) Where the host is physically (wired/wireless) connected to the Provider Edge (PE)
.............................................
+---+ .+----+ +----+. +---+
| H +--+ PE |--- 1 Provider Network ---| PE +--+ H |
+---+ .+----+ (Single/Multiple domains) +----+. +---+
.............................................
Figure 2: An Edge-2-Edge Network Direct
In this case the provider controls the whole path and can certify the correct operation of the service according to contract.
b) Where the host is connected via its own network to the PE
+---+ +---+
| H | | H |
+-+-+ +-+-+
| |
| ............................................. |
+-+--+ .+----+ +----+. +-+-+
| CE +--+ PE |--- 1 Provider Network ---| PE +--+ H |
+----+ .+----+ (Single/Multiple domains) +----+. +---+
..............................................
Figure 3: An Edge-2-Edge Network Indirect
In this case the provider controls only the path to the CE and can certify the correct operation of the service according to contract from that point but the user is responsible for providing the required service characteristics into their own network.
There are two cases to consider:
The first case is very difficult to support since it is unlikely that the whole path is know to support extended capabilities in the forwarding plane. It is not infeasible, and it would be possible to set up such paths in principle given suitable enhancements to the routing system. However such a scenario must be considered infeasible for the foreseeable future.
The second case is more tractable provided there is co-operation between the providers.
The industry move is towards content and application service providers embedding themselves within the edge network. This is currently done to save bandwidth and improve response time. As the need for high precision low latency networking develops the need for edge computing rises since the closer the client and the server the less the scope for network induced performance degradation.
+---+
| H |
+-+-+
|
| .....................................
+-+--+ .+----+ +---+ .
| CE +--+ PE |--------+ S | .
+----+ .+----+ +---+ Provider 1 .
.....................................
Figure 4: An Edge-2-Provider
In this network the server S (owned by the content and applications provider) has a contractual relationship with provider 1 and is thus able to negotiate the network characteristics needed to meet its service requirement. This model in which the server brokers the user to network interface (UNI) requirements removes many of the objections to the classical UNI model in which the client requests the service requirements. In this model the host authenticates itself with the server, having formed a previous business relationship (for example by purchasing a holographic conferencing service). The server has a relationship with Provider1, and thus is a trusted party able to request that the service be set up between itself and and its client, paying as necessary. As this is a requested paid service traversing a limited distance over a defined network, a bespoke packet protocol can, if necessary, be used with in a contained and constrained way.
How the server communicates with any other part of the application domain is out of scope for this document and possibly out of scope for Provider 1.
This takes us to consider the embedded global service described in {#EGS}.
+---+
| H1|
+-+-+
|
| ......................................
+-+--+ . +----+ +---+ .
| CE +---+ PE |--------+ S1| .
+----+ . +----+ +-+-+ Provider 1 .
..................|...................
|
|
..................|...................
+----+ . +----+ +-+-+ .
| CE +---+ PE |--------+ S2| .
+----+ . +----+ +---+ Provider 2 .
| ......................................
|
+-+-+
| H2|
+-+-+
Figure 5: Edge-2-Edge via Provider
In this network model, the server S1 (owned by the content and applications provider) has a contractual relationship with provider 1 and is thus able to negotiate the network characteristics needed to meet its service requirement. It is servicing the needs of host H1.
Similarly that same provider has a contractual relationship with provider 2 where it is servicing the needs of host H2.
By a method outside the scope of this document and outside the scope of the global Internet the contents and applications provider has a private path between S1 and S2.
This scenario shown in Figure 5 is important because it removes the overwhelming issues associated with providing enhanced service across the global Internet. Furthermore it describes a model where there is commercial incentive, at scale, for the edge providers (Provider 1 and 2 above) to invest in providing and enhanced access service.
Despite IPv4 still having a large user base, and having a number of useful properties the IETF has abandoned future development of IPv4 as a way to force the deployment of IPv6. For example, in terms of traffic steering the segment routing could have usefully been applied to IPv4 to support network operators that wished to retain IPv4 as their preferred internal protocol.
Given the gaps in each of the existing network layer protocols the IETF may wish to look at the design of a protocol that both fills the gaps and unifies its three existing network layer protocols.
Additionally there is a clear need for a more sophisticated approach to indicating the required quality of service that a packet, or flow, needs in an IP network.
IPv6 and specifically [RFC8200] was designed to fit within an Internet architecture centered around the end-to-end model with “Internet Paths” potentially passing through one or more networks without any relationship to the endpoints of a communication such as most so-called transit-AS. As history already from IPv4 had shown, anything more than the most simple per-hop processing options can cause interoperability issues. In result, [RFC8200] has drastically limited such per-hop processing options.
Two core restrictions of RFC8200 are the following:
See
Section 6.1.1.1 for further discussion.While [RFC8200] is a conservative set of requirements to enable proliferation of the target use case of “Internet Paths”, the same set of requirements limit the flexibility of IPv6 unnecessarily when it is used in controlled networks where the constraints and interoperability issues for “Internet Paths” do not equally apply, for example the deployment scenarios shown in Section 5.4 and Section 5.5.
One typical type of controlled networks are service providers (SP) where SRv6 is used as the architecture within the SP network.
For use-cases like this, it would be a lot easier to innovate IPv6 by clone & modify: E.g.: defining (say) IPv7 to be similar to IPv6, but without the constraints that are not useful for the controlled network use-case. A better alternative would be to create different profiles of IPv6 with [RFC8200] being one. However, there is, as yet, no concept of “profiles” in IPv6.
The issue of IP protocol operation in limited domains is discussed in [I-D.carpenter-limited-domains].
Some possible solutions are described in [I-D.herbert-6man-eh-attrib]. This will be considered further in a future version of this text.
Today, the majority of end-to-end connections already do not pass via the traditional “Internet-Path” but instead toward a server in data center co-located with the access service provider Figure 4[DOT]. In this case, there is no transit service provider, but there is a well-established commercial relationship between either end of the communications and the access service provider.
Today, the majority of traffic consists of video-streaming/TV services, but in the future, Edge-Compute will enable ever more applications to operate in such a controlled environment.
The difference between the aforementioned use-case of IPv6 within an service provider, and this use-case is that enhanced services in this would naturally operate end-to-end between a Data Center application server and the subscriber endpoints.
In the case of SRv6, it is not necessary to incur the overhead of an IPv6 in IPv6 encapsulation, the SRH can be inserted by the endpoint and removed by the endpoint on the other side. Nevertheless, the [RFC8200] limitations of not being able to add/remove or freely change the content of the SRH payload or any other EH on a midpoint router still exists. This seriously limits the usage and evolution of of IPv6 to the edge-to-edge model.
Hop-by-hop IPv6 extension headers caused interoperability and performance issues and as a result caused resistance to further leverage and extend them except for SRv6-SRH RPL-SRH [RFC6554]. In the authors opinions, this regression on hop-by-hop extension headers is because of a combination of insufficient specifications and resulting implementation issues. Both could be solved in future work with new hop-by-hop processing specifications.
For example, router alert (RA) was (and still maybe) implemented in routers so that all router alert packets are punted from the fast-path to the slow-path even when the “value” field identifies a protocol that the router can not process. As a result, protocols that rely on RA such as RSVP [RFC2205] or even more so Pragmatic General Multicast (PGM) [RFC3208] where filtered in networks because they caused high control plane load on routers that did not support either protocols but still unnecessarily punted their packets with RA.
There are no normative statements about the need that fast-path forwarding planes “MUST” be able to ignore unsupported/not-enabled EH features at a speed such that such a packet can be forward at the same speed as the same packet without the EH. For example, for RA, there is only a “SHOULD” requirement to do this in [RFC6398], a BCP published a decade after IPv6 router alert [RFC2711]. With such a gap in time between the specification and the BCP, it is impossible to rely on the existing RA and expect safe deployment across the Internet without still running into performance issues.
The same design paradigm could have been used for the Segment Routing Header (SRH) [I-D.ietf-6man-segment-routing-header], but there is no distinction possible for IPv6 instances running in such a controlled network or running as an Internetwork instance to form the Internet. This is particularly unfortunate as we are evolving to a model where, as noted earlier in this document, in most cases the packet will only travel through two well-known networks: the hosts network and the service provider network hosting the server to which the client is interacting.
When IPv6 was designed, the key focus was on solving the problem of growth of the Internet and resulting growth of global Internet address space. Variable length and a hetrogenious address approach were proposed [RFC1347] however, these were rejected partially for political reasons and partially out of a concern over the difficulty of parsing the packet and doing a fast address lookup.
There was seemingly no focus on better supporting the now millions of often network-layer isolated TCP/IP networks in industrial, defense, research, embedded, industrial or other commercial environments.
One key problems with with 128 bit addresses is the overhead on low-speed radio/IoT-wire networks. This is especially the case when using source-routing, where multiple of these addresses have to be included in the header. Current solutions are only able to resolve these issues with CPU expensive IETF standardized header compression techniques [RFC2507], [RFC3095], [RFC5795]. Even though these approaches are feasible in many of todays IoT networks, there is a strong desire to reduce power consumption in such devices. This is particularly the case where they are powered by a single-for-life-battery, or are self-powering through automatic replenished energy sources. As a result of this CPU performance in future IoT network should not be expected to increase but whenever feasible is more likely to decrease.
Another, often overlooked, problem of the 128 bit IPv6 addresses is that global address prefix allocation is a a big up-front burden on many IoT networks, but also isolated networks (industrial, defense, research, industrial). Often, this leads to the use of Unique Local Addresses (ULA) [RFC4193], which have the risk of conflicts when those previously isolated networks need to interconnect with other networks.
While solutions to these problems may look easier enough, it should be noted that in the time when IPv6 was designed, variable length addresses in the fastest forwarding planes were not seen as feasible, and there was also a lack of experience with the impact of interconnecting heterogeneous address spaces other than as ships-in-the-night parallel operation of protocols. A lot of that experience came later through 14++ IPv4/IPv6 transition solutions designed in the past 20 years and respective work on address discovery in IETF frameworks such as SIP/STUN/ICE.
Another issue with the fixed length homogeneous address approach is the constraints this places on the current practice of overloading addresses with other functionality for example [I-D.filsfils-spring-srv6-network-programming].
Some of the fastest growing network segments where new services are being introduced in an End-2-End manner belong to deployment models as described in Section 5.2. The requirements here for service delivery involves stringent E2E latency with no retransmission and no packet loss. Not all scenarios need “lower” latency but bounded to a particular value/range. Example use cases involving an user equipment (UE) consuming service from the provider cloud network or another UE (e.g. Vehicular device, IIoT) in the same network. Here the service endpoints could be connected over wire or wireless (LTE/NR) and the service termination happens in the provider network either close to the access network or provider core network as illustrated in Figure 2, Figure 3. The existing network layer and best-effort model simply cannot guarantee needed service level objectives in these scenarios.
Some specific needs and requirements from cellular fixed transport networks are:
Even without going to future application requirements as described elsewhere in this document, even the majority of existing Internet traffic is lacking competitively usable and standardized service to support quality of service.
The majority of traffic today is Adaptive Bit-rate (ABR) based audio/video streaming. The primary benefit of this approach is that it can adjust itself to much lower bandwidth than the bandwidth to offer the ideal/target experience quality to the user. It therefore enabled Over The Top (OTT) services to offer streaming media. Nevertheless, ABR itself does not provide any actual quality guarantees.
Service providers that use ABR streaming to their subscribers do therefore combine ABR with IP developments, some non-published, which are often out-of-band bandwidth reservation schemes. These allow ABR video streams to have their ideal/target experience bandwidth within the SP’s network and only need to degrade if there was bandwidth contention in the subscribers (home) network.
If a subscriber, or a content provider which is not the access service provider wanted to get the same type of bandwidth guarantees for other content across the access providers network, they could do so with existing IETF standards via RSVP [RFC2205] which is widely implemented, or NSIS [RFC4080], which was to the knowledge of the authors never implemented in widely used router products (because it does not offer sufficient benefits over RSVP). In either case, the per-flow control-plane based signaling architecture including the aforementioned router-alert issues make these protocols a difficult, likely not future-proof solution.
Even more fundamentally, ABR has shown that media streaming can easily support elastic adjustment between a range of bandwidth limits in which the quality is between acceptable and ideal, but there is as of today no standardized mechanisms by which to express relative bandwidth allocations when streams compete against each other that goes beyond the very loosely defined “internet fairness”. For example, more intelligent congestion management could defend bandwidth the more the bandwidth approaches the minimum acceptable bandwidth, or admission control of bandwidth could be elastic. Some work in these direction exists in [RFC8698] with its ability for weighted congestion control or [I-D.ietf-tsvwg-intserv-multiple-tspec] for (limited) elastic admission control management.
Time Critical (TC), Ultra-Reliable, Low Latency (URLLC), Internet-of-Things is another important use case scenario-set that highlights requirements that are difficult to satisfy with existing Internet connectivity paths where a part of that path includes a radio access link. These kind of close-loop control systems borne over heterogeneous communications networks have very precision and bounded latency requirements for the E2E network connecting the sensor and actuator.
Deterministic networking within the IETF is focused on only one dimension of the URLLC problem.
DetNet is also far from attempting to identify currently if/how the services it plans to introduce could be made to operate over the Internet in general, instead, it focusses mostly on the shorter term goal to enable them in controlled networks within a limited domains.
Currently, the requirements for a DetNet forwarding plane have been reasonably mapped out for an MPLS based forwarding layer. Nevertheless, in addressing these needs within an IP network [I-D.ietf-detnet-ip] the solution has of necessity been limited to the capabilities of the IP as it exists today. It has not, for example, been possible to add the packet replication elimination and reordering function (PREOF)which allows multiple concurrent packet delivery attempts in an MPLS network [I-D.ietf-detnet-mpls]. The DETNET body of requirements needs to be revisited in the light of any development to network forwarding capabilities.
High-end hardware with accelerated forwarding plane devices, can support a significant number of forwarding states including destination entries (IP destination/mask, MPLS label, SR SID) as well as 2, 3 or 5 tuple IP/IPv6 “flow” entries. Nevertheless, the control plane that builds and changes these entries often limits their usability because the control plane does not even scale to the number of hardware accelerated forwarding entries possible, or because the supported rate of changes is slow.
The root of this problem is that with the increase of speed and scale of hardware accelerated forwarding hardware, control plane had challenges to keep up in performance. The performance of appropriately priced control plane CPUs (relative to the cost of the forwarding plane) has not grown at the same speed as that of hardware accelerated forwarding plane chips.
One of the directions to overcome these challenges is invisible outside these forwarder devices and it is to optimize the control-plane to forwarding plane interactions, such as programming the building of forwarding state directly on the accelerated forwarding infrastructure (e.g. NPU), but using otherwise existing control plane protocols.
A more fundamental approach is to redesign control plane protocols such that they are lighter weight in their signaling and state machinery, and can therefore be completely implemented in the hardware accelerated forwarding plane. Effectively turning a control plane protocol into an advanced forwarding plane protocol function.
This approach is logically most easily applicable to on-path per-flow signaling mechanisms such as RSVP or RSVP-TE, both of which are quite complex with their signaling messaging and state keeping and therefore directly infeasible to become hardware accelerated forwarding implementations. An example approach to provide similar functionality to RSVP with signaling light-weight enough to allow hardware accelerated implementation are the in-band signaling mechanisms (e.g. for TCP or UDP) described in [DIP1] [I-D.han-tsvwg-ip-transport-qos] [I-D.han-tsvwg-enhanced-diffserv].
Signaling that is feasible to become part of a complete in-forwarding-plane signaling solution is not limited to in-band on-path flow signaling, but would likely also be applied to other signaling options. Of the aforementioned existing signaling protocols, IGPs are likely the ones whose signaling could most easily be processed in an NPU compute elements except that the SPF calculation itself introduces a complexity that would make this very complex. One example of a solution that solves this problem by signaling the actual per-hop adjacencies in IGP and therefore eases NPU implementation can be found in [I-D.chunduri-isis-preferred-path-routing].
In summary: The scope of what should be considered forwarding plane today is defined by decade historic architectures, but should for the future be scoped by the realities of the new, different “layers” of hardware and their capabilities. Hence also the use of the term forwarding plane, because it can span not only across classical bridging (L2), label/tag/SIG switching (L2.5), network/internetwork (L3) and transport (L4) layers, but also across the classical “data plane” and “control plane” components of each such layer.
Some of the deployment models as described in Section 5.2, needs specific signaling mechanism from user/applications. These are needed for E2E service offering for better than best effort Section 6.2 or high-precision networking Section 6.4. These may involve new transport mechanisms at hosts, middle-boxes and routers to meet the E2E service requirements in these limited domain deployments.
Here one of the functional requirements is to signal the service level objectives (SLOs) dynamically for a particular service from the network. This signaling includes the service description, the service negotiation with the network, the service setup or modification, or the need to execute some functions at network device and send the results back to the sender. However, the current IP was not designed for this. For example, the result of SLO negotiation at any hop needs to be updated in the IP packet at the router and returned back to the sender (originating host or gateway device for a Service Provider).
There are some attempts to achieve the above as described in [I-D.han-tsvwg-ip-transport-qos], which describes general in-band signaling for QoS control with IPv6 protocol and [I-D.han-tsvwg-enhanced-diffserv], which proposes a backward compatible class-based queuing and scheduling schema for hybrid service to support guaranteed service from the network (e.g. for latency and bandwidth).
In summary, it is difficult to do better than best effort or High Precision Services described in Section Section 6.4, in closed domains with current IP given the best effort congestion control (TCP/QUIC) and explicit congestion notification (ECN) framework. A comprehensive mechanism needs to be explored as the limitations in silicon technologies or deployment models 30 years ago are not relevant with respect to security, scalability, packet size change, MSS or FCS recalculation, etc.
This section describes a number of solution considerations, but is not prescriptive about any specific approach or technical solution, and is provided to stimulate thought on the subject.
When private networks are set up, they only need to use an address length that allows the construction of networks sufficiently large to meet the expected service requirements. If a future network layer protocol could support address length of e.g.: 16, 24, 32, 48, 64 and 128 bits (or maybe more), it would be easy for such networks to pick a right size. This would allow them to have as efficient packets without compression as possible, and it would also avoid for them to have to think about allocation procedures for “global” addresses.
Whenever networks with a smaller address size would later on have to interconnect to other networks, the shorter length address would have to be interpreted as the suffix of a sufficiently larger address space through which those connecting networks could achieve unique, non-overlapping addresses. At the border between these networks, high speed forwarding planes could easily perform per-packet stateless prefix addition/deletion transformations of addresses in the packet header when the interconnection should be free of further policy. When such an interconnection is desired to employ specific traffic control policies, mapping of addresses in a stateful manner is a convenient way to enforce and support such policies through the forwarding plane.
Classically IP unicast addresses identify an interface. There is the special case of a loop-back address, but this is normally modeled as an internal interface. Addresses are often silently mapped to include other semantics and this is most developed in the IP network programming concept [I-D.filsfils-spring-srv6-network-programming].
MPLS is more general. It defines the concept of a Forwarding Equivalence Class in which a Label which can be visualized as an offset into a specific table with up to 2^20 entries, with the table containing the instruction to be executed. Thus a single identifier is able to specify: forward towards an egress, forward along a specific path, decapsulate and sent to an interface, decapsulate and forward via an IP lookup in a label specific address table etc.
The semantics of the MPLS label and the size of the label are such that it is not possible to include any instruction parameters in the label and very inefficient to include those parameters in one or more further labels. The only example of doing this is the Entropy Label indicator [RFC6790] which uses two Label Stack Entries (LSEs). Any future development along these lines will need at least three LSEs.
Whilst an IPv6 is larger there is still limited space to add parameters within the address. In the current work on this the size is limited to 16 bits, and there is a fundamental limit of 64 bits.
It is clear that move is towards a multiplicity of semantic for the network layer address, and indeed a formal recognition that the address is in reality an instruction with a specific scope.
What we have learned from MPLS and then from SRv6 is that it is often desirable for a node (be that the originating host or a router) to impose on a packet a set of instructions to be executed in sequence by one or more entities in the network. An development of IP or any successor needs to recognize this and provide a simple and efficient way to incorporate a list (or stack) of instructions within the packet header.
There is an established need to do node specific instructions as is indicated by the design of MPLS and Segment Routing (SR). Any development of the forwarding system needs to retain this feature and ideally develop a method that is simultaneously both general and efficient.
References to efficiency include efficiency in packet size and efficiency decoding and and executing the instruction. The efficiency of encoding is not simply a matter of on the wire bandwidth, but is also a matter of the size of the forwarder packet header cache. This cache has to operate at wire speed can be an expensive silicon element.
There is also a need to do path specific operations as are done in RSVP-TE. However RSVP has a significant path set-up and path maintenance cost. Clearly a per path instruction can be specified as a set of N per node instructions where N is the number of hops along the path, for example by using SR, but that is not an efficient encoding where N is large. It is thus a useful optimization to include the ability to include per path instructions, and this is the subject of further study.
Being best effort in nature, assurance for services provided using IP is left to add-on solutions built after the fact. How to perform tasks such as verifying of service levels is left as an exercise for network providers, often approached using statistical approaches that are themselves “best effort” in nature. This will be no longer sufficient for mission-critical services such as tele-driving or tele-operations that demand guarantees, where failure to meet those guarantees may expose providers and users exposed to liability demands and call the feasibility of applications relying on those services into question.
Moving forward, network protocols suitable to deliver high-precision services for mission critical applications need to address assurance as an intrinsic property, not left to afterthoughts.
This document does not request any allocations from IANA.
Security is likely to be more significant with the applications being considered in this work. With interest in tightly controlled access and latency, and contractual terms of business it is going to be necessary to have provable right of access to network resources. However heavyweight security is a contra-requirement to the light-weight process needed for power efficiency, fast forwarding and low latency. Addressing this will require new insights into network security.
Further information on the issue of providing security in latency sensitive environments can be found in [I-D.ietf-detnet-security] which are a sub-set of the considerations applicable to the new use cases considered in this text.
This work projects that we will move towards a holographic society where users remotely interact with the physical world over the network. In industry the digital twin model will enable the control of real objects through digital replicas. Telepresence will move to a new level with multi-site collaborations becoming much closer to physical meetings that can take place without the time and environmental cost of physical travel. 3D medical scans will become full 3D views rather than the body/ organ slices that too many of us are regrettably familiar with. It is easy to imagine that this technology will take message delivery to a completely new level.
Analysis of these concepts results in the conclusion that the following key network requirements are necessary:
Two cases were proposed as examples of this class of application. The first is remote industrial management which involves the real-time monitoring and control of industrial infrastructure operations. The second involves remote robotic surgery. Remote robotic surgery within an operating suite complex is a standard practice today, however there are cases where it would be desirable to extend the range of this facility.
Analysis of these concepts results in the conclusion that the following key network requirements are necessary:
The game-changer in the area of space-terrestrial networking is the active deployment of huge clusters of cheap Low Earth Orbit (LEO) satellite constellations. These LEOs have a number of properties that make them attractive, but arguably the most important is that they combine global coverage with low latency. Studies [Handley] show that for distances over 3000Km latency via a LEO cluster is lower than the latency of terrestrial networks. The up-link to a LEO cluster has to constantly change the point of attachment to the cluster as the satellites that form the cluster rapidly move across the sky relative to both the ground and relative to the satellites in other orbits. In this scenario a number of access and connection models need to be considered.
Analysis of these concepts results in the conclusion that the following key network requirements are necessary:
There is evidence that there is a change in direction from the Internet as a single hetrogenious network back to a true Internet, that is an interconnection of a number of networks each optimized for its local use but capable of inter-working.
For example, satellite and the terrestrial networks adopt different protocol architecture, which causes the difficulty to internetwork between them, yet the common goal is to provide access to the Internet. Secondly, there will be a massive number of IoT-type devices connecting to the networks but the current interconnection schemes are too complex for these services. There are further trends in 5G/B5G back-haul infrastructure, requiring diverse set of resource guarantees in networks to support different industry verticals. The collection of such special purpose networks, existing together and requiring interconnection among themselves are called ManyNets.
Much closer the traditional Internet model is the move to edge computing services in which the client traffic is terminated at a compute node very close to access edge. [DOT] Any resultant application traffic is a private matter between the application on the edge server and the servers it communicates with in the fulfillment of those needs. Furthermore the application on the client may be using a tunnel to the edge compute server. In such a network the protocol used inside the tunnel and the protocol used between the servers executing the service is a private matter.
The ManyNets concept aims to support flexible methods to support the communication among such heterogeneous devices and their networks.
This appendix expands on the ITU-T Sub-G2 new network capabilities and services introcuced in Section 3 It builds upon the analysis that was conducted at ITU-T FG-NET2030; readers are also referred to output produced by that group [NET2030SubG2] for more detail.
[NET2030SubG2] identifies a number of network services that will be needed to support many of the new use cases. These network services are divided into two categories:
The following sections focus on Foundational Services only, as these are the ones that provide the basic building blocks with which the needs of all other services can be addressed, and which are the ones that potentially introduce new foundational requirements on network system nodes.
High-Precision Communications Services refers to services that have precisely defined service level objectives related to end-to-end latency, in many cases coupled with stringent requirements regarding to packet loss and to bandwidth needs. These requirements are in stark contrast to the best effort nature with related to existing network services.
Of course, existing services often go to great lengths in order to optimize service levels and minimize latency, and QoS techniques aim to mitigate adverse effects of e.g. congestion by applying various forms of prioritization and admission control. However, fundamentally all of these techniques still constitute patches that, while alleviating the symptoms of the underlying best-effort nature, do not address the underlying cause and fall short of providing service level guarantees that will not be just of a statistical nature but that will be met by design.
The high-precision communications services that have been identified are described in the following three sub-sections.
In-time services require end-to-end latency within a quantifiable limit. They specific a service level objective that is not to be exceeded, such as a maximum acceptable latency (putting a hard boundary on the worst case). In-time services are required by applications and use cases that have clear bounds on acceptable latency, beyond which the Quality of Experience would deteriorate rapidly, rendering the application unusable. An example concerns use cases that involve providing tactile feedback to users. Creating an illusion of touch requires a control loop with a hard-bounded round-trip time that is determined by human / biological factors, beyond which the sense of touch is lost and with it the ability to e.g. operate a piece of machinery from remote. Because many such use cases are mission-critical (such as tele-driving or remote surgery), in addition any loss or need for retransmission is unacceptable.
This service is similar to the service provided by DetNet [RFC8655] but with more demanding applications which need to be satisfied over IP.
On-time services require end-to-end-latency to be of an exact duration, with the possibility of a small quantifiable variance as can be specified either by an acceptable window around the targeted latency or by a lower bound in addition to an upper bound. Examples of use cases include applications that require synchronization between multiple flows that have the same in-time latency target, or applications requiring fairness between multiple participants regardless of path lengths, such as gaming or market exchanges when required by regulatory authorities. The concept of a lowest acceptable latency imposes new requirements on networks to potentially slow down packets by buffering or other means, which introduces challenges due to high data rates and the cost e.g. of associated memory.
Coordinated services require multiple interdependent flows to be delivered with the same end-to-end latency, regardless of any (potential additional) service level objective. Use cases and applications include applications that require synchronization between multiple flows, such as use cases involving data streams from multiple cameras and telemetry sources. In the special case where an on-time service is required, no additional service is needed (as synchronization occurs by virtue of the fact that each flow adheres to the same SLO), but coordination may also be required in cases where no specific end-to-end latency is required, as long as all flows are serviced with service levels that are identical.
Qualitative communication services (QCS) are able to suppress retransmission of portions of the payload that are deemed less relevant when necessary in order to meet requirements on latency by applications that are tolerant of certain quality degradation. They may involve the application of network coding schemes.
QCS is a new service type that is needed to support AR/VR, holographic-type communications Industrial Internet and services such as autonomous driving. This needs the support of a new network capability that is as yet to be developed.
[NET2030SubG2] identifies also a number of network capabilities that will become increasingly important going forward, in addition to the support for any particular services. These were introduced in Section 3.2. A number of these capabilities need to be taken into consideration from the very beginning when thinking about how future dataplanes need to evolve.
While many of those capabilities are well known, the past has shown that retrofitting dataplanes with such capabilities after the fact and in a way that is adequate to the problem at hand is very hard.
Many of the services that need to be supported in the future have in common that they place very high demands on latency and precision that need to be supported at very high scales, coupled with expectations of zero packet loss and much higher availability than today.
In order to assure in-time and on-time services with high levels of accuracy, advances in measurements and telemetry will be required in order to monitor and validate that promised service levels are indeed being delivered. This requires advanced instrumentation that is ideally built-in all the way to the protocol level.
For example, the ability to identify and automatically eliminate potential sources of service-level degradations and fluctuations will become of increasing importance. This requires the ability to generate corresponding telemetry data and the ability to observe the performance of packets as they traverse the network. Some of the challenges that need to be addressed include the very high volume of data that gets generated and needs to be assessed, and the effects of the collection itself on performance. In general, greater emphasis will need to be placed on the ability to monitor, observe, and validate packet performance and behavior than is the case today. For seamless support, these capabilities will be inherently integrated with the forwarding function itself, for example delivered together with the packets. Today’s solution approach, IOAM, is a promising technology currently that points in the right direction, and that also highlights some of the challenges - from MTU considerations due to extending packet sizes to the ability to customize and obtain the “right” data. It will therefore be not sufficient by itself. Data to be generated from the network will need to be “smarter”, i.e. more insightful and action-able. This will require additional abilities to process data “on-device”. In additional, the need for new management functions may arise, such as functions that allow to validate adherence with agreed-upon service levels for a flow as a whole, and to prevent data or privacy leakage as well as provide evidence for the possibility or absence of such leakage.
Operators need to be able to rapidly introduce new network services and adapt to new contexts and application needs. This will require advances in network programmability. Today’s model of vendor-defined (supporting service features via new firmware or hardware-based networking features) or operator-defined (supporting service features via programmable software-defined networking (SDN) controllers, virtualized network functions (VNF) and Network Function Virtualization (NFV), and service function chaining (SFC) will no longer be sufficient.
Software Defined Networking and Network Function Virtualization (NFV) have opened up the possibility to accelerate development life-cycles and enable network providers to develop new networking features on their own if needed. Segment Routing is being evolved for that purpose as well. Furthermore, network slicing promises more agility in the introduction of new network services. However, the complexity of the associated controller software results in its own challenges with software development cycles that, while more agile than life-cycles before, are still prohibitive and that can only be undertaken by network providers, not by their customers. Rapid customization of networking services for specific needs or adaptation to unique deployments are out of reach for network provider customers. What is lacking is the ability for applications to rapidly introduce and customize novel behavior at the network flow level, without need to introduce application-level over-the-top (OTT) overlays. Such a capability would be analogous to server-less computing that is revolutionizing cloud services today. In addition, it should be noted that softwarized networks are built on relatively stable (and slowly evolving) underlying physical commodity hardware network infrastructure. This is insufficient to deliver on new high-precision network services, which require hardware advances at many levels to provide programmable flow and QoS behavior at line rate, affecting everything from queuing and scheduling to packet processing pipelines.
The evolution of forwarding planes should allow development life-cycles that are much more agile than today and move from “Dev Ops” to “Flow Ops” (i.e. dynamic programmability of networks at the flow level).
This requires support of novel network and data-plane programming models which can possibly be delivered and effected via the forwarding plane itself.
The possibility of security threats increases with complexity of networks, the potential ramifications of attacks are growing more serious with increasing mission-criticality of networking services and applications.
The forwarding plane plays a large role in the ability to thwart attacks.
For example, the fact that source addresses are not authenticated in existing IP is at the root of a wide range security problems from phishing and fraudulent impersonation designed to compromise and steal user assets to amplification attacks designed to bring down services.
Going forward, it is absolutely critical, then, to minimize the attack surface of the forwarding plane as it evolves.
A key security aspects needed from the network point of view includes to verify if the packet is authorized to enter into the network and if it is sufficiently integrity protected. However, when packets are emitted from the host for these new communication services, the network portion of the packet (e.g., an extension header or an overlay header) should not be encrypted because network nodes may need to interpret the header and provide the desired service.
Lack of encryption and integrity validation, of course, would at the same time increase the threat surface and open up the possibility for attacks.
Mechanisms for authorization and integrity protection must be developed to meet the line rate performance as services delivered can be time sensitive. At the same time, the size of packets should not be significantly increased to avoid negative impact on utilization and overhead tax.
This limits the options for additional security collateral that can be included with packets.
Homomorphic forms of encryption may need to be devised in which network operations can be performed in privacy-preserving manner on encrypted packet headers and tunneled packets without exposing any of their contents.
Another dimension to security arises when the end to end service that needs to be delivered crosses the administrative boundary of the originating host. For those cases, additional mechanisms need to be specified to sufficiently ensure the privacy and confidentiality of the network layer information. While there are lot of avenues to tackle these issues and some aspects are being looked into by various Standards Development Organizations, e.g. IRTF PANRG on Path-Aware Networking, comprehensive solutions are yet to be worked out.
Any mechanisms specified for authorization, integrity protection, and network header confidentiality should be orthogonal to the transport layer and above transport layer security mechanisms set in place by the end host/user. Regardless of whether or not the latest security advances in transport and layers above (e.g. TLS1.3, QUIC or HTTPSx) are applied on the payload, network nodes should not have to act on this information to deliver new services to avoid layer violations.
As future network services are deployed, deployment scenarios will include cases in which packets need to traverse trust boundaries which are under different administrative domains. As the forwarding plane evolves, it should do so in such a way that trustworthiness of packets is maintained - i.e. integrity of data is protected, tampering with packet meta-data (such as source authentication or service level telemetry) would be evident, and privacy of users is guarded.
Ultra-low-latency requirements and the huge increase of bandwidth demands of new services such as holographic type communication services make retransmission as a mechanism to recover data that was lost in transit increasingly less feasible. Therefore, network resilience and avoidance of loss becomes of paramount importance.
There are many methods for providing network resilience. This includes providing redundancy and diversity of both physical (e.g. ports, routers, line cards) and logical (e.g. shapers, policers, classifiers) entities. It also includes the use of protocols that provide quick re-convergence and maintain high availability of existing connections after a failure event occurs in the network. Other techniques include packet replication or network coding and error correction techniques to overcome packet loss.
As the forwarding plane evolves, mechanisms to provide network resilience should be inherently supported.
Today, there is a growing awareness of the lack of privacy in the Internet and its implications.
New network services have to be sensitive to and comply with heightened user privacy expectations.
At the same time, the need for privacy needs to be balanced with legitimate needs of network providers to operate and maintain their networks, which requires some visibility into what is happening on the network and how it is being used.
Likewise, mechanisms to provide privacy must be provided in such a way to not compromise security, such as allowing anonymous attackers to prey on other users.
An evolved forwarding plane must provide mechanisms that ensure users privacy by design and prevent illegitimate exposing of personally-identifiable information (PII), while preventing abuse of those mechanisms by attack exploits and while affording network providers with legitimate visibility into use of their network and services.
There are a variety of privacy-related requirements that ensue, such as:
Potential mechanisms to consider include (but are not limited to) avoiding the need for long-lived addresses (to prevent trackablity) and the use of homomorphic encryption for packet headers and tunneled packets (in addition to traditional payload encryption) that allow to perform network operations in privacy-preserving manner without exposing meta-data carried in headers.
Many new services demand guarantees instead of being accepting of “best effort”.
As a result, today’s “best effort” accounting may no longer be sufficient.
Today’s accounting technology largely relies on interface statistics and flow records.
Those statistics and records may not be entirely accurate.
For example, in many cases their generation involves sampling and is thus subject to sampling inaccuracies.
In addition, this data largely accounts for volume but not so much for actual service levels (e.g. latencies, let alone coordination across flows) that are delivered.
Service level measurements can be used to complement other statistics but come with significant overhead and also have various limitations, from sampling to the consumption of network and edge node processing bandwidth.
Techniques that rely on passive measurements are infeasible in many network deployments and hampered by encryption as well as issues relating to privacy.
Guarantees demand their price. This makes it increasingly important both for providers and users of services to be able to validate that promised service levels were delivered on.
For example, proof of service delivery (including proof of service level delivery) may need to be provided to account and charge for network services.
This will require advances in accounting technology that should be considered as forwarding technology evolves, possibly providing accounting as a function that is intrinsically coupled with forwarding itself.
| [RFC0791] | Postel, J., "Internet Protocol", STD 5, RFC 791, DOI 10.17487/RFC0791, September 1981. |
| [RFC8200] | Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", STD 86, RFC 8200, DOI 10.17487/RFC8200, July 2017. |