| DetNet | J. Korhonen, Ed. |
| Internet-Draft | Broadcom |
| Intended status: Informational | J. Farkas |
| Expires: February 18, 2017 | G. Mirsky |
| Ericsson | |
| P. Thubert | |
| Cisco | |
| Y. Zhuang | |
| Huawei | |
| L. Berger | |
| LabN | |
| August 17, 2016 |
DetNet Data Plane Protocol and Solution Alternatives
draft-dt-detnet-dp-alt-03
This document identifies existing IP and MPLS, and other encapsulations that run over IP and/or MPLS data plane technologies that can be considered as the base line solution for deterministic networking data plane definition.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 18, 2017.
Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Deterministic Networking (DetNet) [I-D.ietf-detnet-problem-statement] provides a capability to carry unicast or multicast data flows for real-time applications with extremely low data loss rates, timely delivery and bounded packet delay variation [I-D.finn-detnet-architecture]. The deterministic networking Quality of Service (QoS) is expressed as 1) the minimum and the maximum end-to-end latency from source (talker) to destination (listener), and 2) probability of loss of a packet. Only the worst-case values for the mentioned parameters are concerned.
There are three techniques to achieve the QoS required by deterministic networks: [RFC3031], layer-2 or layer-3 encapsulations and transport protocols that could be considered as foundations for a deterministic networking data plane. The full scope of the deterministic networking data plane solution is considered including, as appropriate: quality of service (QoS); Operations, Administration and Maintenance (OAM); and time synchronization among other criteria described in Section 4.
This document identifies existing IP and Multiprotocol Label Switching (MPLS)
This document does not select a deterministic networking data plane protocol. It does, however, elaborate what it would require to adapt and use a specific protocol as the deterministic networking data plane solution. This document is only concerned with data plane considerations and, specifically, with topics that potentially impact potential deterministic networking aware data plane hardware. Control plane considerations are out of scope of this document.
This document uses the terminology established in the DetNet architecture [I-D.finn-detnet-architecture].
A "Deterministic Network" will be composed of DetNet enabled nodes i.e., End Systems, Edge Nodes, Relay Nodes and collectively deliver DetNet services. DetNet enabled nodes are interconnected via Transit Nodes (i.e., routers) which support DetNet, but are not DetNet service aware. Transit nodes see DetNet nodes as end points. All DetNet enabled nodes are connect to sub-networks, where a point-to-point link is also considered as a simple sub-network. These sub-networks will provide DetNet compatible service for support of DetNet traffic. Examples of sub-networks include IEEE 802.1TSN and OTN. Of course, multi-layer DetNet systems may also be possible, where one DetNet appears as a sub-network, and provides service to, a higher layer DetNet system. A simple DetNet concept network is shown in Figure 1.
TSN Edge Transit Relay DetNet
End System Node Node Node End System
+---------+ +.........+ +---------+
| Appl. |<---:Svc Proxy:-- End to End Service ---------->| Appl. |
+---------+ +---------+ +---------+ +---------+
| TSN | |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service |
+---------+ +---+ +---+ +---------+ +---------+ +---------+
|Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport|
+-------.-+ +-.-+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+
: Link : / ,-----. \ : Link : / ,-----. \
+........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+
[Network] [Network]
`-----' `-----'
Figure 1: A Simple DetNet Enabled Network
The DetNet data plane is logically divided into two layers (also see Figure 2):
Distinguishing the function of these two DetNet data plane layers helps to explore and evaluate various combinations of the data plane solutions available. This separation of DetNet layers, while helpful, should not be considered as formal requirement. For example, some technologies may violate these strict layers and still be able to deliver a DetNet service.
.
.
+-----------+
| Service | PW, RTP/(UDP), GRE
+-----------+
| Transport | (UDP)/IPv6, (UDP)/IPv4, MPLS LSPs, BIER, BIER-TE
+-----------+
.
.
Figure 2: DetNet adaptation to data plane
The two logical layers defined here aim to help to identify which data plane technology can be used for what purposes in the DetNet context. This layering is similar to the data plane concept of MPLS, where some part of the label stack is "Service" specific (e.g., PW labels, VPN labels) and an other part is "Transport" specific (e.g, LSP label, TE label(s)).
In some networking scenarios, the end system initially provides a DetNet flow encapsulation, which contains all information needed by DetNet nodes (e.g., Real-time Transport Protocol (RTP) [RFC3550] based DetNet flow transported over a native UDP/IP network or PseudoWire). In other scenarios, the encapsulation formats might differ significantly. As an example, a CPRI "application's" I/Q data mapped directly to Ethernet frames may have to be transported over an MPLS-based packet switched network (PSN).
There are many valid options to create a data plane solution for DetNet traffic by selecting a technology approach for the DetNet Service layer and also selecting a technology approach for the DetNet Transport layer. There are a high number of valid combinations. Therefore, not the combinations but the different technologies are evaluated along the criteria collected in Section 4. Different criteria apply for the DetNet Service layer and the DetNet Transport layer, however, some of the criteria are valid for both layers.
One of the most fundamental differences between different potential data plane options is the basic addressing and headers used by DetNet end systems. For example, is the basic service a Layer 2 (e.g., Ethernet) or Layer 3 (i.e., IP) service. This decision impacts how DetNet end systems are addressed, and the basic forwarding logic for the DetNet Service layer.
In an attempt to illustrate a DetNet date plane, this document uses the Multi-Segment Pseudowire Emulation Edge-to-Edge (PWE3) [RFC5254] reference model shown in Figure 3 as the foundation for different DetNet data plane deployment options and how layering could work. Other reference models are possible but not covered in this document. Note that other technologies can be also used to implement DetNet, Multi-Segment PW is only used here to illustrate functions, features and layering from the perspective of the architecture.
Native |<--------Multi-Segment Pseudowire----->| Native
Service | PSN PSN | Service
(AC) | |<-Tunnel->| |<-Tunnel->| | (AC)
| V V 1 V V 2 V V |
| +-----+ +-----+ +---- + |
+---+ | |T-PE1|==========|S-PE1|==========|T-PE2| | +---+
| |---|-----|........PW1...........|...PW3..........|---|----| |
|CE1| | | | | | | | | |CE2|
| |---------|........PW2...........|...PW4..........|--------| |
+---+ | | |==========| |==========| | | +---+
^ +-----+ +-----+ +-----+ ^
| Provider Edge 1 ^ Provider Edge 3 |
| | |
| PW switching point |
| |
|<------------------- Emulated Service ------------------->|
Figure 3: Pseudo Wire switching reference model
Figure 4 illustrates how DetNet can provide services for IEEE 802.1TSN end systems over a DetNet enabled network. The edge nodes insert and remove required DetNet data plane encapsulation. The 'X' in the edge and relay nodes represents a potential DetNet flow packet replication and elimination point. This conceptually parallels L2VPN services, and could leverage existing related solutions as discussed below.
TSN |<----- End to End DetNet Service ----->| TSN
Service | Transit Transit | Service
TSN (AC) | |<-Tunnel->| |<-Tunnel->| | (AC) TSN
End | V V 1 V V 2 V V | End
System | +-----+ +-----+ +---- + | System
+---+ | |T-PE1|==========|S-PE1|==========|T-PE2| | +---+
| |---|-----|.X_..DetNet Flow1..X..|...DF3........X.|---|----| |
|CE1| | | \ | | | | / | | |CE2|
| | |...X_...DF2........X..|...DF4......X_..| | |
+---+ | |==========| |==========| | +---+
^ +-----+ +-----+ +-----+ ^
| Edge Node Relay Node Edge Node |
| |
|<--------------- Emulated TSN Service ------------------->|
Figure 4: IEEE 802.1TSN over DetNet
Figure 5 illustrates how end to end native DetNet service can be provided. In this case, the end systems are able to send and receive native DetNet flows. For example, as PseudoWire (PW) encapsulated IP. Like earlier the 'X' in the end systems, edge and relay nodes represents potential DetNet flow packet replication and elimination points. Here the relay nodes may change the underlying transport, for example replacing IP with MPLS or tunneling IP over MPLS (e.g., via L3VPNs), or simply interconnect network domains.
DetNet DetNet
Service Transit Transit Service
DetNet | |<-Tunnel->| |<-Tunnel->| | DetNet
End | V 1 V V 2 V | End
System | +-----+ +-----+ +-----+ | System
+---+ | |S-PE1|==========|S-PE2|==========|S-PE3| | +---+
| X....DFa.....X_.......DF1.......X_....DF3........X.....DFa...X |
|CE1|=========| \ | | / | | / |========|CE2|
| | | | \......DF2.....X_......DF4....../ | | | |
+---+ | |==========| |==========| | +---+
^ +-----+ +-----+ +-----+ ^
| Relay Node Relay Node Relay Node |
| |
|<------------- End to End DetNet Service ---------------->|
Figure 5: Native DetNet
Figure 6 illustrates how a IEEE 802.1TSN end system could communicate with a native DetNet end system through an edge node which provides a TSN to DetNet inter-working capability. The edge node would add and remove required DetNet data plane encapsulation as well as provide any needed address mapping. As in previous figures, the 'X' in the end systems, edge and relay nodes represents potential DetNet flow packet duplication and elimination points.
TSN |<----- End to End DetNet Service -------------->|
Service | Transit Transit |
TSN (AC) | |<-Tunnel->| |<-Tunnel->| DetNet | DetNet
End | V V 1 V V 2 V Service | End
System | +-----+ +-----+ +-----+ | V System
+---+ | |T-PE1|==========|S-PE1|==========|S-PE2| | +---+
| |---|-----|.X_.......DF1......X..|...DF3........X.|...DFa...X |
|CE1| | | \ | | | | / |========|CE2|
| | | \.....DF2.......X..|...DF4....../ | | | |
+---+ | |==========| |==========| | +---+
^ +-----+ +-----+ +-----+ ^
| Edge Node Relay Node Relay Node |
| |
|<----------------- End to End Service ------------------->|
Figure 6: IEEE 802.1TSN to native DetNet
This section provides criteria to help to evaluate potential options. Each deterministic networking data plane solution alternative is described and evaluated using the criteria described in this section. The used criteria enumerated in this section are selected so that they highlight the existence or lack of features that are expected or seen important to a solution alternative for the data plane solution.
The criteria for the DetNet Service layer:
The criteria for the DetNet Transport layer:
[Editor's Note: numbering is off because #7 is removed.]
[Editor's Note: #9 should(?) be integrated into #6.]
Most of the criteria is relevant for both the DetNet Service and DetNet Transport layers. However, different aspects of the same criteria may relevant for different layers, for example, as it is the case with criteria #5 Packet replication and elimination.
Encapsulation and overhead is related to how the DetNet data plane carries DetNet flow. In several cases a DetNet flow has to be encapsulated inside other protocols, for example, when transporting a layer-2 Ethernet frame over an IP transport network. In some cases a tunneling like encapsulation can be avoided by underlying transport protocol translation, for example, translating layer-2 Ethernet frame including addressing and flow identification into native IP traffic. Last it is possible that sources and destinations handle deterministic flows natively in layer-3. This criteria concerns what is the encapsulation method the solution alternative support: tunneling like encapsulation, protocol translation or native layer-3 transport. In addition to the encapsulation mechanism this criteria is also concerned of the processing and specifically the encapsulate header overhead.
The solution alternative has to provide means to identify specific deterministic flows. The flow identification can, for example, be explicit field in the data plane encapsulation header or implicitly encoded into the addressing scheme of the used data plane protocol or their combination. This criteria concerns the availability and details of deterministic flow identification the data plane protocol alternative has.
The solution alternative has to provide means for end systems to number packets sequentially and transport that sequencing information along with the sent packets. In addition to possible reordering packets other important uses for sequencing are detecting duplicates and lost packets.
In a case of intentional packet duplication a combination of flow identification and packet sequencing allows for detecting and eliminating duplicates at the destination (see Section 4.5 for more details).
The solution alternative has to provide a mechanism(s) for establishing explicit routes that all packets belonging to a deterministic flow will follow. The explicit route can be seen as a form of source routing or a pre-reserved path e.g., using some network management procedure. It should be noted that the explicit route does not need to be detailed to a level where every possible intermediate node along the path is part of the named explicit route. RSVP-TE [RFC3209] supports explicit routes, and typically provides pinned data paths for established LSPs. At Layer-2, the IEEE 802.1Qca [IEEE802.1Qca] specification defines how to do explicit path control in a bridged network and its IETF counter part is defined in [RFC7813]. This criteria concerns the available mechanisms for explicit routes for the data plane protocol alternative.
Flow duplication and flow merging are methods being considered to provide DetNet service protection. The objective for supporting flow duplication and flow merging is to enable hitless (or lossless) 1+1 protection. Other methods, if so identified, are also permissible.
The solution alternative has to provide means for end systems, relay and edge nodes to be able to duplicate packets into duplicate flows, and later merge the flows into one for duplicate elimination. The duplication and merging may take place at multiple points in the network in order to ensure that one (or more) equipment failure event(s) still leave at least one path intact for a deterministic networking flow. The goal is again to enable hitless 1+1 protection in a way that no packet gets lost or there is no ramp up time when either one of the paths fails for one reason or another.
Another concern regarding packet duplication is how to enforce duplicated packets to take different route or path while the final destination still remains the same. With strict source routing, all the intermediate hops are listed and paths can be guaranteed to be non-overlapping. Loose source routing only signals some of the intermediate hops and it takes additional knowledge to ensure that there is no single point of failure.
The IEEE 802.1CB (seamless redundancy) [IEEE8021CB] is an example of Ethernet-based solution that defines packet sequence numbering, flow duplication, flow merging, duplicate packet identification and elimination. The deterministic networking data plane solution alternative at layer-3 has to provide equivalent functionality. This criteria concerns the available mechanisms for packet replication and duplicate deletion the data plane protocol alternative has.
The solution alternative should demonstrate an availability of appropriate standardized OAM tools that can be extended for deterministic networking purposes with a reasonable effort, when required. The OAM tools do not necessarily need to be specific to the data plane protocol as it could be the case, for example, with MPLS-based data planes. But any OAM-related implications or requirements on data plane hardware must be considered.
The OAM includes but is not limited to tools listed in the requirements for overlay networks [I-D.ooamdt-rtgwg-ooam-requirement]. Specifically, the performance management requirements are of interest at both service and transport layers.
Class and quality of service, i.e., CoS and QoS, are terms that are often used interchangeably and confused. In the context of DetNet, CoS is used to refer to mechanisms that provide traffic forwarding treatment based on aggregate group basis and QoS is used to refer to mechanisms that provide traffic forwarding treatment based on a specific DetNet flow basis. Examples of CoS mechanisms include DiffServ which is enabled by IP header differentiated services code point (DSCP) field [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer-2, by IEEE 802.1p priority code point (PCP).
Quality of Service (QoS) mechanisms for flow specific traffic treatment typically includes a guarantee/agreement for the service, and allocation of resources to support the service. Example QoS mechanisms include discrete resource allocation, admission control, flow identification and isolation, and sometimes path control, traffic protection, shaping, policing and remarking. Example protocols that support QoS control include Resource ReSerVation Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473].
A critical DetNet service enabled by QoS (and perhaps CoS) is delivering zero congestion loss. There are different mechanisms that maybe used separately or in combination to deliver a zero congestion loss service. The key aspect of this objective is that DetNet packets are not discarded due to congestion at any point in a DetNet aware network.
In the context of the data plane solution there should be means for flow identification, which then can be used to map a flow against specific resources and treatment in a node enforcing the QoS. Hereto, certain aspects of CoS and QoS may be provided by the underlying sub-net technology, e.g., actual queuing or IEEE 802.3x priority flow control (PFC).
For the network management and specifically for tracing implementation or network configuration errors any means to find out whether a packet is a replica, which node performed replication, and which path was intended for the replica, can be very useful. This criteria concerns the availability of solutions for tracing packets in the context of data plane protocol alternative. Packet traceability can also be part of OAM.
The technical maturity of the data plane solution alternative is crucial, since it basically defines the effort, time line and risks involved for the use of the solution in deployments. For example, the maturity level can be categorized as available immediately, available with small extensions, available with re-purposing/redefining portions of the protocol or its header fields. Yet another important measure for maturity is the deployment experience. This criteria concerns the maturity of the data plane protocol alternative as the solution alternative. This criteria is particularly important given, as previously noted, that the DetNet data plane solution is expected to impact, i.e., be supported in, hardware.
The following sections describe and rate deterministic data plane solution alternatives. In "Analysis and Discussion" section each alternative is evaluated against the criteria given in Section 4 and rated using the following: (M)eets the criteria, (W)ork needed, and (N)ot suitable or too much work envisioned.
This section investigates the application of native IPv6 [RFC2460] as the data plane for deterministic networking along the criteria collected in Section 4.
The application of higher OSI layer headers, i.e., headers deeper in the packet, can be considered. Two aspects have to be taken into account for such solutions. (i) Those header fields can be encrypted. (ii) Those header fields are deeper in the packet, therefore, routers have to apply deep packet inspection. See further details in Section 5.2.5.
IPv6 supports a significant portion of the identified DetNet data plane criteria today. There are aspects of the DetNet data plane that are not fully supported, notably QoS, but these can be incrementally added or supplemented by the underlying sub-network layer. IPv6 may be a choice as the DetNet Transport layer in networks where other technologies such as MPLS are not deployed.
IPv4 [RFC0791] is in principle the same as IPv6, except that it has a smaller address space. However, IPv6 was designed around the fact that extension headers are an integral part of the protocol and operation from the beginning, although the practice may some times prove differently [RFC7872]. IPv4 does support header options, but these have historically not been supported on in hardware-based forwarding so are generally blocked or handled at a much slower rate. In either case, the use of IP header options is generally avoided. In the context of deterministic networking data plane solutions the major difference between IPv4 and IPv6 seems to be the practical support for header extensibility. Anything below and above the IP header independent of the version is practically the same.
The IPv4 has specifications to support most of the identified DetNet data plane criteria today. However, several of those have already been deprecated or their wide support is not guaranteed. The DetNet data plane criteria that are not fully supported could be incrementally added or supplemented by the underlying sub-network layer. Unfortunately, the IPv4 has had limited success getting its extensions deployed at large. However, introducing new extensions might have a better success in closed networks (like DetNet) than in Internet. Due to the popularity of the IPv4, it should be considered as a potential choice for the DetNet Transport layer.
Multiprotocol Label Switching Architecture (MPLS) [RFC3031] and its variants, MPLS with Traffic Engineering (MPLS-TE) [RFC3209] and [RFC3473], and MPLS Transport Profile (MPLS-TP) [RFC5921] is a widely deployed technology that switches traffic based on MPLS label stacks [RFC3032] and [RFC5960]. MPLS is the foundation for Pseudowire-based services Section 5.2.3 and emerging technologies such as Bit-Indexed Explicit Replication (BIER) Section 5.1.4 and Source Packet Routing.
MPLS supports the equivalent of both the DetNet Service and DetNet Transport layers, and provides a very rich set of mechanisms that can be reused directly, and perhaps augmented in certain cases, to deliver DetNet services. At the DetNet Transport layer, MPLS provides forwarding, protection and OAM services. At the DetNet Service Layer it provides client service adaption, directly, via Pseudowires Section 5.2.3 and via other label-like mechanisms such as EPVN Section 5.2.4. A representation of these options are shown in Figure 7.
PW-Based EVPN Labeled IP
Services Services Transport
|------------| |-----------------------------| |------------|
Emulated EVPN over LSP EVPN w/ ESI ID IP
Service
+------------+
| Payload |
+------------+ +------------+ +------------+ (Service)
| PW Payload | | Payload | |ESI Lbl(S=1)|
+------------+ +------------+ +------------+ +------------+
|PW Lbl(S=1) | |VPN Lbl(S=1)| |VPN Lbl(S=0)| | IP |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=1)|
+------------+ +------------+ +------------+ +------------+
. . . .
. . . . (Transport)
. . . .
~~~~~~~~~~~ denotes DetNet Service <-> DetNet Transport layer boundary
Figure 7: MPLS-based Services
MPLS can be controlled in a number of ways including via a control plane, via the management plane, or via centralized controller (SDN) based approaches. MPLS also provides standard control plane reference points. Additional information on MPLS architecture and control can be found in [RFC5921]. A summary of MPLS control plane related functions can be found in [RFC6373]. The remainder of this section will focus [RFC6373]. The remainder of this section will focus on the MPLS transport data plane, additional information on the MPLS service data plane can be found below in Section 5.2.2.
The following draws heavily from [RFC5960].
Encapsulation and forwarding of packets traversing MPLS LSPs follows standard MPLS packet encapsulation and forwarding as defined in [RFC3031], [RFC3032], [RFC5331], and [RFC5332].
Data plane Quality of Service capabilities are included in the MPLS in the form of Traffic Engineered (TE) LSPs [RFC3209] and the MPLS Differentiated Services (DiffServ) architecture [RFC3270]. Both E-LSP and L-LSP MPLS DiffServ modes are defined. The Traffic Class field (formerly the EXP field) of an MPLS label follows the definition of [RFC5462] and [RFC3270].
Except for transient packet reordering that may occur, for example, during fault conditions, packets are delivered in order on L-LSPs, and on E-LSPs within a specific ordered aggregate.
The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL processing models are described in [RFC3270] and [RFC3443] and may be used for MPLS LSPs.
Equal-Cost Multi-Path (ECMP) load-balancing is possible with MPLS LSPs and can be avoided using a number of techniques. The same holds for Penultimate Hop Popping (PHP).
MPLS includes the following LSP types:
Point-to-point unidirectional LSPs are supported by the basic MPLS architecture [RFC3031].
A point-to-point associated bidirectional LSP between LSRs A and B consists of two unidirectional point-to-point LSPs, one from A to B and the other from B to A, which are regarded as a pair providing a single logical bidirectional transport path.
A point-to-point co-routed bidirectional LSP is a point-to-point associated bidirectional LSP with the additional constraint that its two unidirectional component LSPs in each direction follow the same path (in terms of both nodes and links). An important property of co-routed bidirectional LSPs is that their unidirectional component LSPs share fate.
A point-to-multipoint unidirectional LSP functions in the same manner in the data plane, with respect to basic label processing and packet-switching operations, as a point-to-point unidirectional LSP, with one difference: an LSR may have more than one (egress interface, outgoing label) pair associated with the LSP, and any packet it transmits on the LSP is transmitted out all associated egress interfaces. Point-to-multipoint LSPs are described in [RFC4875] and [RFC5332]. TTL processing and exception handling for point-to- multipoint LSPs is the same as for point-to-point LSPs.
Additional data plane capabilities include Linear Protection, [RFC6378] and [RFC7271]. And the in progress work on MPLS support for time synchronization [I-D.ietf-mpls-residence-time].
MPLS is a mature technology that has been widely deployed. Numerous vendor products and multiple generations of MPLS hardware have been built and deployed. MPLS LSPs support a significant portion of the identified DetNet data plane criteria today. Aspects of the DetNet data plane that are not fully supported can be incrementally added. It's worth noting that a number of limitations are in shipping hardware, versus at the protocol specification level, e.g., shaping.
Bit Indexed Explicit Replication [I-D.ietf-bier-architecture] (BIER) is a network plane replication technique that was initially intended as a new method for multicast distribution. In a nutshell, a BIER header includes a bitmap that explicitly signals the destinations that are intended for a particular packet, which means that 1) the source is aware of the individual destinations and 2) the BIER control plane is a simple extension of the unicast routing as opposed to a dedicated multicast data plane, which represents a considerable reduction in OPEX. For this reason, the technology faces a lot of traction from Service Providers. Section 5.1.4 discusses the applicability of BIER for replication in the DetNet.
The simplicity of the BIER technology makes it very versatile as a network plane signaling protocol. Already, a new Traffic Engineering variation is emerging that uses bits to signal segments along a TE path. While the more classical BIER is mainly a multicast technology that typically leverages a unicast distributed control plane through IGP extensions, BIER-TE is mainly a unicast technology that leverages a central computation to setup path, compute segments and install the mapping in the intermediate nodes. Section 5.1.5 discusses the applicability of BIER-TE for replication, traceability and OAM operations in DetNet.
Bit-Indexed Explicit Replication (BIER) layer may be considered to be included into Deterministic Networking data plane solution. Encapsulation of a BIER packet in MPLS network presented in Figure 8
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Label Stack Element | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Label Stack Element | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | BIER-MPLS label | |1| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0 1 0 1| Ver | Len | Entropy | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | BitString (first 32 bits) ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ BitString (last 32 bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |OAM| Reserved | Proto | BFIR-id | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8: BIER packet in MPLS encapsulation
The DetNet may be presented in BIER as distinctive payload type with its own Proto(col) ID. Then it is likely that DetNet will have the header that would identify:
DetNet node, collocated with BFIR, may use multiple BIER sub-domains to create replicated flows. Downstream DetNet nodes, collocated with BFER, would terminate redundant flows based on Sequence Number and/or Timestamp information. Such DetNet may be BFER in one BIER sub-domain and BFIR in another. Thus DetNet flow would traverse several BIER sub-domains.
+-----+
| A |
+-----+
/ \
. .
/ .
. \
/ .
. .
/ \
+-----+ +-----+
| B | | C |
+-----+ +-----+
/ \ / \
. . . .
/ \ . .
. . / \
/ \ . .
. . . .
/ \ / \
+-----+ +-----+ +-----+
| D | | E | | F |
+-----+ +-----+ +-----+
\ . . /
. . . .
\ . . .
. . . /
\ . . .
. . . .
\ . . /
+-----+ +-----+
| G | | H |
+-----+ +-----+
Figure 9: DetNet in BIER domain
Consider DetNet flow that must traverse BIER enabled domain from A to G and H. DetNet may use three BIER subdomains:
DetNet node A sends DetNet into red and purple BIER sub-domains. DetNet node E receives DetNet packet and sends into green sub-domain while terminating duplicates and those that deemed too-late.
DetNet nodes G and H receive DetNet flows, terminate duplicates and those that are too-late.
BIER over MPLS supports a significant portion of the identified DetNet data plane requirements, including controlled packet replication, traffic engineering, while some requirements, e.g. duplicate and too-late packet elimination may be realized as function of the DetNet overlay. BIER over MPLS is a viable candidate as the DetNet Transport layer in MPLS networks.
An alternate use of Bit-Indexed Explicit Replication (BIER) uses bits in the BitString to represent adjacencies as opposed to destinations, as discussed in BIER Traffic Engineering (TE) [I-D.eckert-bier-te-arch].
The proposed function of BIER-TE in the DetNet data plane is to control the process of replication and elimination, as opposed to the identification of the flows or and the sequencing of packets within a flow.
At the path ingress, BIER-TE identifies the adjacencies that are activated for this packet (under the rule of the controller). At the egress, BIER-TE is used to identify the adjacencies where transmission failed. This information is passed to the controller, which in turn can modify the active adjacencies for the next packets.
The value is that the replication can be controlled and monitored in a loop that may involve an external controller, with the granularity of a packet and an adjacency .
BIER-TE enables to activate the replication and elimination functions in a manner that is abstract to the data plane forwarding information. An adjacency, which is represented by a bit in the BIER header, can correspond in the data plane to an Ethernet hop, a Label Switched Path, or it can correspond to an IPv6 loose or strict source routed path.
In a nutshell, BIER-TE is used as follows:
===> (A) ====> (C) ====
// ^ | ^ | \\
ingress (I) | | | | (E) egress
\\ | v | v //
===> (B) ====> (D) ====
Figure 10: Ladder Shape with replication and elimination Points
===> (A) ====> (C) ====
// 1 ^ | 4 ^ | 7 \\
ingress (I) |2| |6| (E) egress
\\ 3 | v 5 | v 8 //
===> (B) ====> (D) ====
Figure 11: Assigning Bits
| Bit # | Adjacency | Owner | Example Bit Setting |
|---|---|---|---|
| 1 | I->A | I | 1 |
| 2 | A->B | A | 1 |
| B->A | B | ||
| 3 | I->C | I | 0 |
| 4 | A->C | A | 1 |
| 5 | B->D | B | 1 |
| 6 | C->D | C | 1 |
| D->C | D | ||
| 7 | C->E | C | 1 |
| 8 | D->E | D | 0 |
replication and elimination Protecting A->C
====> Repl ===> Elim ====
// | ^ \\
ingress | | egress
v |
Fwd ====> Fwd
Figure 12: Enabled Adjacencies
| Adjacency | BIER BitString |
|---|---|
| I->A | 01011110 |
| A->B | 00011110 |
| B->D | 00010110 |
| D->C | 00010010 |
| A->C | 01001110 |
BitString in BIER Header as Packet Progresses
| Operation | BIER BitString |
|---|---|
| D->C | 00010010 |
| A->C | 01001110 |
| -------- | |
| AND in C | 00000010 |
| C->E | 00000000 |
BitString Processing at Elimination Point C
| Failing Adjacency | Egress BIER BitString |
|---|---|
| I->A | Frame Lost |
| I->B | Not Tried |
| A->C | 00010000 |
| A->B | 01001100 |
| B->D | 01001100 |
| D->C | 01001100 |
| C->E | Frame Lost |
| D->E | Not Tried |
BitString indicating failures
In more details:
The BIER header is of variable size, and a DetNet network of a limited size can use a model with 64 bits if 64 adjacencies are enough, whereas a larger deployment may be able to signal up to 256 adjacencies for use in very complex paths. Figure 8 illustrates a BIER header as encapsulated within MPLS. The format of this header is common to BIER and BIER-TE.
For the DetNet data plane, a replication point is an ingress point for more than one adjacency, and an elimination point is an egress point for more than one adjacency.
A pre-populated state in a replication node indicates which bits are served by this node and to which adjacency each of these bits corresponds. With DetNet, the state is typically installed by a controller entity such as a PCE. The way the adjacency is signaled in the packet is fully abstracted in the bit representation and must be provisioned to the replication nodes and maintained as a local state, together with the timing or shaping information for the associated flow.
The DetNet data plane uses BIER-TE to control which adjacencies are used for a given packet. This is signaled from the path ingress, which sets the appropriate bits in the BIER BitString to indicate which replication must happen.
The replication point clears the bit associated to the adjacency where the replica is placed, and the elimination points perform a logical AND of the BitStrings of the copies that it gets before forwarding.
As is apparent in the examples above, clearing the bits enables to trace a packet to the replication points that made any particular copy. BIER-TE also enables to detect the failing adjacencies or sequences of adjacencies along a path and to activate additional replications to counter balance the failures.
Finally, using the same BIER-TE bit for both directions of the steps of the ladder enables to avoid replication in both directions along the crossing adjacencies. At the time of sending along the step of the ladder, the bit may have been already reset by performing the AND operation with the copy from the other side, in which case the transmission is not needed and does not occur (since the control bit is now off).
BIER-TE occupies a particular position in the DetNet data plane. In the one hand it is optional, and only useful if replication and elimination is taking place. In the other hand, it has unique capabilities to:
However, as noted earlier, BIER-TE is not yet fully specified and the required specification work is outside the scope of the current DetNet WG charter.
Generic Routing Encapsulation (GRE) [RFC2784] provides an encapsulation of an arbitrary network layer protocol over another arbitrary network layer protocol. The encapsulation of a GRE packet can be found in Figure 13.
+-------------------------------+
| Delivery Header |
+-------------------------------+
| GRE Header |
+-------------------------------+
| Payload packet |
+-------------------------------+
Figure 13: Encapsulation of a GRE packet
Based on RFC2784, [RFC2890] further includes sequencing number and Key in optional fields of the GRE header, which may help to transport DetNet traffic flows over IP networks. The format of a GRE header is presented in Figure 14.
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |C| |K|S| Reserved0 | Ver | Protocol Type | +-----------------------------------------------------------------+ | Checksum (optional) | Reserved1 (Optional) | +-----------------------------------------------------------------+ | Key (optional) | +-----------------------------------------------------------------+ | Sequence Number (optional) | +-----------------------------------------------------------------+
Figure 14: Format of a GRE header
As a tunneling protocol, GRE can encapsulate a wide variety of network layer protocols over another network layer, which can naturally serve as the service layer protocol for DetNet. Currently, it supports a portion of the Detnet service layer criteria, and still some are not fully supported but can be incrementally added or supported by delivery protocols at as the transport layer. In general, GRE can be a choice as the DetNet service layer and can work with IPv6 and IPv4 as the DetNet Transport layer.
MPLS based technologies supports both the DetNet Service and DetNet Transport layers. This, as well as a general overview of MPLS, is covered above in Section 5.1.3. These sections focus on the DetNet Service Layer it provides client service adaption, via Pseudowires Section 5.2.3 and via native and other label-like mechanisms such as EPVN in Section 5.2.4. A representation of these options was previously discussed and is shown in Figure 7.
The following text is adapted from [RFC5921]:
Pseudo Wire Emulation Edge-to-Edge (PWE3) [RFC3985] or simply PseudoWires (PW) provide means of emulating the essential attributes and behaviour of a telecommunications service over a packet switched network (PSN) using IP or MPLS transport. In addition to traditional telecommunications services such as T1 line or Frame Relay, PWs also provide transport for Ethernet service [RFC4448] and for generic packet service [RFC6658]. Figure 15 illustrate the reference PWE3 stack model.
+----------------+ +----------------+
|Emulated Service| |Emulated Service|
|(e.g., Eth, ...)|<= Emulated Service =>|(e.g., Eth, ...)|
+----------------+ +----------------+
| Payload | | Payload | CW,
| Encapsulation |<=== Pseudo Wire ====>| Encapsulation | Timing,
| | | | Seq., ..
+----------------+ +----------------+
|PW Demultiplexer| |PW Demultiplexer|
| PSN Tunnel, |<==== PSN Tunnel ====>| PSN Tunnel, | MPLS.
| PSN & Physical | | PSN & Physical | L2TP,
| Layers | | Layers | IP, ..
+-------+--------+ ___________ +---------+------+
| / \ |
+============/ PSN \==============+
\ /
\___________/
Figure 15: PWE3 protocol stack reference model
PWs appear as a good data plane solution alternative for a number of reasons. PWs are a proven and deployed technology with a rich OAM control plane [RFC4447], and enjoy the toolbox developed for MPLS networks in a case of MPLS-based PSN. Furthermore, PWs may have an optional Control Word (CW) as part of the payload encapsulation between the PSN and the emulated service that is, for example, capable of frame sequencing and duplicate detection. The encapsulation layer may also provide timing using RTP as described in Sections 5.2.2, 5.4.1 and 5.4.2 of [RFC3985] and utilized by [RFC4553][RFC5087]. Furthermore, advanced DetNet node functions are conceptually already supported by PW framework (with some added functional required), such as the DetNet Relay node modeled after the Multi-Segment PWE3 [RFC5254].
PWs can be also used if the PSN is IP, which enables the application of PWs in networks that do not have MPLS enabled in their core routers. One approach to provide PWs over IP is to provide MPLS over IP in some way and then leverage what is available for PWs over MPLS. The following standard solutions are available both for IPv4 and IPv6 to follow this approach. The different solutions have different overhead as discussed in the following subsection. The MPLS-in-IP encapsulation is specified by [RFC4023]. The IPv4 Protocol Number field or the IPv6 Next Header field is set to 137, which indicates an MPLS unicast packet. (The use of the MPLS-in-IP encapsulation for MPLS multicast packets is not supported.) The MPLS-in-GRE encapsulation is specified in [RFC4023], where the IP header (either IPv4 or IPv6) is followed by a GRE header, which is followed by an MPLS label stack. The protocol type field in the GRE header is set to MPLS Unicast (0x8847) or Multicast (0x8848). MPLS over L2TPv3 over IP encapsulation is specified by [RFC4817]. The MPLS-in-UDP encapsulation is specified by [RFC7510], where the UDP Destination Port indicates tunneled MPLS packet and the UDP Source Port is an entropy value that is generated by the encapsulator to uniquely identify a flow. MPLS-in-UDP encapsulation can be applied to enable UDP-based ECMP (Equal-Cost Multipath) or Link Aggregation. All these solutions can be secured with IPsec [RFC4303].
PseudoWires appear to be a strong candidate as the deterministic networking data plane solution alternative for the DetNet Service layer. The strong points are the technical maturity and the extensive control plane for OAM. This holds specifically for MPLS-based PSN.
Extensions are required to realize the packet replication and duplicate detection features of the deterministic networking data plane.
MPLS-Based Ethernet VPN (EVPN), in the form documented in [RFC7432] and [RFC7209], is an increasingly popular approach to delivering MPLS-based Ethernet services and is designed to be the successor to Virtual Private LAN Service (VPLS), [RFC4664].
EVPN provides client adaptation and reuses the MPLS data plane discussed above in Section 5.2.2. While not required, the PW Control Word is also used. EVPN control is via BGP, [RFC7432], and may use TE-LSPs, e.g., controlled via [RFC3209] for MPLS transport. Additional EVPN related RFCs and in progress drafts are being developed by the BGP Enabled Services Working Group.
EVPN is the emerging successor to VPLS. EVPN is standardized, implemented and deployed. It makes use of the mature MPLS data plane. While offering a mature and very comprehensive set of features, certain DetNet required features are not fully/directly supported and additional standardization in these areas are needed. Examples include: mapping CoS and QoS; use of labels per DetNet flow, and hitless 1+1 protection.
Fields of headers belonging to higher OSI layers can be used to implement functionality that is not provided e.g., by the IPv6 or IPv4 header fields. However, this approach cannot be always applied, e.g., due to encryption. Furthermore, even if this approach is applicable, it requires deep packet inspection from the routers and switches. There are implementation dependent limits how far into the packet the lookup can be done efficiently in the fast path. When encryption is not used, a safe bet is generally between 128 and 256 octets for the maximum lookup depth. Various higher layer protocols can be applied. Some examples are provided here for the sequence numbering feature (Section 4.3).
The TCP header includes a sequence number parameter, which can be applied to detect and eliminate duplicate packets if DetNet service protection is used. As the TCP header is right after the IP header, it does not require very deep packet inspection; the 4-byte sequence number is conveyed by bits 32 through 63 of the TCP header. In addition to sequencing, the TCP header also contain source and destination port information that can be used for assisting the flow identification.
Real-time Transport Protocol (RTP) [RFC3550] is often used to deliver time critical traffic in IP networks. RTP is typically carried on top of UDP/IP. However, as noted earlier in Section 5.2.3 PseudoWires also have a well-defined way of embedding and transposting RTP header as part of its payload encapsulation headers/sub-layer. RTP is also augmented by its own control protocol RTCP, which monitors of the data delivery and provides minimal control and identification functionality. RTCP packets do not carry "media payload". Although both RTP and RTCP are typically used with UDP/IP transport they are designed to be independent of the underlying transport and network layers.
The RTP header includes a 2-byte sequence number, which can be used to detect and eliminate duplicate packets if DetNet service protection is used. The sequence number is conveyed by bits 16 through 31 of the RTP header. In addition to the sequence number the RTP header has also timestamp field (bits 32 through 63) that can be useful for time synchronization purposes. Furthermore, the RTP header has also one or more synchronization sources (bits starting from 64) that can potentially be useful for flow identification purposes.
RTP appears to be a good candidate as the deterministic networking data plane solution alternative for the DetNet Service layer. The strong points are the technical maturity and the fact it was designed for transporting time-sensitive payload from the beginning. RTP is specifically well suited to be used with (UDP)/IP transport.
Extensions may be required to realize the packet replication and duplicate detection features of the deterministic networking data plane. However, there is already precedence of similar solutions that could potentially be leveraged [ST20227][RFC7198].
The following table summarizes the criteria (Section 4) used for the evaluation of data plane options.
Applicability per Alternative
| Item # | Meaning |
|---|---|
| #1 | Encapsulation and overhead |
| #2 | Flow identification |
| #3 | Packet sequencing and duplicate elimination |
| #4 | Explicit routes |
| #5 | Flow duplication and merging |
| #6 | Operations, Administration and Maintenance |
| #8 | Class and quality of service capabilities |
| #9 | Packet traceability |
| #10 | Technical maturity |
There is no single technology that could meet all the criteria on its own. Distinguishing the DetNet Service and the DetNet Transport, as explained in (Section 3), allows a number of combinations, which can meet most of the criteria. There is no room here to evaluate all possible combinations. Therefore, only some combinations are highlighted here, which are selected based on the number of criteria that are met and the maturity of the technology (#10).
The following table summarizes the evaluation of the data plane options that can be used for the DetNet Transport Layer against the evaluation criteria. Each value in the table is from the corresponding section.
Applicability per Transport Alternative
| Solution | #1 | #2 | #4 | #5 | #6 | #8 | #9 | #10 |
|---|---|---|---|---|---|---|---|---|
| IPv6 | M | W | W | W | M | W | W | M/W |
| IPv4 | M | W | W | W/N | M | M/W | W | M/W |
| MPLS | M | M | M | M/W | M | M/W | M | M |
| BIER | M | M | M | M/W | M/W | M/W | M | W |
| BIER-TE | W/M | N | N | M/W | W | N | W | W |
Summarizing Transport capabilities
The following table summarizes the evaluation of the data plane options that can be used for the DetNet Service Layer against the criteria evaluation criteria. Each value in the table is from the corresponding section.
Applicability per Service Alternative
| Solution | #1 | #2 | #3 | #5 | #6 | #8 | #10 |
|---|---|---|---|---|---|---|---|
| GRE | M | W | M/W | W/N | M | W | M |
| PWE3 | M | M | M | W | M/W | M/W | M |
| EVPN | M | W | M | M/W | M/W | M/W | M |
| RTP | M | M | M | M/W | M | M/W | M |
Summarizing Service capabilities
PseudoWire (Section 5.2.3) is a technology that is mature and meets most of the criteria for the DetNet Service layer as shown in the table above. From upper layer protocols PWs or RTP can be a candidate for non-MPLS PSNs. The identified work for PWs is to figure out how to implement duplicate detection for these protocols (e.g., based on [RFC3985]). In a case of RTP there is precedence of implementing packet duplication and duplicate elimination [ST20227][RFC7198].
PWs can be carried over MPLS or IP. MPLS is the most common technology that is used as PSN for PseudoWires; furthermore, MPLS is a mature technology and meets most DetNet Transport layer criteria. IPv[46] can be also used as PSN and both are mature technologies, although both generally only support CoS (DiffServ) in deployed networks. RTP is independent of the underlying transport technology and network. However, it is well suited for UDP/IP transport or embedded as a part of the PseudoWire timing sub-layer.
This document does not add any new security considerations beyond what the referenced technologies already have.
This document has no IANA considerations.
The author(s) ACK and NACK.
The following people were part of the DetNet Data Plane Design Team:
Substantial contributions were received from:
The DetNet chairs serving during the DetNet Data Plane Design Team: