Deterministic Networking Architecture
draft-finn-detnet-architecture-06

Abstract

Deterministic Networking (DetNet) provides a capability to carry specified unicast or multicast data flows for real-time applications with extremely low data loss rates and bounded latency. Techniques used include: 1) reserving data plane resources for individual (or aggregated) DetNet flows in some or all of the intermediate nodes (e.g. bridges or routers) along the path of the flow; 2) providing fixed paths for DetNet flows that do not rapidly change with the network topology; and 3) sequentializing, replicating, tracing and eliminating duplicate packets at various points to ensure delivery of each packet over at least one path. The capabilities can be managed by configuration, or by manual or automatic network management.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on January 9, 2017.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 2. Terminology
• 2.1. Terms used in this document
• 2.2. IEEE 802 TSN to DetNet dictionary
• 3. Providing the DetNet Quality of Service
• 3.1. Zero Congestion Loss
• 3.2. Explicit routes
• 3.3. Jitter Reduction
• 3.4. Packet Replication and Elimination
• 4. DetNet Architecture
• 4.1. Traffic Engineering for DetNet
• 4.1.1. The Application Plane
• 4.1.2. The Controller Plane
• 4.1.3. The Network Plane
• 4.2. DetNet flows
• 4.2.1. Source guarantees
• 4.2.2. Incomplete Networks
• 4.3. Queuing, Shaping, Scheduling, and Preemption
• 4.4. Coexistence with normal traffic
• 4.5. Fault Mitigation
• 4.6. Representative Protocol Stack Model
• 4.7. Exporting flow identification
• 4.8. Advertising resources, capabilities and adjacencies
• 4.9. Provisioning model
• 4.9.1. Centralized Path Computation and Installation
• 4.9.2. Distributed Path Setup
• 4.10. Scaling to larger networks
• 4.11. Connected islands vs. networks
• 5. Compatibility with Layer-2
• 6. Open Questions
• 6.1. DetNet flow identification and sequencing
• 6.2. Flat vs. hierarchical control
• 6.3. Peer-to-peer reservation protocol
• 6.4. Wireless media interactions
• 6.5. Packet encoding for loss prevention
• 7. Security Considerations
• 8. Privacy Considerations
• 9. IANA Considerations
• 10. Acknowledgements
• 11. Access to IEEE 802.1 documents
• 12. Informative References
• Authors' Addresses

1. Introduction

Deterministic Networking (DetNet) is a service that can be offered by a network to data flows (DetNet flows) that that are limited, at their source, to a maximum data rate specified by that source. DetNet provides these flows extremely low packet loss rates and assured maximum end-to-end delivery latency. This is accomplished by dedicating network resources such as link bandwidth and buffer space to DetNet flows and/or classes of DetNet flows, and by replicating packets along multiple paths. Unused reserved resources are available to non-DetNet packets.

The Deterministic Networking Problem Statement [I-D.ietf-detnet-problem-statement] introduces Deterministic Networking, and Deterministic Networking Use Cases [I-D.ietf-detnet-use-cases] summarizes the need for it.

A goal of DetNet is a converged network in all respects. That is, the presence of DetNet flows does not preclude non-DetNet flows, and the benefits offered DetNet flows should not, except in extreme cases, prevent existing QoS mechanisms from operating in a normal fashion, subject to the bandwidth required for the DetNet flows. A single source-destination pair can trade both DetNet and non-DetNet flows. End systems and applications need not instantiate special interfaces for DetNet flows. Networks are not restricted to certain topologies; connectivity is not restricted. Any application that generates a data flow that can be usefully characterized as having a maximum bandwidth should be able to take advantage of DetNet, as long as the necessary resources can be reserved. Reservations can be made by the application itself, via network management, by an applications controller, or by other means.

Many applications of interest to Deterministic Networking require the ability to synchronize the clocks in end systems to a sub-microsecond accuracy. Some of the queue control techniques defined in Section 4.3 also require time synchronization among relay and transit nodes. The means used to achieve time synchronization are not addressed in this document. DetNet should accommodate various synchronization techniques and profiles that are defined elsewhere to solve exchange time in different market segments.

The present document is an individual contribution, but it is intended by the authors for adoption by the DetNet working group.

2. Terminology

2.1. Terms used in this document

The following special terms are used in this document in order to avoid the assumption that a given element in the architecture does or does not have Internet Protocol stack, functions as a router, bridge, firewall, or otherwise plays a particular role at Layer-2 or higher.

destination

An end system capable of receiving a DetNet flow.

DetNet domain

The portion of a network that is DetNet aware. It includes end systems and other DetNet nodes.

DetNet flow

A DetNet flow is a sequence of packets to which the DetNet service is to be applied. It can be limited by the source in its maximum packet size and transmission rate, and can thus be provided congestion-free delivery by the network.

DetNet compound flow and DetNet member flow

A DetNet compound flow is a DetNet flow that has been separated into multiple duplicate DetNet member flows, which are eventually merged back into a single DetNet compound flow, at the DetNet transport layer. "Compound" and "member" are strictly relative to each other, not absolutes; a DetNet compound flow comprising multiple DetNet member flows can, in turn, be a member of a higher-order compound.

DetNet intermediate node

A DetNet relay node or transit node.

DetNet relay edge node

An instance of a DetNet relay node that includes a service layer proxy function for DetNet loss prevention (e.g. packet sequencing and/or elimination) for one or more end systems, analogous to a Label Edge Router (LER).

end system

Commonly called a "host" or "node" in IETF documents, and an "end station" is IEEE 802 documents. End systems of interest to this document are either sources or destinations of DetNet flows. And end system may or may not be DetNet transport layer aware or DetNet service layer aware.

link

A connection between two DetNet nodes. It may be composed of a physical link or a sub-network technology that can provide appropriate traffic delivery for DetNet flows.

DetNet node

A DetNet aware end system, transit node, or relay node. "DetNet" may be omitted in some text.

Detnet relay node

A DetNet service layer function that interconnects different DetNet transport layer protocols or networks (instances) to perform packet replication and elimination (Section 3.4. A DetNet relay node typically incorporates DetNet transport layer functions as well, in which case it is collocated with a transit node, such as a bridge, a router, a Label Switch Router (LSR), a firewall, or any other system that participates in the DetNet service layer.

reservation

A trail of configuration between source to destination(s) through transit nodes and subnets associated with a DetNet flow, required to deliver the benefits of DetNet.

DetNet service layer

The layer at which loss prevention services such as packet sequencing and the elimination part of replication and elimination (Section 3.4) are performed.

source

An end system capable of sourcing a DetNet flow.

DetNet transit node

A node operating at the DetNet transport layer, that utilizes link layer and/or network layer switching across multiple links and/or sub-networks to provide paths for DetNet service layer functions. An MPLS LSR is an example of a DetNet transit node.

DetNet transport layer

The layer that splits and merges Detnet flows for packet replication and elimination (Section 3.4).

2.2. IEEE 802 TSN to DetNet dictionary

This section also serves as a dictionary for translating from the terms used by the IEEE 802 Time-Sensitive Networking (TSN) Task Group to those of the DetNet WG.

Listener

The IEEE 802 term for a destination of a DetNet flow.

relay system

The IEEE 802 term for a DetNet intermediate node.

Stream

The IEEE 802 term for a DetNet flow.

Talker

The IEEE 802 term for the source of a DetNet flow.

3. Providing the DetNet Quality of Service

The DetNet Quality of Service can be expressed in terms of:

• Minimum and maximum end-to-end latency from source to destination; timely delivery and jitter avoidance derive from these constraints
• Probability of loss of a packet, under various assumptions as to the operational states of the nodes and links. A derived property is whether it is acceptable to deliver a duplicate packet, which is an inherent risk in highly reliable and/or broadcast transmissions

It is a distinction of DetNet that it is concerned solely with worst-case values for the end-to-end latency. Average, mean, or typical values are of no interest, because they do not affect the ability of a real-time system to perform its tasks. In general, a trivial priority-based queuing scheme will give better average latency to a data flow than DetNet, but of course, the worst-case latency can be essentially unbounded.

Three techniques are used by DetNet to provide these qualities of service:

• Bandwidth reservation and enforcement (Section 3.1).
• Explicit routes (Section 3.2).
• A DetNet loss protection mechanism.

The DetNet techniques are meant to address both of the DetNet QoS requirements (latency and packet loss). Given that DetNet nodes have a finite amount of buffer space, zero congestion loss necessarily results in a maximum end-to-end latency. It also addresses the largest contribution to packet loss, which is buffer congestion.

After congestion, the most important contributions to packet loss are typically from random media errors and equipment failures. Additional mechanisms, such as encoding schemes and/or data replication techniques are needed. The mechanisms employed are constrained by the requirement to meet the users' latency requirements. Packet replication and elimination (Section 3.4) is one possible mechanism to provide DetNet loss protection.

These three techniques can be applied independently, giving eight possible combinations, including none (no DetNet), although some combinations are of wider utility than others. This separation keeps the protocol stack coherent and maximizes interoperability with existing and developing standards in this (IETF) and other Standards Development Organizations. Some examples of typical expected combinations:

• Explicit routes (a) plus packet replication (b) are exactly the techniques employed by [HSR-PRP]. Explicit routes are achieved by limiting the physical topology of the network, and the sequentialization, replication, and duplicate elimination are facilitated by packet tags added at the front or the end of Ethernet frames.
• Zero congestion loss (a) alone is is offered by IEEE 802.1 Audio Video bridging [IEEE802.1BA-2011]. As long as the network suffers no failures, zero congestion loss can be achieved through the use of a reservation protocol (MSRP), shapers in every bridge, and a bit of network calculus.
• Using all three together gives maximum protection.

There are, of course, simpler methods available (and employed, today) to achieve levels of latency and packet loss that are satisfactory for many applications. Prioritization and over-provisioning is one such technique. However, these methods generally work best in the absence of any significant amount of non-critical traffic in the network (if, indeed, such traffic is supported at all), or work only if the critical traffic constitutes only a small portion of the network's theoretical capacity, or work only if all systems are functioning properly, or in the absence of actions by end systems that disrupt the network's operations.

There are any number of methods in use, defined, or in progress for accomplishing each of the above techniques. It is expected that this DetNet Architecture will assist various vendors, users, and/or "vertical" Standards Development Organizations (dedicated to a single industry) to make selections among the available means of implementing DetNet networks.

3.1. Zero Congestion Loss

The primary means by which DetNet achieves its QoS assurances is to completely eliminate congestion at an output port as a cause of packet loss. Given that a DetNet flow cannot be throttled, this can be achieved only by the provision of sufficient buffer storage at each hop through the network to ensure that no packets are dropped due to a lack of buffer storage.

Ensuring adequate buffering requires, in turn, that the source, and every intermediate node along the path to the destination (or nearly every node -- see Section 4.2.2) be careful to regulate its output to not exceed the data rate for any DetNet flow, except for brief periods when making up for interfering traffic. Any packet sent ahead of its time potentially adds to the number of buffers required by the next hop, and may thus exceed the resources allocated for a particular DetNet flow.

The low-level mechanisms described in Section 4.3 provide the necessary regulation of transmissions by an end system or intermediate node to ensure zero congestion loss. The reservation of the bandwidth and buffers for a DetNet flow requires the provisioning described in Section 4.9. A DetNet node may have other resources requiring allocation and/or scheduling, that might otherwise be over-subscribed and trigger the rejection of a reservation.

3.2. Explicit routes

In networks controlled by typical peer-to-peer protocols such as IEEE 802.1 ISIS bridged networks or IETF OSPF routed networks, a network topology event in one part of the network can impact, at least briefly, the delivery of data in parts of the network remote from the failure or recovery event. Thus, even redundant paths through a network, if controlled by the typical peer-to-peer protocols, do not eliminate the chances of brief losses of contact.

Many real-time networks rely on physical rings or chains of two-port devices, with a relatively simple ring control protocol. This supports redundant paths with a minimum of wiring. As an additional benefit, ring topologies can often utilize different topology management protocols than those used for a mesh network, with a consequent reduction in the response time to topology changes. Of course, this comes at some cost in terms of increased hop count, and thus latency, for the typical path.

In order to get the advantages of low hop count and still ensure against even very brief losses of connectivity, DetNet employs explicit routes, where the path taken by a given DetNet flow does not change, at least immediately, and likely not at all, in response to network topology events. When combined with a loss prevention mechanism such as packet replication and elimination (Section 3.4), this results in a high likelihood of continuous connectivity. Explicit routes are commonly used in MPLS TE LSPs.

3.3. Jitter Reduction

A core objective of DetNet is to enable the convergence of Non-IP networks onto a common network infrastructure. This requires the accurate emulation of currently deployed mission-specific networks, which typically rely on point-to-point analog (e.g. 4-20mA modulation) and serial-digital cables (or buses) for highly reliable, synchronized and jitter-free communications. While the latency of analog transmissions is basically the speed of light, legacy serial links are usually slow (in the order of Kbps) compared to, say, GigE, and some latency is usually acceptable. What is not acceptable is the introduction of excessive jitter, which may, for instance, affect the stability of control systems.

Applications that are designed to operate on serial links usually do not provide services to recover the jitter, because jitter simply does not exists there. Streams of information are expected to be delivered in-order and the precise time of reception influences the processes. In order to converge such existing applications, there is a desire to emulate all properties of the serial cable, such as clock transportation, perfect flow isolation and fixed latency. While minimal jitter (in the form of specifying minimum, as well as maximum, end-to-end latency) is supported by DetNet, there are practical limitations on packet-based networks in this regard. In general, users are encouraged to use, instead of, "do this when you get the packet," a combination of:

• Sub-microsecond time synchronization among all source and destination end systems, and
• Time-of-execution fields in the application packets.

3.4. Packet Replication and Elimination

After congestion loss has been eliminated, the most important causes of packet loss are random media and/or memory faults, and equipment failures. Both causes of packet loss can be greatly reduced by apreading the data in a packet over multiple transmissions. One such method is described in this section, which sends the same packets over multiple paths. Other methods, such as ones that use encoding methods to combine the information in multiple packets, may also be applicable. See Section 6.5.

Packet replication and elimination, also known as seamless redundancy [HSR-PRP], or 1+1 hitless protection, is a function of the DetNet service layer. It involves three capabilities:

• Replicating these packets into multiple DetNet member flows and, typically, sending them along at least two different paths to the destination(s), e.g. over the explicit routes of Section 3.2.
• Providing sequencing information, once, at or near the source, to the packets of a DetNet compound flow. This may be done by adding a sequence number or time stamp as part of DetNet, or may be inherent in the packet, e.g. in a transport protocol, or associated to other physical properties such as the precise time (and radio channel) of reception of the packet.
• Eliminating duplicated packets. This may be done at any step along the path to save network resources further down, in particular if multiple Replication points exist. But the most common case is to perform this operation at the very edge of the DetNet network, preferably in or near the receiver.

This function is a "hitless" version of, e.g., the 1+1 linear protection in [RFC6372]. That is, instead of switching from one flow to the other when a failure of a flow is detected, DetNet combines both flows, and performs a packet-by-packet selection of which to discard, based on sequence number.

In the simplest case, this amounts to replicating each packet in a source that has two interfaces, and conveying them through the network, along separate paths, to the similarly dual-homed destinations, that discard the extras. This ensures that one path (with zero congestion loss) remains, even if some intermediate node fails. The sequence numbers can also be used for loss detection and for re-ordering.

Alternatively, Detnet relay nodes in the network can provide replication and elimination facilities at various points in the network, so that multiple failures can be accommodated.

This is shown in the following figure, where the two relay nodes each replicate (R) the DetNet flow on input, sending the DetNet member flows to both the other relay node and to the end system, and eliminate duplicates (E) on the output interface to the right-hand end system. Any one link in the network can fail, and the Detnet compound flow can still get through. Furthermore, two links can fail, as long as they are in different segments of the network.

             > > > > > > > > > relay > > > > > > > >
            > /------------+ R node E +------------\ >
           > /                  v + ^               \ >
   end    R +                   v | ^                + E end
   system   +                   v | ^                +   system
           > \                  v + ^               / >
            > \------------+ R relay E +-----------/ >
             > > > > > > > > >  node > > > > > > > >

    End                                                     End
    System                                               System

   -+-+-+-+-+-+-+ Northbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

             PCE         PCE              PCE              PCE

   -+-+-+-+-+-+-+ Southbound -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

           intermediate  intermed.   intermed.  intermed.
               Node       Node       Node       Node
    NIC                                                     NIC
           intermediate  intermed.   intermed.  intermed.
               Node       Node       Node       Node

  |  packets going  |        ^  packets coming   ^
  v down the stack  v        |   up the stack    |
+----------------------+   +-----------------------+
|        source        |   |      destination      |
+----------------------+   +-----------------------+
|    Service layer     |   |     Service layer     |
|  Packet sequencing   |   | Duplicate elimination |
+----------------------+   +-----------------------+
|   Transport layer    |   |    Transport layer    |
|  flow duplication    |   |      flow merging     |
+----------------------+   +-----------------------+
|    Subnet services   |   |    Subnet services    |
+----------------------+   +-----------------------+
|     Lower layers     |   |     Lower layers      |
+----------------------+   +-----------------------+
           v                           ^
            \_________________________/