Internet DRAFT - draft-thubert-tsvwg-detnet-transport
draft-thubert-tsvwg-detnet-transport
TSVWG P. Thubert, Ed.
Internet-Draft Cisco
Intended status: Standards Track October 30, 2017
Expires: May 3, 2018
A Transport Layer for Deterministic Networks
draft-thubert-tsvwg-detnet-transport-01
Abstract
This document specifies the behavior of a Transport Layer operating
over a Deterministic Network and implementing a DetNet Service Layer
and a Northbound side of the DetNet User-to-Network Interface.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 3, 2018.
Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Thubert Expires May 3, 2018 [Page 1]
Internet-Draft DetTrans October 2017
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. On Deterministic Networking . . . . . . . . . . . . . . . . . 5
3.1. Applications and Requirements . . . . . . . . . . . . . . 5
3.2. The DetNet User-to-Network Interface (UNI) . . . . . . . 7
3.3. The DetNet Stack . . . . . . . . . . . . . . . . . . . . 8
3.4. The DetNet Service Model . . . . . . . . . . . . . . . . 8
4. DetTrans Operations . . . . . . . . . . . . . . . . . . . . . 9
4.1. DetTrans Overview . . . . . . . . . . . . . . . . . . . . 9
4.2. Application Requirements . . . . . . . . . . . . . . . . 9
4.2.1. Packet Normalization . . . . . . . . . . . . . . . . 9
4.2.2. Packet Streaming . . . . . . . . . . . . . . . . . . 10
4.3. Deterministic Flow Services . . . . . . . . . . . . . . . 10
4.3.1. Deterministic Flows . . . . . . . . . . . . . . . . . 10
4.3.2. Deterministic Flow Encapsulation and Stitching . . . 11
4.3.2.1. Flow Stitching . . . . . . . . . . . . . . . . . 11
4.3.2.2. Load Sharing . . . . . . . . . . . . . . . . . . 11
4.3.2.3. Flow Aggregation . . . . . . . . . . . . . . . . 12
4.3.3. Deterministic Service Protection . . . . . . . . . . 13
4.3.3.1. PRE vs. 1+1 Redundancy . . . . . . . . . . . . . 13
4.3.3.2. Network Coding . . . . . . . . . . . . . . . . . 13
4.3.3.3. Multipath DetTrans Services . . . . . . . . . . . 13
5. The DetNet-UNI . . . . . . . . . . . . . . . . . . . . . . . 14
5.1. Local Loop Flow Control . . . . . . . . . . . . . . . . . 16
5.1.1. Dichotomy of a DetNet End System . . . . . . . . . . 16
5.1.2. Local Loop Location . . . . . . . . . . . . . . . . . 17
5.1.3. Network Pull vs. Rate Based Flow Control . . . . . . 18
5.2. DetNet-UNI Protocol Exchanges . . . . . . . . . . . . . . 18
5.2.1. the "More" Message . . . . . . . . . . . . . . . . . 18
5.2.2. the "Time-Correction" Message . . . . . . . . . . . . 19
5.2.3. Loss of a Control Message . . . . . . . . . . . . . . 19
6. Security Considerations . . . . . . . . . . . . . . . . . . . 20
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20
8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 20
9. Informative References . . . . . . . . . . . . . . . . . . . 20
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 22
1. Introduction
Over last twenty years, voice, data and video networks have converged
to digital over IP. Mail delivery has become quasi-immediate and
volumes have multiplied; long distance voice is now mostly free and
the videophone is finally a reality; TV is available on-demand and
games became interactive and massively multi-player. The convergence
of highly heterogeneous networks over IP resulted in significant
drops in price for the end-user while adding new distinct value to
Thubert Expires May 3, 2018 [Page 2]
Internet-Draft DetTrans October 2017
the related services. Yet, and even though similar benefits can be
envisioned when converging new applications over the Internet, there
are still many disjoint branches in the networking family tree, many
use-cases where mission-specific applications continue to utilize
dedicated point-to-point analog and digital technologies for their
operations.
Forty years ago, Control Information was first encoded as an analog
modulation of current (typically 4 to 20 mA) that can be carried
virtually instantly and with no loss over a distance. Then came
digitization, which enabled to multiplex data with the control signal
and manage the devices, but at the same time introduced latency to
industrial processes, the necessary delay to encode a series of bits
on a link and transport them along, which in turn may limit the
amount of transported information. The need to save cable and
simplify wiring lead to the Time Division Multiplexing (TDM) of
signals from multiple devices over shared digital buses, each signal
being granted access to the medium at a fixed period for a fixed
duration; with TDM, came more latency, waiting for the next reserved
access time. Statistical multiplexing, with Ethernet and IP, was
then introduced to achieve higher speeds at lower cost, and with it
came jitter and congestion loss.
A number of Operational Technology (OT) applications are now
migrating to Ethernet and IP, but that comes at the expense of
additional latency for the flows, to compensate for the degradation
of the transport discussed above. This also comes at the expense of
additional complexity in particular, applications may need to
transport a sense of time, provide some Forward Error Correction
(FEC) and include a jitter absorption buffer. for that reason, many
applications were never ported and OT networks are still largely
operated on point-to-point serial links and TDM buses.
A sense of what Deterministic Networking is has emerged as the
capability to make the Application simple again and enable a larger
migration of existing applicationsby absorbing the complexity lower
in the stack, at the Transport, Network and Link layers. A
Deterministic Network should be capable to emulate point-to-point
wires over a packet network, sharing the network resources between
deterministic and non-deterministic flows in such a fashion that
there can no observable influence whatsoever on a deterministic flow
from any other flow, regardless of the load of the network.
The generalization of the needs for more deterministic networks have
led to the IEEE 802.1 AVB Task Group becoming the Time-Sensitive
Networking (TSN) [IEEE802.1TSNTG] Task Group (TG), with a much-
expanded constituency from the industrial and vehicular markets. In
order to address the problem at the network layer, the DetNet Working
Thubert Expires May 3, 2018 [Page 3]
Internet-Draft DetTrans October 2017
Group was formed to specify the signaling elements to be used to
establish a path and the tagging elements to be used identify the
flows that are to be forwarded along that path.
The "Deterministic Networking Use Cases" [I-D.ietf-detnet-use-cases]
indicates that beyond the classical case of industrial automation and
control systems (IACS), there are in fact multiple industries with
strong and yet relatively similar needs for deterministic network
services such as latency guarantees and ultra-low packet loss. The
"Deterministic Networking Problem Statement"
[I-D.ietf-detnet-problem-statement] documents the specific
requirements for the use of routed networks to support these
applications and the "Deterministic Networking Architecture"
[I-D.ietf-detnet-architecture] introduces the model that must be
proposed to integrate determinism in IT technology.
A DetNet network will guarantee a bounded latency and a very low
packet loss as long as the incoming flows respect a certain Service
Level Agreement (SLA), as typically expressed in the form of a
maximum packet size, a time window of observation and a maximum
number of packets per time window.
Outside the scope of DetNet, the IETF will also need to specify the
necessary protocols, or protocol additions, based on relevant IETF
technologies, to enable end-to-end deterministic flows. One critical
element is the Deterministic Transport Layer (DetTrans) that adapts
the flows coming from the Application Layer to the SLA of the DetNet
Network and provide end-to-end guarantees such as loss, latency and
timeliness.
The DetTrans Layer should in particular ensure that:
o the Deterministic Network setup matches the needs of the
Application
o the Application flows are presented to the Deterministic Network
in accordance to the SLA regardless of the way the data is passed
from the application
o the use of the network is optimized so as to ensure that every
byte from the application can effectively be transported
o the application flow is delivered reliably and with a bounded
latency to the other Transport End Point, which may imply a FEC
technique such as Network Coding, Packet Replication and
Elimination (PRE), or basic 1+1 redundancy.
Thubert Expires May 3, 2018 [Page 4]
Internet-Draft DetTrans October 2017
o the full of the application flow is served, which may require the
use of multiple reservations in parallel, and the reordering of
the flows
On the one hand, the Deterministic Network will typically guarantee a
constant rate, so the classical Transport feature of flow control
will not be needed in a Deterministic Transport. On the other hand,
the Application and Transport layers may not reside in the same
device as the DetNet Router and/or the IEEE Std. 802.1 TSN Bridge
that acts as ingress point to the Deterministic Network. It results
that a minimum reliability and flow control must take place over the
Local Loop between these devices to ensure that the Deterministic
Network is kept optimally fed, meaning that packets are received just
in time for their scheduled transmission opportunities.
2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
3. On Deterministic Networking
3.1. Applications and Requirements
The Internet is not the only digital network that has grown
dramatically over the last 30-40 years. Video and audio
entertainment, and control systems for machinery, manufacturing
processes, and vehicles are also ubiquitous, and are now based almost
entirely on digital technologies. Over the past 10 years, engineers
in these fields have come to realize that significant advantages in
both cost and in the ability to accelerate growth can be obtained by
basing all of these disparate digital technologies on packet
networks.
The goals of Deterministic Networking are to enable the migration of
applications that use special-purpose fieldbus technologies (HDMI,
CANbus, ProfiBus, etc... even RS-232!) to packet technologies in
general, and the Internet Protocol in particular, and to support both
these new applications, and existing packet network applications,
over the same physical network.
Considerable experience ([ODVA]/[EIP], [AVnu], [Profinet],[HART],
[IEC62439], [ISA100.11a] and [WirelessHART], etc...) has shown that
these applications need a some or all of a suite of deterministic
features.
That suite of deterministic features includes:
Thubert Expires May 3, 2018 [Page 5]
Internet-Draft DetTrans October 2017
1. Time synchronization of all Host and network nodes (Routers and/
or Bridges), accurate to something between 10 nanoseconds and 10
microseconds, depending on the application.
2. Support for critical packet flows that:
* Can be unicast or multicast;
* Need absolute guarantees of minimum and maximum latency end-
to-end across the network; sometimes a tight jitter is
required as well;
* Need a packet loss ratio beyond the classical range for a
particular medium, in the range of 10^-9 to 10^-12, or better,
on Ethernet, and in the order of 10^-5 in Wireless Sensor Mesh
Networks;
* Can, in total, absorb more than half of the network's
available bandwidth (that is, massive over-provisioning is
ruled out as a solution);
* Cannot suffer throttling, flow control, or any other network-
imposed latency, for flows that can be meaningfully
characterized either by a fixed, repeating transmission
schedule, or by a maximum bandwidth and packet size;
3. Multiple methods to schedule, shape, limit, and otherwise control
the transmission of critical packets at each hop through the
network data plane;
4. Robust defenses against misbehaving Hosts, Routers, or Bridges,
both in the data and control planes, with guarantees that a
critical flow within its guaranteed resources cannot be affected
by other flows whatever the pressures on the network;
5. One or more methods to reserve resources in Bridges and Routers
to carry these flows.
Robustness is a common need for networking protocols, but plays a
more important part in real-time control networks, where expensive
equipment, and even lives, can be lost due to misbehaving equipment.
Reserving resources before packet transmission is the one fundamental
shift in the behavior of network applications that is impossible to
avoid. In the first place, a network cannot deliver finite latency
and practically zero packet loss to an arbitrarily high offered load.
Secondly, achieving practically zero packet loss for un-throttled
(though bandwidth limited) flows means that Bridges and Routers have
to dedicate buffer resources to specific flows or to classes of
Thubert Expires May 3, 2018 [Page 6]
Internet-Draft DetTrans October 2017
flows. The requirements of each reservation have to be translated
into the parameters that control each Host's, Bridge's, and Router's
queuing, shaping, and scheduling functions and delivered to the
Hosts, Bridges, and Routers.
3.2. The DetNet User-to-Network Interface (UNI)
The "Deterministic Networking Architecture"
[I-D.ietf-detnet-architecture] presents the end-to-end networking
model and the DetNet services; in particular, it depicts the DetNet
User-to-Network Interfaces (DetNet-UNIs) ("U" in Figure 1) between
the Edge nodes (PE) of the Deterministic Network and the End Systems.
These UNIs are assumed to be packet-based reference points and
provide connectivity over the packet network. The Architecture also
mentions internal reference points between the Central Processing
Unit (CPU) and the Network Interface Card (NIC) in the End System.
The DetNet-UNIs provide congestion protection services and belong to
the DetNet Transport Layer.
DetNet DetNet
End System End System
_ _
/ \ +----DetNet-UNI (U) / \
/App\ | /App\
/-----\ | /-----\
| NIC | v ________ | NIC |
+--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+
| / \__/ \ | |
| / +----+ +----+ \_____ | |
| / | | | | \_______ | |
+------U PE +----+ P +----+ \ _ v |
| | | | | | | ___/ \ |
| +--+-+ +----+ | +----+ | / \_ |
\ | | | | | / \ |
\ | +----+ +--+-+ +--+PE |-------- U------+
\ | | | | | | | | | \_ _/
\ +---+ P +----+ P +--+ +----+ | \____/
\___ | | | | /
\ +----+__ +----+ DetNet-1 DetNet-2
| \_____/ \___________/ |
| |
| | End-to-End-Service | | | |
<---------------------------------------------------------------->
| | DetNet-Service | | | |
| <--------------------------------------------------> |
| | | | | |
Figure 1: DetNet Service Reference Model (multi-domain)
Thubert Expires May 3, 2018 [Page 7]
Internet-Draft DetTrans October 2017
A specific hardware is necessary for the time-sensitive functions of
synchronization, shaping and scheduling. This hardware may or may
not be fully available on a NIC inside the Host system. This
specification makes a distinction between a fully DetNet-Capable NIC,
and a DetNet-Aware NIC that participates to the DetNet-UNI, but is
not synchronized and scheduled with the Deterministic Network.
3.3. The DetNet Stack
The "Deterministic Networking Architecture"
[I-D.ietf-detnet-architecture] presents a conceptual DetNet data
plane layering model. The protocol stack includes a Service Layer
and a Transport Layer and is illustrated in Figure 2.
| packets going | ^ packets coming ^
v down the stack v | up the stack |
+------------------------+ +------------------------+
| Source | | Destination |
+------------------------+ +------------------------+
| Service layer | | Service layer |
| ------------------- | | -------------------- |
| Adaptation | | Presentation |
| Packet | Network | | Duplicate | Network |
| sequencing | Coding | | elimination | Coding |
| Flow | Packet | | Flow | Packet |
| duplication | encoding | | merging | decoding |
+-------------+----------+ +-------------+----------+
| Transport layer | | Transport layer |
| ------------------- | | ------------------- |
| Encapsulation | | Decapsulation |
| Congestion protection | | Congestion protection |
+------------------------+ +------------------------+
| Lower layers | | Lower layers |
+------------------------+ +------------------------+
DetNet End System DetNet End System
v ^
\________DetNet Transport______/
Figure 2: DetNet-Capable End-System Protocol Stack
3.4. The DetNet Service Model
The "DetNet Service Model" [I-D.varga-detnet-service-model] provides
more details on the distribution of DetNet awareness and services.
Thubert Expires May 3, 2018 [Page 8]
Internet-Draft DetTrans October 2017
4. DetTrans Operations
4.1. DetTrans Overview
The DetNet Service Layer mostly operates between the end-points,
though it is possible that some operations such as Packet Replication
and Elimination are also performed in selected intermediate nodes.
The DetNet Transport Layer represents the methods that ensure that a
packet is deterministically forwarded hop-by-hop from a Detnet Relay
to the next. The term "Transport" in the DetNet terminology must not
be confused with the function described in this document. This
document defines Detrans as a Layer-4 operation and an IETF Transport
Layer; DetTrans provides DetNet End-To-End Services for its
Applications, as well as intermediate services in selected points.
Following the DetNet Architecture, DetTrans mostly corresponds to the
DetNet Service Layer and its interface with the Detnet Transport
Layer for congestion protection services through the DetNet_UNI, as
well as for encapsulation and decapsulation services. Compared to a
traditional IETF Transport Layer, DetTrans performs similar operation
of end-to-end reliability, flow control and multipath load sharing,
but differs on how those functionalities are achieved.
Architectural variations are also introduced, for instance:
o Multipath operations are not necessarily end-to-end and a DetTrans
function may be present inside the network to relay between N
parallel paths and M parallel path, and or perform reliability
functionality such as Packet Replication and Elimination.
o The flow control is only needed between the DetTrans Layer and the
first Deterministic Transit or Relay Node, for instance a DetNet
Router or an IEEE Std. 802.1 TSN Bridge. From that point on, the
flow is strictly controlled by the DetNet operation.
Architecturally speaking, the flow control does not belong to the
DetNet Service Layer but to the DetNet Transport Layer, which
means that this specification also defines a sublayer from the
DetNet Transport Layer for DetNet-UNI operations.
4.2. Application Requirements
4.2.1. Packet Normalization
A typical SLA for DetNet must be simple, for instance a maximum
packet size, and a maximum number of packets per window of time.
Smaller packets will mean wasted bandwidth, and excess packets within
a time window will be destroyed by the ingress shaping at the first
DetNet Bridge or Router.
Thubert Expires May 3, 2018 [Page 9]
Internet-Draft DetTrans October 2017
The way the application layer feed the DetTrans layer may not
necessarily match the SLA with the Deterministic Network and in order
to provide the expected service, the DetTrans layer must pack the
data in packets that are as close to the maximum packet size as
possible, and yet make them available for transmission before
scheduled time.
4.2.2. Packet Streaming
The DetTrans Layer operates on its own sense of time which may be
loosely connected to the shared sense of time in the Deterministic
Network.
The DetTrans layer must shape its transmissions so as to ensure that
packets are delivered just in time to be injected along schedule in
the Deterministic Network.
4.3. Deterministic Flow Services
4.3.1. Deterministic Flows
Deterministic forwarding can only apply on flows with well-defined
characteristics such as periodicity and burstiness. Before a path
can be established to serve them, the expression of those
characteristics, and how the network can serve them, for instance in
shaping and forwarding operations, must be specified.
At the time of this writing, the distinction between application
layer flows and lower layer flows is not clearly stated in the
"Deterministic Networking Architecture"
[I-D.ietf-detnet-architecture]. For the purpose of this document, we
use the term Determistic End-to-End Service Flow (DEESF), or DetTrans
Flow, to refer to an end-to-end application flow, and the term
Determistic Service Flow (DSF), or DetNet Flow, to refer to a lower
layer deterministic transport. This is illustrated in Figure 3.
+---------------+ +---------------+
| Application | | Application |
+---------------+ +---------------+
| Service >--------- DetNet End-to-End Service Flow ---------> Service |
+---------------+ +--------+ +--------+ +---------------+
| Transport | | | | | | Transport |
+---------------+ |>-- DetNet Service -->| +---------------+
| Lower Layers | | Flow | | Lower Layers |
+---------------+ +--------+ +--------+ +---------------+
Figure 3: DetTrans vs. DetNet Flows
Thubert Expires May 3, 2018 [Page 10]
Internet-Draft DetTrans October 2017
An application flow is established end-to-end between the DetTrans
layers and uses one or more lower-layer deterministic flows either in
parallel or in serial modes.
At Application and DetTrans Layers, the characteristics of a flow
relate to aggregate properties such as throughput, loss, and traffic
shape, and the Traffic Specification (TSPec) is expressed as a
Constant Bit Rate (CBR) or a Variable Bit Rate (VBR), burstiness
(e.g. video I-Frames), reliability (e.g. five nines), worst case
latency, amount of data to transfer, and expected duration of the
flow.
At the DetNet Transport Layer (between Relays), metrics are very
different, and relate to immediate actions on a packet as opposed to
general characteristics of a flow. DetNet Transport Layer
characteristics include time sync precision, time interval between
packets, packet size, jitter, and number of packets per window of
time. This is how the network SLA is defined, but this is not the
native terms for the application and a complex mapping must ensure
that the path that is setup and the DetNet Transport Layer
effectively matches the requirements from the DetNet Services Layer
and above.
4.3.2. Deterministic Flow Encapsulation and Stitching
4.3.2.1. Flow Stitching
The DetNet encapsulation and decapsulation of one-in-one, one-in-many
and many-in-one Deterministic flows belongs to the DetNet Transport
Layer. Direct one-in-one flow stitching also belongs to the DetNet
Transport Layer. This happens when a deterministic flow can be
directly bridged into another, resource-to-resource, without the need
of an upper layer adaptation such as service protection from the
Service Layer. A Detnet End-to-End Service flow may be stitched into
one Detnet Service flow, or encapsulated in one or multiple Detnet
Service flows.
4.3.2.2. Load Sharing
Load Sharing refers to the encapsulation of a DetNet Flow in more
than one DetNet flows, for instance using multiple small and more
manageable DetNet Service Flows in parallel to carry a large
Determistic End-to-End Service Flow, in order to avoid the need to
periodically defragment the network. Packets are sequenced at the
DetTrans Layer and distributed over the DetNet Transports paths in
accordance to their relative capacities. In case of inconsistent
jitter and Latency characteristics, packets may need to be reordered
at the receiving DetTrans Layer based on the DSF Sequence.
Thubert Expires May 3, 2018 [Page 11]
Internet-Draft DetTrans October 2017
+---------------+ +--------+ +--------+ +---------------+
| Application | |>-- DetNet Service -->| | Application |
+---------v-----+ +-------------Flow-------------+ +-----^---------+
| | | | | | | | | | | |
| | | | +--------+ +--------+ | | | |
| | | | | | | |
| | >------- DetNet End-to-End Service Flow -------> | |
| | | | | | | |
| | +-----+ +--------+ +--------+ +-----+ | |
| +---+ | |>-- DetNet Service -->| | +---+ |
| +-------------------Flow-------------------+ |
| DetTrans | | | | | | | | DetTrans |
+---------------+ +--------+ +--------+ +---------------+
DetNet Deterministic DetNet
End System Routers and Bridges End System
Figure 4: Load Sharing
In order to achieve this function, a Load Distribution function is
required at the source and a Re-Ordering Function is required at the
destination DetTrans End Point.
4.3.2.3. Flow Aggregation
Flow Aggregation refers to the encapsulation of more than one DetNet
flows in one DetNet Flow, for instance using one large and long-lived
DetNet Service Flow from a third party provider to carry multiple
more dynamic Deterministic End-to-End Service Flows across domains.
Packets are sequenced at the DetTrans Layer and distributed over the
DetNet Transports paths in accordance to their relative capacities.
In case of inconsistent jitter and Latency characteristics, packets
may need to be reordered at the receiving DetTrans Layer based on the
DSF Sequence.
+---------------+ >-- DetNet Service --> +---------------+
| Application | Flow | Application |
+-----v---v-----+ +--------+ +--------+ +-----^---^-----+
| | | | | | | | | | | |
| | +--------------------------------------------------+ | |
| | >------- DetNet End-to-End Service Flow ---------> | |
| +----------------------------------------------------------+ |
| >----------- DetNet End-to-End Service Flow -------------> |
| DetTrans | | | | | | | DetTrans |
+---------------+ +--------+ +--------+ +---------------+
DetNet Deterministic DetNet
End System Routers and Bridges End System
Figure 5: Flow Aggregation
Thubert Expires May 3, 2018 [Page 12]
Internet-Draft DetTrans October 2017
In order to achieve this function, a multiplexing function is
required at the source and a demultiplexing function is required at
the destination DetTrans End Point.
4.3.3. Deterministic Service Protection
4.3.3.1. PRE vs. 1+1 Redundancy
The DetNet Flows may also be used for Packet Replication and
Elimination, in which case an elimination function is required at the
DetTrans Termination.
1+1 Redundancy refers to injecting identical copies of a packet at
the ingress of two non-congruent paths, and eliminating the excess
copy when both are received at the egress of the paths. Packet
Replication and Elimination extends the concept by enabling more than
2 paths, and allowing non-end-to-end redundant paths with
intermediate Replication and Elimination points.
4.3.3.2. Network Coding
Redundancy and Load Sharing may be combined with the use of Network
Coding whereby a coded packet may carry redundancy information for
previous data packet and cover the loss of one, in which case the
recovery function is required at the other DetTrans End Point.
Network Coding provides a Forward Error Correction between multiple
packets or multiple fragments of a packet. It may be used at the DSF
layer to enable an efficient combination of redundancy and load
sharing.
4.3.3.3. Multipath DetTrans Services
A DetTrans Flow may leverage multiple DetNet Flows in parallel in
order to achieve its requirements in terms of reliability and
Aggregate throughput. The "Deterministic Networking Architecture"
[I-D.ietf-detnet-architecture] clearly states that the capability of
Replication and Elimination is not limited to the DetNet End Systems.
DetNet Relay Nodes that operate DetTrans but then relay the packets
are needed when the DetTrans operations are not end-to-end.
It may be that the DetTrans flow may need to traverse different
domains where those Services are operated differently, e.g.
controlled by different controllers or leveraging different
technologies. It may also be that the bandwidth that is required is
only available one segemnt at a time, and that for each segment, a
different number of DetNet flows must be setup to transport the full
amount of the DetTrans flow.
Thubert Expires May 3, 2018 [Page 13]
Internet-Draft DetTrans October 2017
Figure 6 illustrates an example of the latter case, whereby The
DetTrans Flow is distributed over two DetNet Flows, maybe operating
PRE, then over three DetNet Flows, for instance operating Network
Coding between them but using a smaller banswidth for each flow, and
then two DetNet Flows again.
DetTrans is needed at the interconnection points to adapt the flows,
recover losses and reinject the appropriate rates in the next
segment.
+---+-----------+ +--------+ +--------+
| C |Application| +------------------------------+
| P +-----------+ | |>- DetNet Transport ->| | +---------------+
| U | DetTrans | | +--------+ +--------+ | | DetTrans / |
+---+-----------+ | | | ------- |
| N | Packet +-----+ +--------+ +--------+ | | / DetTrans |
| I | Queues -+ | |>- DetNet Transport ->| +--------+---+---+ |
| C | +---------------------------------------------+---+---+ |
+---+-----------+ +--------+ +--------+ +----|---|---|--+
| | |
DetNet | | |
Transport | | |
| | |
+---+-----------+ +--------+ +--------+ | | |
| C |Application| +------------------------------+ | | |
| P +-----------+ | |<- DetNet Transport -<| | +----|---|---|--+
| U | DetTrans | | +--------+ +--------+ +--------+---+---+ |
+---+-----------+ | +--------+---+---+ |
| N | +-----+ +--------+ +--------+ | | \ DetTrans |
| I | <-+ | |<- DetNet Transport -<| | | ------- |
| C | +------------------------------------+ | DetTrans \ |
+---+-----------+ +--------+ +--------+ +---------------+
DetNet-Aware Deterministic Deterministic
Host Systems Transit Nodes Relay Nodes
Figure 6: Intermediate Systems
5. The DetNet-UNI
Figure 7 illustrates a simple example of classical networked devices
implementing the DetNet architecture. In that example, applications
reside on Host systems and run on main CPUs; DetTrans is collocated
with its applications and provides them with a Deterministic Service
through DetTrans APIs. NICs provides the connectivity to the
Deterministic Routers or Bridges acting at DetNet Edge and Relay
Nodes - say as an example that they are IEEE Std. 802.1 TSN Bridges.
Thubert Expires May 3, 2018 [Page 14]
Internet-Draft DetTrans October 2017
Deterministic
Host System Routers and Bridges Host System
+---+-----------+ | +--------+ +--------+ | +---+-----------+
| C |Application| |---| |----| |---| | C |Application|
| P +-----------+ | |<- DetNet Transport ->|-------+ P +-----------+
| U | DetTrans | | +--------+ +--------+ | | U | DetTrans |
+---+-----------+ <------------ DetTrans ------------> +---+-----------+
| N | Lower | | +--------+ +--------+ | | N | Lower |
| I | Layers |---| |<- DetNet Transport ->| |---| I | Layers |
| C | (queues) | |---| |----| |---| | C | |
+---+-----------+ | +--------+ +--------+ | +---+-----------+
<-UNI-> <-- DetNet Services -> <-UNI->
<------------- DetNet End-To-End Services ------------------>
Figure 7: Example Physical Network
The DetTrans Layer aggregates the data coming from the application up
to a maximum frame size that is part of the SLA with the DetNet
Transport. Packets thus formed can be distributed over any of
multiple DetNet Transport sessions that are defined to accept that
packet size. Packets formed at the DetTrans Layer are queued and
ready to be delivered through the DetNet-UNI either with a Rate-Based
or a Network-Pull mechanism.
If the NIC is DetNet-Aware then the queue can be offboarded to the
NIC and it can be drained with a time gate (Rate-Base) or a message-
driven gate (Network-Pull). Else, the queue is handled by the CPU
and hopefully it can be drained within an interrupt, either for a
timer (Rate-Base) or for a message (Network-Pull).
The DetNet-UNI protocol enables the DetNet transport ingress point to
control when the DetTrans Layer transmits its Data packets. It may
happen that the DetTrans Layer has not formed a fully-sized packet
when time comes for sending it, in which case the packet will be sent
with a size below the maximum.
The DetNet UNI uses ICMPv6 to carry its protocol elements. Data
Packets across the UNI are encapsulated in order to carry DetNet-UNI
control information to identify the reason of a loss or a delay, and
determine the action to be taken in case of a packet lost or delayed
over the interface.
Thubert Expires May 3, 2018 [Page 15]
Internet-Draft DetTrans October 2017
5.1. Local Loop Flow Control
5.1.1. Dichotomy of a DetNet End System
The logical DetNet End System depicted in Figure 2 comprises several
elements which may implemented in one or separate physical Systems.
The example dichotomy in Figure 3 segregates ingress shaping and
DetNet Relay functions, which are performed by IEEE Std. 802.1 TSN
Bridges, from a DetNet-Aware Host.
Hosts and Edge Bridges are connected over Ethernet and together they
form a DetNet End System. As it goes, this example introduces a
further dichotomy within the Host, between the CPU and the NIC,
across a local bus such as PCI, as illustrated in Figure 8.
+-------------+ +-------+ +-----------+
| Application | | MAC | | Ingress |
+-------------+ <--PCI--> +-------+ <---- Ethernet ----> | Shaping |
| DetTrans | | PHY | | and Relay |
+-------------+ +-------+ +-----------+
CPU NIC IEEE Std.
<------------- DetNet-UNI -------------> 802.1
<----------- Host System -------------> TSN Bridge
<------------------------- DetNet End System ------------------------>
Figure 8: Chained Functions
The NICs in the Host System may not participate to the network time
Synchronization and may not be aware of the DetNet protocols running
between the Deterministic Routers and Bridges, and the associated
scheduling rules. In that situation, the DetNet-UNI operates on a
Local Loop to ensure that a packet that leaves the Transport reaches
the Router or Bridge just in time for injection into the
Deterministic data plane and to provide a flow control that avoids
congestion loss at the interface.
It is also possible that the NIC participates to the Deterministic
Network but still has asynchronous communication with DetTrans Layer
running on the the CPU. Either way, a flow control over a local loop
must be implemented to drain the queues from the DetTrans layer and
feed the network just in time for the deterministic transmission.
Depending on the level of support by the NIC, the loop may be placed
on a different interface but remains functionally the same.
Thubert Expires May 3, 2018 [Page 16]
Internet-Draft DetTrans October 2017
5.1.2. Local Loop Location
If the NIC is not aware at all of DetNet, then it is a plain pipe for
the Deterministic Traffic. The Local Loop operates between the Edge
TSN Bridge and the CPU as illustrated in Figure 9.
+---+-----------+ +---+-------+ +-----------+
| C |Application| | N | MAC | | Ingress |
| P +-----------+ <--PCI--> | I +-------+ <-- Ethernet --> | Shaping |
| U |DetTrans | | C | PHY | (Bridged | / Transit |
+---+-----------+ +---+-------+ or P2P) +-----------+
<------------- Local Loop -------------> Edge 802.1
<----------- Host System -------------> TSN Bridge
Figure 9: DetNet Unaware NIC
If the NIC is fully DetNet-Capable and participates to the
deterministic Network including time synchronization and scheduling,
then the local loop operates between the CPU and the NIC as
illustrated in Figure 10.
+---+-----------+ +---+-----------+ +----------+
| C |Application| | N | Ingress | | DetNet |
| P +-----------+ <--PCI--> | I + Shaping | <--DetNet--> | Transit |
| U |DetTrans | | C | / Transit | Transport | Node |
+---+-----------+ +---+-----------+ +----------+
<-Local-
-Loop ->
<-------------- Host System --------------> TSN Bridge
Figure 10: DetNet Capable NIC
If the NIC is DetNet-Aware and does not participates to the
deterministic Network including time synchronization and scheduling,
then there are two local loops, one that operates between the CPU and
the NIC and one that operates between the NIC and the Edge TSN Bridge
as illustrated in Figure 11.
+---+-----------+ +---+-------+ +-----------+
| C |Application| | N | MAC | Ethernet | Ingress |
| P +-----------+ <--PCI--> | I +-------+ <-- (Bridged --> | Shaping |
| U |DetTrans | | C | PHY | or P2P) | / Transit |
+---+-----------+ <-Local- +---+-------+ +-----------+
-Loop -> <-- Local Loop --> Edge
<----------- Host System -------------> DetNet Transit
Figure 11: DetNet Capable NIC
Thubert Expires May 3, 2018 [Page 17]
Internet-Draft DetTrans October 2017
5.1.3. Network Pull vs. Rate Based Flow Control
The flow control at the DetNet-UNI can take any of two forms:
Network Pull In that Model, the DetNet Edge node drains the DetTrans
queue by sending a DetNet-UNI "More" command some estimated amount
of time ahead of the scheduled time of transmission for each
packet; in case of load sharing, multiple DetNet Edge nodes may
drain a queue at their own rates; in case of a high jitter on the
UNI Local Loop (e.g. there is a non-deterministic Bridge in
between, or the NIC is not DetNet-Aware and the flows suffer from
the more erratic response time of the CPU), the DetNet Edge node
may need to pull a window of packets to maintain its own
transmission queues fed at all times
Rate Based In that model, the NIC is aware of the rate of the
deterministic transmission and is drained by its internal timers.
Since the NIC is not synchronized with the Deterministic Network,
the Bridge uses a DetNet-UNI "Time-Correction" command
asynchronously to move forward or backward the next timeout of the
NIC for that flow, in order to keep the Rate-Based transmission by
the NIC in rough alignment with the scheduled transmission over
the DetNet network.
if the NIC is DetNet-Aware, it is expected that it maintains the
DetTrans queues in order to provide a deterministic response to the
DetNet-UNI, and in that case another control loop between the NIC and
the CPU is needed to ensure that the queue in the NIC is always fed
in time by the DetTrans Layer; this second loop may be of a different
nature than the DetNet-UNI one and may for instance be operated
within an interrupt to limit the asynchronism related to message
queueing.
5.2. DetNet-UNI Protocol Exchanges
5.2.1. the "More" Message
The "More" message enables a DetNet Transport Edge to pull one packet
from the DetTrans Layer in Network-Pull mode. The message is
associated with a future transmission opportunity for a packet. The
"More" messages are indexed by a wrapping More Sequence Counter
(MSC). The Transport Edge also maintains wrapping counters of
Successful Packet Transmissions (SPT) and Missed Transmit
Opportunities (MTO). The current value of these counters is placed
in the "More "message.
Upon reception of a "More" message, the DetTrans Layer, or the NIC on
behalf of the DetTrans Layer, sends the next available packet for
Thubert Expires May 3, 2018 [Page 18]
Internet-Draft DetTrans October 2017
that session. The packet is encapsulated and the encapsulation
indicates the MSC. This enables the DetNet Transport Edge to
correlate the packet with the transmission opportunity and drop
packets that are overly delayed.
5.2.2. the "Time-Correction" Message
The "Time-Correction" message enables a DetNet Transport Edge to
adjust the timer associated to the DetNet-UNI session in Rate-Based
mode. In that mode, the DetTrans Layer sends a packet and restarts a
timer at a period that corresponds to the transmission opportunity of
the DetNet Transport Edge. If the clock in the CPU drifts, the
DetNet Transport Edge will start receiving packets increasingly ahead
of expected time or behind expected time. It is expected that the
DetNet Transport Edge is protected against a minimum drift by a guard
time, but if the drift becomes too important, then the DetNet
Transport Edge issues a "Time-Correction" message indicating a number
of time units (e.g. microseconds) by which the DetTrans Layer should
advance or delay is next time out.
5.2.3. Loss of a Control Message
The loss of a packet beween the DetTrans Layer and the DetNet
Transport Edge will correspond to a missed Transmission Opportunity
but this does not mean that packets are piling up at the DetTrans
Layer. OTOH, if a "More" message is lost, then one packet will not
be dequeued and the Detrans queue might grow, increasingly augmenting
the latency. It is thus important to differentiate these situations,
and in the latter case, discard an extraneous packet to restore the
normal level in the DetTrans queue for that session.
In order to do so, the DetTrans Layer maintains the record of the
Number of Packets Sent (NPS), that it compares with the variation of
the MTO and SPT counters in the "More" message. Upon a "More"
message, the DetTrans Layer computes the variation of NPS
(dNPS=NPS2-NPS1) and the variation of SPT (dSPT=SPT2-SPT1) since the
previous "More" Message . dNPS is typically 1 if the transport
always has data to send. Packets in flight when the "More" message
is sent are considered lost since they will be received after their
scheduled transmission opportunity, so the Number of Packets Losses
(NPL) is (NPL=dNPS-dSPT). The DetTrans Layer also computes the
variation of MTO since the previous "More" Message (dMTO=MTO2-MTO1).
Since a packet loss implies a missed transmission opportunity, there
cannot be more packets losses than missed opportunities, so we have
dMTO>=NPL. dMTO-NPL represents the number of missed opportunities
that are not due to a packet lost or late arrival, thus this is the
sub-count of MTOs due to the loss of a "More" message.
Thubert Expires May 3, 2018 [Page 19]
Internet-Draft DetTrans October 2017
For each loss of a "More" message, a packet in the DetTrans queue
should be discarded. In order to simplify that operation and
outboard it to the NIC, the Transports marks some packets as "Discard
Eligible" (DE). A packet can be marked DE if there are enough
alternate transmissions of non-DE packets to recover this. For
instance, in case of Packet Replication and Elimination only one copy
can be marked DE, and the marking should alternate between the
sessions to cover a loss on either one rapidly.
6. Security Considerations
The generic threats against Deterministic Networking are discussed in
the "Deterministic Networking Security" [I-D.ietf-detnet-security]
document.
Security in the context of Deterministic Networking has an added
dimension; the time of delivery of a packet can be just as important
as the contents of the packet, itself. A man-in-the-middle attack,
for example, can impose, and then systematically adjust, additional
delays into a link, and thus disrupt or subvert a real-time
application without having to crack any encryption methods employed.
See [RFC7384] for an exploration of this issue in a related context.
Packet Replication and Elimination of done right can prevent a man-
in-the-middle attack on one leg to actually impact the flow beyond
the loss of an individual packet for lack of redundancy. This
specification expects that PRE is performed at the transport level
and provides specific means to protect one leg against misuse of the
other.
7. IANA Considerations
This document does not require an action from IANA.
8. Acknowledgments
The authors wish to thank Patrick Wetterwald, Leon Turkevitch, Balazs
Varga and Janos Farkas for their various contributions to this work.
Special thanks to Norm Finn for being a (if not the) major thought
leader to the whole deterministic effort, and for some text that is
inlined here from other IETF documents, for the convenience of the
reader.
9. Informative References
Thubert Expires May 3, 2018 [Page 20]
Internet-Draft DetTrans October 2017
[AVnu] http://www.avnu.org/, "The AVnu Alliance tests and
certifies devices for interoperability, providing a simple
and reliable networking solution for AV network
implementation based on the IEEE Audio Video Bridging
(AVB) and Time-Sensitive Networking (TSN) standards.".
[EIP] http://www.odva.org/, "EtherNet/IP provides users with the
network tools to deploy standard Ethernet technology (IEEE
802.3 combined with the TCP/IP Suite) for industrial
automation applications while enabling Internet and
enterprise connectivity data anytime, anywhere.",
<http://www.odva.org/Portals/0/Library/
Publications_Numbered/
PUB00138R3_CIP_Adv_Tech_Series_EtherNetIP.pdf>.
[HART] www.hartcomm.org, "Highway Addressable Remote Transducer,
a group of specifications for industrial process and
control devices administered by the HART Foundation".
[I-D.ietf-detnet-architecture]
Finn, N., Thubert, P., Varga, B., and J. Farkas,
"Deterministic Networking Architecture", draft-ietf-
detnet-architecture-03 (work in progress), August 2017.
[I-D.ietf-detnet-problem-statement]
Finn, N. and P. Thubert, "Deterministic Networking Problem
Statement", draft-ietf-detnet-problem-statement-02 (work
in progress), September 2017.
[I-D.ietf-detnet-security]
Mizrahi, T., Grossman, E., Hacker, A., Das, S., Dowdell,
J., Austad, H., Stanton, K., and N. Finn, "Deterministic
Networking (DetNet) Security Considerations", draft-ietf-
detnet-security-00 (work in progress), October 2017.
[I-D.ietf-detnet-use-cases]
Grossman, E., Gunther, C., Thubert, P., Wetterwald, P.,
Raymond, J., Korhonen, J., Kaneko, Y., Das, S., Zha, Y.,
Varga, B., Farkas, J., Goetz, F., Schmitt, J., Vilajosana,
X., Mahmoodi, T., Spirou, S., Vizarreta, P., Huang, D.,
Geng, X., Dujovne, D., and M. Seewald, "Deterministic
Networking Use Cases", draft-ietf-detnet-use-cases-13
(work in progress), September 2017.
[I-D.varga-detnet-service-model]
Varga, B. and J. Farkas, "DetNet Service Model", draft-
varga-detnet-service-model-02 (work in progress), May
2017.
Thubert Expires May 3, 2018 [Page 21]
Internet-Draft DetTrans October 2017
[IEC62439]
IEC, "Industrial communication networks - High
availability automation networks - Part 3: Parallel
Redundancy Protocol (PRP) and High-availability Seamless
Redundancy (HSR) - IEC62439-3", 2012,
<https://webstore.iec.ch/publication/7018>.
[IEEE802.1TSNTG]
IEEE Standards Association, "IEEE 802.1 Time-Sensitive
Networks Task Group", 2013,
<http://www.ieee802.org/1/pages/avBridges.html>.
[ISA100.11a]
ISA/IEC, "ISA100.11a, Wireless Systems for Automation,
also IEC 62734", 2011, < http://www.isa100wci.org/en-
US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch-
WEB-ETSI.aspx>.
[ODVA] http://www.odva.org/, "The organization that supports
network technologies built on the Common Industrial
Protocol (CIP) including EtherNet/IP.".
[Profinet]
http://us.profinet.com/technology/profinet/, "PROFINET is
a standard for industrial networking in automation.",
<http://us.profinet.com/technology/profinet/>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in
Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384,
October 2014, <https://www.rfc-editor.org/info/rfc7384>.
[WirelessHART]
www.hartcomm.org, "Industrial Communication Networks -
Wireless Communication Network and Communication Profiles
- WirelessHART - IEC 62591", 2010.
Author's Address
Thubert Expires May 3, 2018 [Page 22]
Internet-Draft DetTrans October 2017
Pascal Thubert (editor)
Cisco Systems, Inc
Building D (Regus) 45 Allee des Ormes
MOUGINS - Sophia Antipolis
FRANCE
Phone: +33 4 97 23 26 34
Email: pthubert@cisco.com
Thubert Expires May 3, 2018 [Page 23]
