| TSVWG | P. Thubert, Ed. |
| Internet-Draft | Cisco |
| Intended status: Standards Track | October 30, 2017 |
| Expires: May 3, 2018 |
A Transport Layer for Deterministic Networks
draft-thubert-tsvwg-detnet-transport-01
This document specifies the behavior of a Transport Layer operating over a Deterministic Network and implementing a DetNet Service Layer and a Northbound side of the DetNet User-to-Network Interface.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 3, 2018.
Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Over last twenty years, voice, data and video networks have converged to digital over IP. Mail delivery has become quasi-immediate and volumes have multiplied; long distance voice is now mostly free and the videophone is finally a reality; TV is available on-demand and games became interactive and massively multi-player. The convergence of highly heterogeneous networks over IP resulted in significant drops in price for the end-user while adding new distinct value to the related services. Yet, and even though similar benefits can be envisioned when converging new applications over the Internet, there are still many disjoint branches in the networking family tree, many use-cases where mission-specific applications continue to utilize dedicated point-to-point analog and digital technologies for their operations.
Forty years ago, Control Information was first encoded as an analog modulation of current (typically 4 to 20 mA) that can be carried virtually instantly and with no loss over a distance. Then came digitization, which enabled to multiplex data with the control signal and manage the devices, but at the same time introduced latency to industrial processes, the necessary delay to encode a series of bits on a link and transport them along, which in turn may limit the amount of transported information. The need to save cable and simplify wiring lead to the Time Division Multiplexing (TDM) of signals from multiple devices over shared digital buses, each signal being granted access to the medium at a fixed period for a fixed duration; with TDM, came more latency, waiting for the next reserved access time. Statistical multiplexing, with Ethernet and IP, was then introduced to achieve higher speeds at lower cost, and with it came jitter and congestion loss.
A number of Operational Technology (OT) applications are now migrating to Ethernet and IP, but that comes at the expense of additional latency for the flows, to compensate for the degradation of the transport discussed above. This also comes at the expense of additional complexity in particular, applications may need to transport a sense of time, provide some Forward Error Correction (FEC) and include a jitter absorption buffer. for that reason, many applications were never ported and OT networks are still largely operated on point-to-point serial links and TDM buses.
A sense of what Deterministic Networking is has emerged as the capability to make the Application simple again and enable a larger migration of existing applicationsby absorbing the complexity lower in the stack, at the Transport, Network and Link layers. A Deterministic Network should be capable to emulate point-to-point wires over a packet network, sharing the network resources between deterministic and non-deterministic flows in such a fashion that there can no observable influence whatsoever on a deterministic flow from any other flow, regardless of the load of the network.
The generalization of the needs for more deterministic networks have led to the IEEE 802.1 AVB Task Group becoming the Time-Sensitive Networking (TSN) Task Group (TG), with a much-expanded constituency from the industrial and vehicular markets. In order to address the problem at the network layer, the DetNet Working Group was formed to specify the signaling elements to be used to establish a path and the tagging elements to be used identify the flows that are to be forwarded along that path.
The "Deterministic Networking Use Cases" indicates that beyond the classical case of industrial automation and control systems (IACS), there are in fact multiple industries with strong and yet relatively similar needs for deterministic network services such as latency guarantees and ultra-low packet loss. The "Deterministic Networking Problem Statement" documents the specific requirements for the use of routed networks to support these applications and the "Deterministic Networking Architecture" introduces the model that must be proposed to integrate determinism in IT technology.
A DetNet network will guarantee a bounded latency and a very low packet loss as long as the incoming flows respect a certain Service Level Agreement (SLA), as typically expressed in the form of a maximum packet size, a time window of observation and a maximum number of packets per time window.
Outside the scope of DetNet, the IETF will also need to specify the necessary protocols, or protocol additions, based on relevant IETF technologies, to enable end-to-end deterministic flows. One critical element is the Deterministic Transport Layer (DetTrans) that adapts the flows coming from the Application Layer to the SLA of the DetNet Network and provide end-to-end guarantees such as loss, latency and timeliness.
The DetTrans Layer should in particular ensure that:
On the one hand, the Deterministic Network will typically guarantee a constant rate, so the classical Transport feature of flow control will not be needed in a Deterministic Transport. On the other hand, the Application and Transport layers may not reside in the same device as the DetNet Router and/or the IEEE Std. 802.1 TSN Bridge that acts as ingress point to the Deterministic Network. It results that a minimum reliability and flow control must take place over the Local Loop between these devices to ensure that the Deterministic Network is kept optimally fed, meaning that packets are received just in time for their scheduled transmission opportunities.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
The Internet is not the only digital network that has grown dramatically over the last 30-40 years. Video and audio entertainment, and control systems for machinery, manufacturing processes, and vehicles are also ubiquitous, and are now based almost entirely on digital technologies. Over the past 10 years, engineers in these fields have come to realize that significant advantages in both cost and in the ability to accelerate growth can be obtained by basing all of these disparate digital technologies on packet networks.
The goals of Deterministic Networking are to enable the migration of applications that use special-purpose fieldbus technologies (HDMI, CANbus, ProfiBus, etc... even RS-232!) to packet technologies in general, and the Internet Protocol in particular, and to support both these new applications, and existing packet network applications, over the same physical network.
Considerable experience ([ODVA]/[EIP], [AVnu], [Profinet],[HART], [IEC62439], [ISA100.11a] and [WirelessHART], etc...) has shown that these applications need a some or all of a suite of deterministic features.
That suite of deterministic features includes:
Robustness is a common need for networking protocols, but plays a more important part in real-time control networks, where expensive equipment, and even lives, can be lost due to misbehaving equipment. Reserving resources before packet transmission is the one fundamental shift in the behavior of network applications that is impossible to avoid. In the first place, a network cannot deliver finite latency and practically zero packet loss to an arbitrarily high offered load. Secondly, achieving practically zero packet loss for un-throttled (though bandwidth limited) flows means that Bridges and Routers have to dedicate buffer resources to specific flows or to classes of flows. The requirements of each reservation have to be translated into the parameters that control each Host's, Bridge's, and Router's queuing, shaping, and scheduling functions and delivered to the Hosts, Bridges, and Routers.
The "Deterministic Networking Architecture" presents the end-to-end networking model and the DetNet services; in particular, it depicts the DetNet User-to-Network Interfaces (DetNet-UNIs) ("U" in Figure 1) between the Edge nodes (PE) of the Deterministic Network and the End Systems. These UNIs are assumed to be packet-based reference points and provide connectivity over the packet network. The Architecture also mentions internal reference points between the Central Processing Unit (CPU) and the Network Interface Card (NIC) in the End System. The DetNet-UNIs provide congestion protection services and belong to the DetNet Transport Layer.
DetNet DetNet
End System End System
_ _
/ \ +----DetNet-UNI (U) / \
/App\ | /App\
/-----\ | /-----\
| NIC | v ________ | NIC |
+--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+
| / \__/ \ | |
| / +----+ +----+ \_____ | |
| / | | | | \_______ | |
+------U PE +----+ P +----+ \ _ v |
| | | | | | | ___/ \ |
| +--+-+ +----+ | +----+ | / \_ |
\ | | | | | / \ |
\ | +----+ +--+-+ +--+PE |-------- U------+
\ | | | | | | | | | \_ _/
\ +---+ P +----+ P +--+ +----+ | \____/
\___ | | | | /
\ +----+__ +----+ DetNet-1 DetNet-2
| \_____/ \___________/ |
| |
| | End-to-End-Service | | | |
<---------------------------------------------------------------->
| | DetNet-Service | | | |
| <--------------------------------------------------> |
| | | | | |
Figure 1: DetNet Service Reference Model (multi-domain)
A specific hardware is necessary for the time-sensitive functions of synchronization, shaping and scheduling. This hardware may or may not be fully available on a NIC inside the Host system. This specification makes a distinction between a fully DetNet-Capable NIC, and a DetNet-Aware NIC that participates to the DetNet-UNI, but is not synchronized and scheduled with the Deterministic Network.
The "Deterministic Networking Architecture" presents a conceptual DetNet data plane layering model. The protocol stack includes a Service Layer and a Transport Layer and is illustrated in Figure 2.
| packets going | ^ packets coming ^
v down the stack v | up the stack |
+------------------------+ +------------------------+
| Source | | Destination |
+------------------------+ +------------------------+
| Service layer | | Service layer |
| ------------------- | | -------------------- |
| Adaptation | | Presentation |
| Packet | Network | | Duplicate | Network |
| sequencing | Coding | | elimination | Coding |
| Flow | Packet | | Flow | Packet |
| duplication | encoding | | merging | decoding |
+-------------+----------+ +-------------+----------+
| Transport layer | | Transport layer |
| ------------------- | | ------------------- |
| Encapsulation | | Decapsulation |
| Congestion protection | | Congestion protection |
+------------------------+ +------------------------+
| Lower layers | | Lower layers |
+------------------------+ +------------------------+
DetNet End System DetNet End System
v ^
\________DetNet Transport______/
Figure 2: DetNet-Capable End-System Protocol Stack
The "DetNet Service Model" provides more details on the distribution of DetNet awareness and services.
The DetNet Service Layer mostly operates between the end-points, though it is possible that some operations such as Packet Replication and Elimination are also performed in selected intermediate nodes. The DetNet Transport Layer represents the methods that ensure that a packet is deterministically forwarded hop-by-hop from a Detnet Relay to the next. The term "Transport" in the DetNet terminology must not be confused with the function described in this document. This document defines Detrans as a Layer-4 operation and an IETF Transport Layer; DetTrans provides DetNet End-To-End Services for its Applications, as well as intermediate services in selected points.
Following the DetNet Architecture, DetTrans mostly corresponds to the DetNet Service Layer and its interface with the Detnet Transport Layer for congestion protection services through the DetNet_UNI, as well as for encapsulation and decapsulation services. Compared to a traditional IETF Transport Layer, DetTrans performs similar operation of end-to-end reliability, flow control and multipath load sharing, but differs on how those functionalities are achieved.
Architectural variations are also introduced, for instance:
A typical SLA for DetNet must be simple, for instance a maximum packet size, and a maximum number of packets per window of time. Smaller packets will mean wasted bandwidth, and excess packets within a time window will be destroyed by the ingress shaping at the first DetNet Bridge or Router.
The way the application layer feed the DetTrans layer may not necessarily match the SLA with the Deterministic Network and in order to provide the expected service, the DetTrans layer must pack the data in packets that are as close to the maximum packet size as possible, and yet make them available for transmission before scheduled time.
The DetTrans Layer operates on its own sense of time which may be loosely connected to the shared sense of time in the Deterministic Network.
The DetTrans layer must shape its transmissions so as to ensure that packets are delivered just in time to be injected along schedule in the Deterministic Network.
Deterministic forwarding can only apply on flows with well-defined characteristics such as periodicity and burstiness. Before a path can be established to serve them, the expression of those characteristics, and how the network can serve them, for instance in shaping and forwarding operations, must be specified.
At the time of this writing, the distinction between application layer flows and lower layer flows is not clearly stated in the "Deterministic Networking Architecture". For the purpose of this document, we use the term Determistic End-to-End Service Flow (DEESF), or DetTrans Flow, to refer to an end-to-end application flow, and the term Determistic Service Flow (DSF), or DetNet Flow, to refer to a lower layer deterministic transport. This is illustrated in Figure 3.
+---------------+ +---------------+ | Application | | Application | +---------------+ +---------------+ | Service >--------- DetNet End-to-End Service Flow ---------> Service | +---------------+ +--------+ +--------+ +---------------+ | Transport | | | | | | Transport | +---------------+ |>-- DetNet Service -->| +---------------+ | Lower Layers | | Flow | | Lower Layers | +---------------+ +--------+ +--------+ +---------------+
Figure 3: DetTrans vs. DetNet Flows
An application flow is established end-to-end between the DetTrans layers and uses one or more lower-layer deterministic flows either in parallel or in serial modes.
At Application and DetTrans Layers, the characteristics of a flow relate to aggregate properties such as throughput, loss, and traffic shape, and the Traffic Specification (TSPec) is expressed as a Constant Bit Rate (CBR) or a Variable Bit Rate (VBR), burstiness (e.g. video I-Frames), reliability (e.g. five nines), worst case latency, amount of data to transfer, and expected duration of the flow.
At the DetNet Transport Layer (between Relays), metrics are very different, and relate to immediate actions on a packet as opposed to general characteristics of a flow. DetNet Transport Layer characteristics include time sync precision, time interval between packets, packet size, jitter, and number of packets per window of time. This is how the network SLA is defined, but this is not the native terms for the application and a complex mapping must ensure that the path that is setup and the DetNet Transport Layer effectively matches the requirements from the DetNet Services Layer and above.
The DetNet encapsulation and decapsulation of one-in-one, one-in-many and many-in-one Deterministic flows belongs to the DetNet Transport Layer. Direct one-in-one flow stitching also belongs to the DetNet Transport Layer. This happens when a deterministic flow can be directly bridged into another, resource-to-resource, without the need of an upper layer adaptation such as service protection from the Service Layer. A Detnet End-to-End Service flow may be stitched into one Detnet Service flow, or encapsulated in one or multiple Detnet Service flows.
Load Sharing refers to the encapsulation of a DetNet Flow in more than one DetNet flows, for instance using multiple small and more manageable DetNet Service Flows in parallel to carry a large Determistic End-to-End Service Flow, in order to avoid the need to periodically defragment the network. Packets are sequenced at the DetTrans Layer and distributed over the DetNet Transports paths in accordance to their relative capacities. In case of inconsistent jitter and Latency characteristics, packets may need to be reordered at the receiving DetTrans Layer based on the DSF Sequence.
+---------------+ +--------+ +--------+ +---------------+
| Application | |>-- DetNet Service -->| | Application |
+---------v-----+ +-------------Flow-------------+ +-----^---------+
| | | | | | | | | | | |
| | | | +--------+ +--------+ | | | |
| | | | | | | |
| | >------- DetNet End-to-End Service Flow -------> | |
| | | | | | | |
| | +-----+ +--------+ +--------+ +-----+ | |
| +---+ | |>-- DetNet Service -->| | +---+ |
| +-------------------Flow-------------------+ |
| DetTrans | | | | | | | | DetTrans |
+---------------+ +--------+ +--------+ +---------------+
DetNet Deterministic DetNet
End System Routers and Bridges End System
Figure 4: Load Sharing
In order to achieve this function, a Load Distribution function is required at the source and a Re-Ordering Function is required at the destination DetTrans End Point.
Flow Aggregation refers to the encapsulation of more than one DetNet flows in one DetNet Flow, for instance using one large and long-lived DetNet Service Flow from a third party provider to carry multiple more dynamic Deterministic End-to-End Service Flows across domains. Packets are sequenced at the DetTrans Layer and distributed over the DetNet Transports paths in accordance to their relative capacities. In case of inconsistent jitter and Latency characteristics, packets may need to be reordered at the receiving DetTrans Layer based on the DSF Sequence.
+---------------+ >-- DetNet Service --> +---------------+
| Application | Flow | Application |
+-----v---v-----+ +--------+ +--------+ +-----^---^-----+
| | | | | | | | | | | |
| | +--------------------------------------------------+ | |
| | >------- DetNet End-to-End Service Flow ---------> | |
| +----------------------------------------------------------+ |
| >----------- DetNet End-to-End Service Flow -------------> |
| DetTrans | | | | | | | DetTrans |
+---------------+ +--------+ +--------+ +---------------+
DetNet Deterministic DetNet
End System Routers and Bridges End System
Figure 5: Flow Aggregation
In order to achieve this function, a multiplexing function is required at the source and a demultiplexing function is required at the destination DetTrans End Point.
The DetNet Flows may also be used for Packet Replication and Elimination, in which case an elimination function is required at the DetTrans Termination.
1+1 Redundancy refers to injecting identical copies of a packet at the ingress of two non-congruent paths, and eliminating the excess copy when both are received at the egress of the paths. Packet Replication and Elimination extends the concept by enabling more than 2 paths, and allowing non-end-to-end redundant paths with intermediate Replication and Elimination points.
Redundancy and Load Sharing may be combined with the use of Network Coding whereby a coded packet may carry redundancy information for previous data packet and cover the loss of one, in which case the recovery function is required at the other DetTrans End Point. Network Coding provides a Forward Error Correction between multiple packets or multiple fragments of a packet. It may be used at the DSF layer to enable an efficient combination of redundancy and load sharing.
A DetTrans Flow may leverage multiple DetNet Flows in parallel in order to achieve its requirements in terms of reliability and Aggregate throughput. The "Deterministic Networking Architecture" clearly states that the capability of Replication and Elimination is not limited to the DetNet End Systems. DetNet Relay Nodes that operate DetTrans but then relay the packets are needed when the DetTrans operations are not end-to-end.
It may be that the DetTrans flow may need to traverse different domains where those Services are operated differently, e.g. controlled by different controllers or leveraging different technologies. It may also be that the bandwidth that is required is only available one segemnt at a time, and that for each segment, a different number of DetNet flows must be setup to transport the full amount of the DetTrans flow.
Figure 6 illustrates an example of the latter case, whereby The DetTrans Flow is distributed over two DetNet Flows, maybe operating PRE, then over three DetNet Flows, for instance operating Network Coding between them but using a smaller banswidth for each flow, and then two DetNet Flows again.
DetTrans is needed at the interconnection points to adapt the flows, recover losses and reinject the appropriate rates in the next segment.
+---+-----------+ +--------+ +--------+
| C |Application| +------------------------------+
| P +-----------+ | |>- DetNet Transport ->| | +---------------+
| U | DetTrans | | +--------+ +--------+ | | DetTrans / |
+---+-----------+ | | | ------- |
| N | Packet +-----+ +--------+ +--------+ | | / DetTrans |
| I | Queues -+ | |>- DetNet Transport ->| +--------+---+---+ |
| C | +---------------------------------------------+---+---+ |
+---+-----------+ +--------+ +--------+ +----|---|---|--+
| | |
DetNet | | |
Transport | | |
| | |
+---+-----------+ +--------+ +--------+ | | |
| C |Application| +------------------------------+ | | |
| P +-----------+ | |<- DetNet Transport -<| | +----|---|---|--+
| U | DetTrans | | +--------+ +--------+ +--------+---+---+ |
+---+-----------+ | +--------+---+---+ |
| N | +-----+ +--------+ +--------+ | | \ DetTrans |
| I | <-+ | |<- DetNet Transport -<| | | ------- |
| C | +------------------------------------+ | DetTrans \ |
+---+-----------+ +--------+ +--------+ +---------------+
DetNet-Aware Deterministic Deterministic
Host Systems Transit Nodes Relay Nodes
Figure 6: Intermediate Systems
Figure 7 illustrates a simple example of classical networked devices implementing the DetNet architecture. In that example, applications reside on Host systems and run on main CPUs; DetTrans is collocated with its applications and provides them with a Deterministic Service through DetTrans APIs. NICs provides the connectivity to the Deterministic Routers or Bridges acting at DetNet Edge and Relay Nodes - say as an example that they are IEEE Std. 802.1 TSN Bridges.
Deterministic
Host System Routers and Bridges Host System
+---+-----------+ | +--------+ +--------+ | +---+-----------+
| C |Application| |---| |----| |---| | C |Application|
| P +-----------+ | |<- DetNet Transport ->|-------+ P +-----------+
| U | DetTrans | | +--------+ +--------+ | | U | DetTrans |
+---+-----------+ <------------ DetTrans ------------> +---+-----------+
| N | Lower | | +--------+ +--------+ | | N | Lower |
| I | Layers |---| |<- DetNet Transport ->| |---| I | Layers |
| C | (queues) | |---| |----| |---| | C | |
+---+-----------+ | +--------+ +--------+ | +---+-----------+
<-UNI-> <-- DetNet Services -> <-UNI->
<------------- DetNet End-To-End Services ------------------>
Figure 7: Example Physical Network
The DetTrans Layer aggregates the data coming from the application up to a maximum frame size that is part of the SLA with the DetNet Transport. Packets thus formed can be distributed over any of multiple DetNet Transport sessions that are defined to accept that packet size. Packets formed at the DetTrans Layer are queued and ready to be delivered through the DetNet-UNI either with a Rate-Based or a Network-Pull mechanism.
If the NIC is DetNet-Aware then the queue can be offboarded to the NIC and it can be drained with a time gate (Rate-Base) or a message-driven gate (Network-Pull). Else, the queue is handled by the CPU and hopefully it can be drained within an interrupt, either for a timer (Rate-Base) or for a message (Network-Pull).
The DetNet-UNI protocol enables the DetNet transport ingress point to control when the DetTrans Layer transmits its Data packets. It may happen that the DetTrans Layer has not formed a fully-sized packet when time comes for sending it, in which case the packet will be sent with a size below the maximum.
The DetNet UNI uses ICMPv6 to carry its protocol elements. Data Packets across the UNI are encapsulated in order to carry DetNet-UNI control information to identify the reason of a loss or a delay, and determine the action to be taken in case of a packet lost or delayed over the interface.
The logical DetNet End System depicted in Figure 2 comprises several elements which may implemented in one or separate physical Systems. The example dichotomy in Figure 3 segregates ingress shaping and DetNet Relay functions, which are performed by IEEE Std. 802.1 TSN Bridges, from a DetNet-Aware Host.
Hosts and Edge Bridges are connected over Ethernet and together they form a DetNet End System. As it goes, this example introduces a further dichotomy within the Host, between the CPU and the NIC, across a local bus such as PCI, as illustrated in Figure 8.
+-------------+ +-------+ +-----------+
| Application | | MAC | | Ingress |
+-------------+ <--PCI--> +-------+ <---- Ethernet ----> | Shaping |
| DetTrans | | PHY | | and Relay |
+-------------+ +-------+ +-----------+
CPU NIC IEEE Std.
<------------- DetNet-UNI -------------> 802.1
<----------- Host System -------------> TSN Bridge
<------------------------- DetNet End System ------------------------>
Figure 8: Chained Functions
The NICs in the Host System may not participate to the network time Synchronization and may not be aware of the DetNet protocols running between the Deterministic Routers and Bridges, and the associated scheduling rules. In that situation, the DetNet-UNI operates on a Local Loop to ensure that a packet that leaves the Transport reaches the Router or Bridge just in time for injection into the Deterministic data plane and to provide a flow control that avoids congestion loss at the interface.
It is also possible that the NIC participates to the Deterministic Network but still has asynchronous communication with DetTrans Layer running on the the CPU. Either way, a flow control over a local loop must be implemented to drain the queues from the DetTrans layer and feed the network just in time for the deterministic transmission.
Depending on the level of support by the NIC, the loop may be placed on a different interface but remains functionally the same.
If the NIC is not aware at all of DetNet, then it is a plain pipe for the Deterministic Traffic. The Local Loop operates between the Edge TSN Bridge and the CPU as illustrated in Figure 9.
+---+-----------+ +---+-------+ +-----------+
| C |Application| | N | MAC | | Ingress |
| P +-----------+ <--PCI--> | I +-------+ <-- Ethernet --> | Shaping |
| U |DetTrans | | C | PHY | (Bridged | / Transit |
+---+-----------+ +---+-------+ or P2P) +-----------+
<------------- Local Loop -------------> Edge 802.1
<----------- Host System -------------> TSN Bridge
Figure 9: DetNet Unaware NIC
If the NIC is fully DetNet-Capable and participates to the deterministic Network including time synchronization and scheduling, then the local loop operates between the CPU and the NIC as illustrated in Figure 10.
+---+-----------+ +---+-----------+ +----------+
| C |Application| | N | Ingress | | DetNet |
| P +-----------+ <--PCI--> | I + Shaping | <--DetNet--> | Transit |
| U |DetTrans | | C | / Transit | Transport | Node |
+---+-----------+ +---+-----------+ +----------+
<-Local-
-Loop ->
<-------------- Host System --------------> TSN Bridge
Figure 10: DetNet Capable NIC
If the NIC is DetNet-Aware and does not participates to the deterministic Network including time synchronization and scheduling, then there are two local loops, one that operates between the CPU and the NIC and one that operates between the NIC and the Edge TSN Bridge as illustrated in Figure 11.
+---+-----------+ +---+-------+ +-----------+
| C |Application| | N | MAC | Ethernet | Ingress |
| P +-----------+ <--PCI--> | I +-------+ <-- (Bridged --> | Shaping |
| U |DetTrans | | C | PHY | or P2P) | / Transit |
+---+-----------+ <-Local- +---+-------+ +-----------+
-Loop -> <-- Local Loop --> Edge
<----------- Host System -------------> DetNet Transit
Figure 11: DetNet Capable NIC
The flow control at the DetNet-UNI can take any of two forms:
if the NIC is DetNet-Aware, it is expected that it maintains the DetTrans queues in order to provide a deterministic response to the DetNet-UNI, and in that case another control loop between the NIC and the CPU is needed to ensure that the queue in the NIC is always fed in time by the DetTrans Layer; this second loop may be of a different nature than the DetNet-UNI one and may for instance be operated within an interrupt to limit the asynchronism related to message queueing.
The "More" message enables a DetNet Transport Edge to pull one packet from the DetTrans Layer in Network-Pull mode. The message is associated with a future transmission opportunity for a packet. The "More" messages are indexed by a wrapping More Sequence Counter (MSC). The Transport Edge also maintains wrapping counters of Successful Packet Transmissions (SPT) and Missed Transmit Opportunities (MTO). The current value of these counters is placed in the "More "message.
Upon reception of a "More" message, the DetTrans Layer, or the NIC on behalf of the DetTrans Layer, sends the next available packet for that session. The packet is encapsulated and the encapsulation indicates the MSC. This enables the DetNet Transport Edge to correlate the packet with the transmission opportunity and drop packets that are overly delayed.
The "Time-Correction" message enables a DetNet Transport Edge to adjust the timer associated to the DetNet-UNI session in Rate-Based mode. In that mode, the DetTrans Layer sends a packet and restarts a timer at a period that corresponds to the transmission opportunity of the DetNet Transport Edge. If the clock in the CPU drifts, the DetNet Transport Edge will start receiving packets increasingly ahead of expected time or behind expected time. It is expected that the DetNet Transport Edge is protected against a minimum drift by a guard time, but if the drift becomes too important, then the DetNet Transport Edge issues a "Time-Correction" message indicating a number of time units (e.g. microseconds) by which the DetTrans Layer should advance or delay is next time out.
The loss of a packet beween the DetTrans Layer and the DetNet Transport Edge will correspond to a missed Transmission Opportunity but this does not mean that packets are piling up at the DetTrans Layer. OTOH, if a "More" message is lost, then one packet will not be dequeued and the Detrans queue might grow, increasingly augmenting the latency. It is thus important to differentiate these situations, and in the latter case, discard an extraneous packet to restore the normal level in the DetTrans queue for that session.
In order to do so, the DetTrans Layer maintains the record of the Number of Packets Sent (NPS), that it compares with the variation of the MTO and SPT counters in the "More" message. Upon a "More" message, the DetTrans Layer computes the variation of NPS (dNPS=NPS2-NPS1) and the variation of SPT (dSPT=SPT2-SPT1) since the previous "More" Message . dNPS is typically 1 if the transport always has data to send. Packets in flight when the "More" message is sent are considered lost since they will be received after their scheduled transmission opportunity, so the Number of Packets Losses (NPL) is (NPL=dNPS-dSPT). The DetTrans Layer also computes the variation of MTO since the previous "More" Message (dMTO=MTO2-MTO1). Since a packet loss implies a missed transmission opportunity, there cannot be more packets losses than missed opportunities, so we have dMTO>=NPL. dMTO-NPL represents the number of missed opportunities that are not due to a packet lost or late arrival, thus this is the sub-count of MTOs due to the loss of a "More" message.
For each loss of a "More" message, a packet in the DetTrans queue should be discarded. In order to simplify that operation and outboard it to the NIC, the Transports marks some packets as "Discard Eligible" (DE). A packet can be marked DE if there are enough alternate transmissions of non-DE packets to recover this. For instance, in case of Packet Replication and Elimination only one copy can be marked DE, and the marking should alternate between the sessions to cover a loss on either one rapidly.
The generic threats against Deterministic Networking are discussed in the "Deterministic Networking Security" document.
Security in the context of Deterministic Networking has an added dimension; the time of delivery of a packet can be just as important as the contents of the packet, itself. A man-in-the-middle attack, for example, can impose, and then systematically adjust, additional delays into a link, and thus disrupt or subvert a real-time application without having to crack any encryption methods employed. See [RFC7384] for an exploration of this issue in a related context.
Packet Replication and Elimination of done right can prevent a man-in-the-middle attack on one leg to actually impact the flow beyond the loss of an individual packet for lack of redundancy. This specification expects that PRE is performed at the transport level and provides specific means to protect one leg against misuse of the other.
This document does not require an action from IANA.
The authors wish to thank Patrick Wetterwald, Leon Turkevitch, Balazs Varga and Janos Farkas for their various contributions to this work. Special thanks to Norm Finn for being a (if not the) major thought leader to the whole deterministic effort, and for some text that is inlined here from other IETF documents, for the convenience of the reader.