Internet DRAFT - draft-guo-detnet-compensation-reference

draft-guo-detnet-compensation-reference



DetNet                                                         D. Guo
Internet Draft                                                 X. You
Intended status: Informational                                 R. Liu
Expires: 11 June 2024                                          S. Zhu
                                         New H3C Technologies Co., Ltd
                                                      11 December 2023



           Compensation Reference for Jitter Reduction Mechanism
              draft-guo-detnet-compensation-reference-00.txt


   Abstract

   This document describes several methods for obtaining reference delay
   applicable to different scenarios. These methods are in conjunction
   with mechanisms aimed at reducing end-to-end delay jitter in a
   scalable deterministic network. The goal is to meet specific business
   requirements and provide greater flexibility in the multipath
   planning for service protection.

   Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on June 11, 2024.

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents



Guo, et al.             Expires June 11, 2024                 [Page 1]

Internet-Draft     Compensation Reference for JRM        December 2023


   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

   Table of Contents


   Abstract..........................................................1
   Status of this Memo...............................................1
   Table of Contents.................................................2
   1. Introduction...................................................2
   2. Terminology....................................................3
   3. Network Model..................................................4
      3.1. SN Model..................................................6
      3.2. AN Model..................................................7
   4. Compensation Reference Delay Collection Methods...............10
      4.1. Compensation Reference Collection for SN.................12
         4.1.1. Implementation Principle............................12
         4.1.2. Data Plane..........................................13
         4.1.3. Control Plane.......................................13
      4.2. One-Way Compensation Reference Collection for AN.........15
         4.2.1. Implementation Principle............................15
         4.2.2. Data Plane..........................................18
         4.2.3. Control Plane.......................................18
      4.3. Round-Trip Compensation Reference Collection for AN......20
         4.3.1. Implementation Principle............................21
         4.3.2. Data Plane..........................................24
         4.3.3. Control Plane.......................................25
   5. Security Considerations.......................................26
   6. IANA Considerations...........................................26
   7. Acknowledgments...............................................26
   8. Contributors..................................................26
   9. References....................................................27
      9.1. Informative References...................................27
   Authors' Addresses...............................................29

1. Introduction

   In scaling deterministic networks, as defined in [I-D.ietf-detnet-
   scaling-requirements], end-to-end services may traverse multiple
   network domains and employ various queuing mechanisms within each
   domain. [I-D.guo-detnet-jitter-reduction-mechanism]describes a
   compensation mechanism designed to reduce end-to-end jitter and meet
   the requirements of applications with tightly bounded jitter. This



Guo, et al.             Expires June 11, 2024                 [Page 2]

Internet-Draft     Compensation Reference for JRM        December 2023


   document expands on the Jitter Reduction Mechanism for DetNet [I-
   D.guo-detnet-jitter-reduction-mechanism].

   For many applications, it is crucial that different copies of the
   same data transmitted through multiple paths maintain consistent
   delays within specified upper and lower bounds. This ensures that in
   the event of a path failure, the application remains unaware of the
   fault. Therefore, a unified value is required as the compensation
   delay reference for the transmission delays in multi-path transport
   providing service protection. This value is referred to as the Path
   Delay Reference, denoted as PthRefD in [I-D.guo-detnet-jitter-
   reduction-mechanism]. This value will be greater than the actual
   transmission delays on each path to accommodate additional delays
   introduced by anticipated future path changes (e.g., when a new path
   is added to the service protection path group after a path failure,
   and the new path is longer than the original paths in the service
   protection path group). By using the Path Delay Reference to
   compensate for transmission delays, applications remain unaware of
   such fault occurrences and recoveries within a bounded delay range.

   In certain scenarios, complete transmission delays for data during
   transmission may not be obtainable. According to [I-D.guo-detnet-
   jitter-reduction-mechanism], it is sufficient to obtain the upper
   limit of the compensation delay required for the corresponding path.
   In this document, the "upper limit of the compensation delay required
   for the corresponding path" is referred to as Cap_i, simplified as
   the compensation reference delay value in this document.

   Given the existence of both time-synchronized and non-time-
   synchronized scenarios in practical requirements, and considering the
   diversity of actual network deployments and the complexity of the
   issues themselves, it is almost impossible to have a unified method
   to address all requirements. Our approach is to abstract the network
   into two typical models and provide suitable methods for each model.
   In the following sections, we will first categorize and describe the
   network models and then provide methods for collecting the
   compensation reference delay values corresponding to different types.
   Different methods will be implemented for different network types in
   actual deployment to simplify the realization.

2. Terminology

   The following terminology is introduced in this document on the basis
   of [I-D.guo-detnet-jitter-reduction-mechanism]:

   HN



Guo, et al.             Expires June 11, 2024                 [Page 3]

Internet-Draft     Compensation Reference for JRM        December 2023


       Abbreviation of Head Node. For the definition of Head Node,
       please refer to [I-D.guo-detnet-jitter-reduction-mechanism].

   CN

       Abbreviation of Compensation Node. For the definition of
       Compensation Node, please refer to [I-D.guo-detnet-jitter-
       reduction-mechanism].

   Synchronous Networking (SN)

       It is a networking model in which time synchronization is
       implemented between HN and CN.

   Asynchronous Networking (AN):

       It is a networking model in which there is no time
       synchronization between HN and CN.

   Synchronous Domain (SD)

       If time synchronization is achieved between the I-GW and E-GW of
       the deterministic network domain through which the deterministic
       flow path passes, the deterministic network domain where the I-GW
       and E-GW are located is considered a synchronous domain.

3. Network Model

   Before deploying deterministic service flows, the controller plane
   performs path planning for the network, identifies the networking
   model to which the completed planned paths belong, and labels the
   paths accordingly. In the path from the Head Node (HN) to the
   Compensation Node (CN), if time synchronization is achieved between
   HN and CN, we consider the networking model for the deployment of
   deterministic flows to be SN, and we collect reference values using
   the SN method. In the path from HN to CN, if time synchronization is
   not achieved between HN and CN, we consider the networking model for
   the deployment of deterministic flows to be AN, and we collect
   reference values using the AN method.

   For AN, the path from HN to CN may contain Synchronous Domains (SD)
   consisting of multiple network nodes, each with independent Ingress
   Gateway (I-GW) and Egress Gateway (E-GW) devices. There may also be
   individual network nodes within the path that are not time-
   synchronized with other nodes. These nodes can be considered as
   special SDs, where a single node forms its own domain, with its I-GW



Guo, et al.             Expires June 11, 2024                 [Page 4]

Internet-Draft     Compensation Reference for JRM        December 2023


   and E-GW being the node itself, and time synchronization is required
   within the node.

   Upon entering an SD from the I-GW, Packet Replication Functions (PRF)
   may be implemented, and PRF and Packet Elimination Functions (PEF)
   may be implemented before or within the E-GW. The packets output from
   the E-GW of the SD may not be the original packets but rather
   multiple copies of the original packets. The copies generated by the
   internal PRF function within the SD may follow different transmission
   paths, and the transmission delays on different paths are different.
   The differences in delay introduced by the selection of packets from
   different paths by the PEF are considered as jitter within the SD.



   The controller plane actually performs path planning to plan out
   candidate paths that meet application requirements. Some of the
   candidate paths are selected to form a service protection group,
   while the remaining candidate paths can serve as backup paths in the
   event of a failure. The method described in this document is based on
   the paths that have already been planned, to collect compensation
   reference delay values.


























Guo, et al.             Expires June 11, 2024                 [Page 5]

Internet-Draft     Compensation Reference for JRM        December 2023


3.1. SN Model

            HN(PE1) and CN(PE2) synchronized
             ^                           ^
             ^                           ^
             ^                           ^
             ^     P1--------------P2    ^
             ^    /|               |\    ^
             ^   / |               | \   ^
            PE1 +  |               |  + PE2
           (HN) +  |               |  + (CN)
             |   \ |               | /   |
             |    \|               |/    |
             |     P3--------------P4    |
             |                           |
             |                           |
             |                           |
             |                           |
           Talker                      Listener

                     (a) an example of SN


                   +---------------------+
      Tanlker------+ I-GW(HN)---E-GW(CN) +-------Listener
                   +---------------------+

                     (b) the model of SN

               Figure 1 Example and Model Abstraction of SN



   Figure 1(a) shows an example of SN. The Talker is connected to PE1,
   which acts as the Head Node (HN), while the Listener is connected to
   PE2, which acts as the Compensation Node (CN). Time synchronization
   exists between PE1 and PE2.

   The application networking for accessing SN adopts the abstraction
   model shown in Figure 1(b), which is represented by the first I-GW
   (i.e., HN) and the last E-GW (i.e., CN). The residence delay between
   HN and CN is considered as variable delay within the SN, without
   focusing on the specific paths within the SN, thus simplifying
   implementation.





Guo, et al.             Expires June 11, 2024                 [Page 6]

Internet-Draft     Compensation Reference for JRM        December 2023


3.2. AN Model


             no time synchronization between HN and CN
                 ^                           ^
                 ^                           ^
                 ^                           ^
                 ^     P1--------------P2    ^
                 ^    /|               |\    ^
                 ^   / |               | \   ^
                PE1 +  |               |  + PE2
               (HN) +  |               |  + (CN)
                 |   \ |               | /   |
                 |    \|               |/    |
                 |     P3--------------P4    |
                 |                           |
                 |                           |
                 |                           |
                 |                           |
               Talker                      Listener

                    (a) an example of AN


            +------+                             +------+
            |      +---P1----P2------------------+      |
            |      |                             |      |
            |      +---P1----P2----P4------------+      |
            |      |                             |      |
            |      +---P1----P3----P4------------+      |
            | HN   |                             |  CN  |
            |(PE1) +---P1----P3----P4----P2------+ (PE2)|
            |      |                             |      |
            |      +---P3----P1----P2----P4------+      |
            |      |                             |      |
            |      +---P3----P1----P2------------+      |
            |      |                             |      |
            |      +---P3----P4----P2------------+      |
            |      |                             |      |
            |      +---P3----P4------------------+      |
            +------+                             +------+

                   (b) the model of AN

               Figure 2 Example and Model Abstraction of AN




Guo, et al.             Expires June 11, 2024                 [Page 7]

Internet-Draft     Compensation Reference for JRM        December 2023


   Figure 2(a) shows a simple example of AN. PE1 is connected to the
   Talker and acts as the Head Node (HN), while PE2 is connected to the
   Listener and acts as the Compensation Node (CN). There is no
   requirement for time synchronization between the various P, PE1, and
   PE2 nodes (Note: time synchronization can be present but is not
   utilized). Assuming each P node is a physical node, two-downstream-
   interface PRF is planned for each P node. Figure 2(b) shows the
   abstract model of AN. In this diagram, there are 8 different paths
   formed between PE1 and PE2 through different nodes.

   After path planning, a portion or all of the paths are used to form
   service protection members between HN and CN.




































Guo, et al.             Expires June 11, 2024                 [Page 8]

Internet-Draft     Compensation Reference for JRM        December 2023


                no time synchronization between HN and CN
                    ^                           ^
                    ^                           ^
                    ^       1           2       ^
                    ^    SD1--------------SD2   ^
                    ^   2/|0             0|\1   ^
                    ^ 1 / |               | \   ^
                   PE1 +  |               |  + PE2
                  (HN) +  |               |  + (CN)
                    | 0 \ |               | /   |
                    |   2\|1             1|/0   |
                    |    SD3--------------SD4   |
                    |       0           2       |
                    |                           |
                    |                           |
                    |                           |
                  Talker                      Listener

                     (a) an example of complex AN


          +------+                                      +------+
          |      +--2-SD1-1--2-SD2-1--------------------+      |
          |      |                                      |      |
          |      +--2-SD1-1--2-SD2-0--1-SD4-0-----------+      |
          |      |                                      |      |
          |      +--2-SD1-0--1-SD3-0--2-SD4-0-----------+      |
          | HN   |                                      |  CN  |
          |(PE1) +--2-SD1-0--1-SD3-0--2-SD4-1--0-SD2-1--+ (PE2)|
          |      |                                      |      |
          |      +--2-SD3-1--0-SD1-1--2-SD2-0-----------+      |
          |      |                                      |      |
          |      +--2-SD3-1--0-SD1-1--2-SD-1------------+      |
          |      |                                      |      |
          |      +--2-SD3-0--2-SD4-1--0-SD2-1-----------+      |
          |      |                                      |      |
          |      +--2-SD3-0--2-SD4-0--------------------+      |
          +------+                                      +------+

                      (b) a model of complex AN

             Figure 3 Complex AN Example and Model Abstraction







Guo, et al.             Expires June 11, 2024                 [Page 9]

Internet-Draft     Compensation Reference for JRM        December 2023


   Figure 3(a) shows an example of complex AN. PE1 is connected to the
   Talker and acts as the Head Node (HN), while PE2 is connected to the
   Listener and acts as the Compensation Node (CN). SD represents a
   deterministic network domain after time synchronization by edge
   gateways. There is no requirement for time synchronization between
   SD, PE1, and PE2 (Note: time synchronization can be present but is
   not utilized). Assuming two-downstream-interface PRF copies are
   planned for each SD, Figure 3(b) shows the abstract model of this
   complex AN. Taking "2-SD1-1" in the figure as an example, 2
   represents the input interface number of the I-GW of SD1, the first 1
   represents the corresponding SD number in Figure 3(a), and the second
   1 represents the output interface number of E-GW of SD1. The
   numbering of SDs and interfaces is for illustrative purposes,
   primarily for the sake of simplicity and clarity in describing the
   network topology as multiple paths. In Figure 3(b), the differences
   in transmission delay within the SD are considered as jitter within
   the domain, and 8 different paths are formed between PE1 and PE2
   through different nodes. After path planning, a portion or all of the
   paths are used to form service protection members between HN and CN.

4. Compensation Reference Delay Collection Methods

   As mentioned earlier, time synchronization between the I-GW and E-GW
   of SD (treated the same as SN) allows for the direct acquisition of
   residence delay within it, eliminating the need to consider where
   copies are generated inside the SD/SN, which path each copy is
   transmitted on, and the final selection of the copy. Instead, the
   differences in transmission delay of copies from different paths are
   uniformly considered as jitter within the SD/SN, greatly simplifying
   implementation without compromising the reliability of deterministic
   elements, covering the application scenarios proposed in [I-D.draft-
   ietf-detnet-mpls-over-ip-preof].

   In AN, there is no time synchronization between HN and CN, making it
   impossible to directly obtain end-to-end network residence time. In a
   complex AN containing SD, the residence delay within the SD through
   which the HN->CN path passes can be obtained through the cooperation
   of the SD's I-GW and E-GW, independent of other nodes within the SD.

   The input interface of the SD's I-GW and the output interface of the
   E-GW participate in path identification, as do the input and output
   interfaces of individual nodes.







Guo, et al.             Expires June 11, 2024                [Page 10]

Internet-Draft     Compensation Reference for JRM        December 2023



                                         +------------+
                +---------------E1---+    |            |
    +---+       |               |    +---R3---+        |          +---+
    |src|------R1           +---+             |        E3----O----+dst|
    +---+       |           |                 E2-------+          +---+
                +----------R2                 |
                            +-----------------+

    R: replication function (PRF)
    E: elimination function (PEF)
    O: ordering function (POF)

                Figure 4 PREOF scenario in a DetNet network

   Figure 4 is copied from [I-D.ietf-detnet-mpls-over-ip-preof]. During
   data forwarding, the E-node needs to implement PEF to remove the
   redundant copies generated by multiple PRFs. However, when collecting
   compensation reference delay values in OAM packets, the copies
   generated by PRFs need to be retained, and the CN discards them after
   compensation reference delay values are applied.

   Next, we will introduce the compensation reference value collection
   methods for synchronous and asynchronous networks, respectively.
























Guo, et al.             Expires June 11, 2024                [Page 11]

Internet-Draft     Compensation Reference for JRM        December 2023


4.1. Compensation Reference Collection for SN

4.1.1. Implementation Principle


         |                                                   |
         |<------------------PthRefD------------------------>|
         |                                                   |
         |t0                      |t1   |t2         |t3      |tr
         +------------------------+-----+-----------+------->+
         |                        |     |           |        |
         |                        |     |           |        |
         |    Copy3 /------------>+     |           |        |
         |         /              |     |           |        |
    Copy2|  /-----/---------------------+---------->+<-Tr_2->+
         | /                            |           |        |
    Copy1+/---------------------------->+                    |
         |                              |                    |

                 Figure 5 SN Compensation Reference Delay



   For the SN, the compensation reference delay is denoted as the path
   reference delay (PthRefD).

   When the timestamp entering the HN is set as the starting time t0,
   and the arrival times of the copies generated by each PRF at the CN
   are denoted as t_i (where 0 < i < n + 1, and n is the number of
   received copies). Let t_k (where 0 < k < n + 1) represent the latest
   arrival time of all the copies at the CN. Based on t_k - t_0, an
   additional value is added as the compensation reference value,
   determined by the user or controller based on engineering
   requirements. This additional value includes the extra transmission
   delay, the margin for variable delay, and the processing delay at the
   CN before scheduling. This value is the PthRefD.

   Note: To simulate obtaining the HN receive timestamp, after
   constructing the OAM message, the HN sends it to the receive
   interface, and the receive interface then loops it back to the HN.

   As shown in Figure 5, the timestamp for receiving the OAM loopback
   message at the HN is t0. On the HN, PRF is implemented, generating
   Copy1 and Copy2. Copy2 is further processed by a node inside the SD,
   generating Copy3. The timestamps for the arrival of the three copies
   at the CN are denoted as t3. Adding Tr_3 to t3 provides the value tr,
   where Tr_3 is a value chosen by the user or controller based on


Guo, et al.             Expires June 11, 2024                [Page 12]

Internet-Draft     Compensation Reference for JRM        December 2023


   engineering requirements. The value tr represents the reference time
   point for the path reference delay (PthRefD) relative to t0 on the
   CN. It can also be understood as adding the user or controller-
   selected value Tr_3 to (t3-t0) to obtain the path reference delay
   (PthRefD).

   Therefore, we have:

   PthRefD = tr - t0, or

   PthRefD = (t3 - t0) + Tr_3.

   The PthRefD for the SN represents the compensation reference.

4.1.2. Data Plane

   For the one-way collection of compensation reference for the SN, the
   data plane behaviors of each node are described as follows:

   1. The HN data plane receives the OAM message for collecting
   compensation reference delay and transforms it into an OAM packet for
   collecting compensation reference delay.

   2. The HN sends the OAM packet for collecting compensation reference
   delay. (e.g.: To simulate the reception of app-flow through loopback
   on the HN: assuming the Talker is connected to interface if_0 of the
   HN, an OAM packet with loopback indication is sent to if_0. Upon
   receiving this packet after looping back, the receiving timestamp is
   recorded. After performing the PRF on the HN, the OAM copies are sent
   to the CN via different paths planned by the controller plane.)

   3. At each node implementing PEF on the HN->CN path, when identifying
   OAM for collecting SN compensation reference delay, the node does not
   eliminate duplicate copies and directly forwards the message.

   4. At the CN node, upon receiving the OAM packet for collecting SN
   compensation reference delay, the measurement data is reported to the
   control plane.

   At the CN, it is necessary to ensure that a set of copies of an OAM
   packet covers all the PREOF paths within the domain.

4.1.3. Control Plane

   For the one-way collection of compensation reference for the SN, the
   control plane behaviors of each node are described as follows:



Guo, et al.             Expires June 11, 2024                [Page 13]

Internet-Draft     Compensation Reference for JRM        December 2023


   1. Plan network paths.

   2. Configure operations for nodes HN and CN.

   3. The control plane of CN needs to specify conditions of integrity
   check (ensuring the delays for all planned paths are collected).

   4. The control plane needs to allocate OAM packet sequence numbers
   and send messages to HN to construct OAM packets with these sequence
   numbers. These OAM packets are used to collect the compensation
   reference delay.

   5. Coordinate with the data plane to perform measurements and
   calculate the reference delay. The control plane receives measurement
   data reported by CN, calculates the longest transmission delay
   between HN and CN, and adds a value chosen by the user or controller
   based on engineering requirements (including additional values for
   line transmission, margin for variable delay, and processing delay at
   CN before scheduling) to obtain the compensation reference delay for
   HN->CN. Subsequently, the transmission delay for packets of DetNet
   flow is compensated based on the actual residence delay and this
   compensation reference delay.

   6. Configure the compensation reference delay value to HN or CN. If
   configured on the HN node, the HN node can determine the point in
   time after compensation, simplifying CN implementation. If configured
   on the CN node, the CN node determines the point in time after
   compensation.

   7. Runtime maintenance: periodically collect and calculate the full
   path reference delay for HN->CN, and refresh the reference delay for
   HN or CN. Isolate paths that have faults, and release isolation after
   recovery.

   For the collection of residence delay, the control plane needs to
   provide relevant information to the data plane of each network node
   in two ways:

   Option 1: The control plane globally assigns node identifiers and
   distributes these identifiers to each network node. The control plane
   sends delay collection-related operations to HN.

   Option 2: The control plane configures operations on a per-flow basis
   to HN and CN.





Guo, et al.             Expires June 11, 2024                [Page 14]

Internet-Draft     Compensation Reference for JRM        December 2023


4.2. One-Way Compensation Reference Collection for AN

   The one-way collection method is suitable for cases where the round-
   trip paths are different. Refer to [RFC9016] for the description of
   BiDir. According to the compensation delay formula in [I-D.guo-
   detnet-jitter-reduction-mechanism]:



   CompD_i = Cap_i - (T'1_i + T'2_i + T'3_i + T'4_i)



   Cap_i is obtained before the deployment of DetNet-flow sessions and
   is used as the compensation reference delay value. This value is the
   difference between PthRefD and FixD_i. Since time synchronization has
   not been performed, PthRefD and FixD_i cannot be directly obtained.
   Therefore, the compensation reference delay value Cap_i needs to be
   transformed into a comparable relative value.

4.2.1. Implementation Principle


          |                                                   |
          |<------------------PthRefD------------------------>|
          |                                                   |
          |t_0                             |t_1      |t_3     |tr
          +--------------------------------+---------+------->+
          |                                |         |        |
          |                                |         |        |
          |                                |         |        |
          |                                |         |        |
     Path2+--------------------------------+-------->+<-Tr_2->+
          |                                |         |        |
     Path1+------------------------------->+<------Tr_1------>+
          |                                |                  |

         Figure 6 Multi-path reference delay and compensation delay

   According to section 3.2, the paths between HN and CN can be
   abstracted as multiple paths. To simplify, consider Figure 6 with 2
   paths and describe the method for calculating the compensation
   reference delay value. Let t0 be the time when the measurement data
   packet is received at HN, and let t_1 and t_2 be the times when two
   copies of the same measurement data packet arrive at CN via Path1 and
   Path2, respectively. Obviously, t_2 is the arrival time of the last
   copy. tr is the reference time point based on t_2 delayed by Tr_2,


Guo, et al.             Expires June 11, 2024                [Page 15]

Internet-Draft     Compensation Reference for JRM        December 2023


   where the value of Tr_2 is chosen by the user or SDN controller based
   on engineering requirements (including additional values for line
   transmission, margin for variable delay, and processing delay at CN
   before scheduling). The time difference between t_0 and tr
   corresponds to PthRefD as described in [I-D.guo-detnet-jitter-
   reduction-mechanism]. Since t_0 represents the time at HN, and t_1,
   t_2, and tr represent the time at CN, and there is no time
   synchronization between HN and CN, PthRefD cannot be directly
   obtained by subtraction. However, t_1, t_2, and tr are all times at
   CN, and the relative delay differences for each path can be obtained.



   PthRefD = (FixD_1 + ActRefD_1) + Tr_1

   = FixD_1 + (ActRefD_1 + Tr_1)

   PthRefD - FixD_1 = ActRefD_1 + Tr_1



   Where Cap_1 = PthRefD - FixD_1,

   Therefore,

   Cap_1 = ActRefD_1 + Tr_1

   That is, Cap_1 is the sum of (ActRefD_1 + Tr_1), FixD_1 is the fixed
   delay of Path1, which is constant and not considered in compensation.
   ActRefD_1 is the sum of the residence delays of the measurement
   packets in each domain, and Tr_1 is the delay from t1 to tr, i.e.,
   tr-t1. Based on the method for obtaining actual transmission delay
   within the domain as described in [I-D.guo-detnet-jitter-reduction-
   mechanism], ActRefD_1 can be obtained, and thus Cap_1 can be
   obtained; similarly, Cap_2 can also be obtained.


                   Path1
                   -------------------------------->
           ----                                         ----
          /    \   Path2                               /    \
         |  HN  |  -------------------------------->  |  CN  |
          \    /                                       \    /
           ----    Path3                                ----
                   -------------------------------->

              Figure 7 Illustration of OAM packet transmission


Guo, et al.             Expires June 11, 2024                [Page 16]

Internet-Draft     Compensation Reference for JRM        December 2023


   As shown in Figure 7, according to the service protection
   implementation, HN will send multiple copies of the same OAM packet
   on multiple paths, and CN will collect information and calculate the
   compensation reference delay value for each member path. The process
   to obtain the Cap_i value is as follows:

   1. Record the corresponding path and reception time at CN;
   simultaneously, according to the method described in [I-D.guo-detnet-
   jitter-reduction-mechanism], obtain the residence delays in each
   domain and accumulate them together to obtain the actual residence
   delay of the OAM packets for each path. For ease of description,
   let's assume that Path i is an arbitrary path for service protection,
   and the accumulated value of the actual residence delay within each
   domain after the OAM packet passes through this path is denoted as
   ActRefD_i.

   2. Collect the latest arrival time t_2 (as shown in Figure 7).

   3. Based on engineering requirements, select the required delay
   adjustment Tr_2 (as shown in Figure 7), and delay the latest arrival
   time to obtain the reference time point tr, i.e., tr = t_2 + Tr_2.

   4. Calculate the difference between the arrival time of the
   measurement packet for each path relative to tr, denoted as Tr_i
   (such as Tr_1, Tr_2 in Figure 7).

   Tr_i = tr - t_i;

   5. Calculate the compensation reference delay value for each path i
   as Cap_i = ActRefD_i + Tr_i.

   When calculating the compensation delay for each path, if a data
   packet passes through Path i and the residence delay in each domain
   is ActD_i (ActD_i is a variable delay and may not be the same after
   each transmission), then the required compensation delay is

   CompD_i = Cap_i - ActD_i.

   Cap_i can be distributed to the HN node, encapsulated into the
   DetNet-flow data packet, and used by CN to perform delay compensation
   using Cap_i and ActD_i obtained from the data packet; or Cap_i can be
   distributed to the flow-related compensation information table at CN
   for delay compensation.






Guo, et al.             Expires June 11, 2024                [Page 17]

Internet-Draft     Compensation Reference for JRM        December 2023


4.2.2. Data Plane

   Description of the data plane behavior for the one-way collection of
   compensation reference delay at each node:

   1. HN's data plane receives the OAM message for collecting
   compensation reference delay values and converts it into an OAM
   packet for collecting compensation reference delay values.

   2. HN sends the OAM packet for collecting compensation reference
   delay.

   Example: Simulating the reception of deterministic data via loopback
   at HN: Assuming the Talker is connected to interface if_0 of HN, an
   OAM packet marked for loopback is sent to if_0. When this packet is
   looped back and received, the reception timestamp is recorded. After
   performing the PRF, different copies of the OAM packet are sent to CN
   via different paths planned by the controller plane.

   3. Each node (if it is an edge gateway I-GW/E-GW of the SD, it is
   handled by the SD's I-GW and E-GW) collects the residence delay of
   the OAM packets passing through the path and, with the assistance of
   the control plane, generates the necessary identification information
   for identifying the path. Similar to the requirements in [I-D.ietf-
   detnet-pof], when CN nodes receive an OAM packet or a service data
   packet, they need to obtain path-related information.

   4. At each node implementing PEF on the HN->CN path, when identifying
   OAM for collecting compensation reference delay, the node does not
   eliminate duplicate copies and directly forwards the message.

   5. When CN nodes receive OAM packets, they record the reception time
   of each OAM packet and send it to the control plane.

4.2.3. Control Plane

   The control plane behavior of each node for one-way collection of
   compensation reference delay for AN is described as follows:

   1. Plan network paths and identify the network as SN or AN.

   2. Configure the necessary information for generating path
   identifiers at each node. Similar to the requirements in [I-D.ietf-
   detnet-pof], when CN nodes receive an OAM packet or a service data
   packet, they need to obtain path-related information.




Guo, et al.             Expires June 11, 2024                [Page 18]

Internet-Draft     Compensation Reference for JRM        December 2023


   3. Configure the path identifier information on CN to identify
   different paths on which OAM packets or app-flow packets transmitted.



   4. Set up a timeout mechanism in the control plane to identify if the
   received OAM packet arrival time is acceptable. The historical window
   mechanism on TSN in [IEEE802.1CB] can be used as a reference.

   5. Configure operations at intermediate nodes to collect residence
   delays.

   6. Perform integrity checks in the control plane.

   7. Allocate OAM packet sequence numbers and send messages to HN for
   sending OAM packets to collect compensation reference delays.

   8. Coordinate with the data plane to perform measurements and
   calculate reference delay values. The control plane receives
   measurement data reported by CN, calculates compensation reference
   delay values, and records the arrival timestamps of multiple copies
   of the same OAM packet received on each path. It also records the
   total number of copies received on each path (used for integrity
   checks) and the sum of residence delays within each SD and non-SD
   node on that path. Based on the latest timestamp recorded on all
   paths, the control plane adds a value chosen by the user or
   controller based on engineering requirements (including additional
   values for line transmission, variable delay upper limit margin, and
   processing delay at CN before scheduling) to establish a common
   compensation delay reference point tr for all paths from HN to CN. tr
   is the timestamp at CN. The compensation reference delay values for
   each path are calculated as described in section 4.2.1.

   9. Configure the compensation reference delay values to HN or CN.

   10. Runtime maintenance: periodically collect and calculate the full
   path reference delay and update the reference delays to HN or CN.

   For delay collection in each domain, the control plane provides the
   relevant information required by the data plane and determines the
   provision method. The control plane can provide this information in
   the following ways:

   Method 1: Globally allocate node identifiers in the control plane and
   distribute this identifier to each node. The control plane
   distributes the information required for delay collection to HN.



Guo, et al.             Expires June 11, 2024                [Page 19]

Internet-Draft     Compensation Reference for JRM        December 2023


   Method 2: The control plane configures per-flow operations at
   intermediate nodes, and if SD exists, configures it at the SD's I-GW
   and E-GW.

4.3. Round-Trip Compensation Reference Collection for AN

   The round-trip collection method is suitable for situations where the
   round-trip paths are the same. See [RFC9016] for a description of
   BiDir.

   Due to the lack of global clock synchronization, we cannot directly
   obtain the end-to-end path delay. It would be very complicated to
   directly obtain the delay of each path segment and then accumulate
   the delay of each segment. Therefore, we draw on the idea of 4.2 and
   use relative-delay method. By using the relative-delay method to
   accurately obtain the approximate invariant part of the delay,
   combined with the variable residence delay on the path and other
   delays, PthRefD is obtained by comprehensive evaluation. At the same
   time, the difference between the approximate invariant part of the
   delay for the specified path and PthrefD is adjusted into the
   compensation reference delay Cap_i. According to the formula of
   compensation delay in [I-D.guo-detnet-jitter-reduction-mechanism]:

   CompD_i = (PthRefD - FixD_i  - (T1_i + T2_i + T3_i + T4_i)

   Let Cap_i = PthRefD - FixD_i,

   We need to design a method to obtain PthRefD and FixD_i in order to
   obtain Cap_i before deploying DetNet flows. During the runtime, the
   approximate invariant part of the delay FixD_i in the service
   protection member path_i is periodically obtained, and the newly
   obtained value is used to update the previous FixD_i. When the
   DetNet-flow packets are actually transmitted, the new FixD_i and
   PthRefD are used to calculate the new Cap_i.














Guo, et al.             Expires June 11, 2024                [Page 20]

Internet-Draft     Compensation Reference for JRM        December 2023


4.3.1. Implementation Principle

             HN                                     CN
       +----------+        Req_1               +----------+
       |    th_R_1|<---------------------------|tc_R_1    |
       |          |                            |          |
       |    th_A_1|        Ack_1               |          |
       |          |--------------------------->|tc_A_1    |
       |          |        Ack_2               |          |
       |    th_A_2|--------------------------->|tc_A_2    |
       |          |        ......              |......    |
       |          |                            |          |
       |          |        Ack_n               |          |
       |    th_A_n|--------------------------->|tc_A_n    |
       +----------+                            +----------+


                     (a) Round-Trip Collection Message


     |<-------t_Req_1------>|<-------------PthRefD----------->|
     |                      |                                 |
     |tc_R_1                |th_R_1      |tc_A_1  |tc_A_2     |
     |----------------------+------------+--------+-----------|->time
     |                      |            |        |           |
     |                      |            |        |           |
     |                Path_2|------------+------->|<---Tr_2-->|
     |                      |            |                    |
     |                Path_1|----------->|<--------Tr_1------>|
     |                      |            |                    |
     |                      |            |                    |

              (b) Delay Model for Round-Trip Collection Method

           Figure 9 Round-Trip Collection Message and Delay model

   According to section 3.2, the paths between HN and CN can be
   abstracted as multiple paths.

   Figure 9(a) is a diagram for measuring and collecting message delay
   information. CN sends a request type OAM message, Req_1, to HN
   through Path_1 (Path_1 being any path within the service protection
   group). Upon receiving the message, HN generates n response type OAM
   messages, Ack_1 to Ack_n, and sends them through Path_1 to Path_n
   back to CN. CN records the transmission timestamp, tc_R_1, when
   sending the request message. During the transmission of Req_1, each
   node (including the SD's I-GW and E-GW, or non-SD nodes) collects and


Guo, et al.             Expires June 11, 2024                [Page 21]

Internet-Draft     Compensation Reference for JRM        December 2023


   calculates the residence delay on the path of CN->HN at SD or non-SD
   nodes, which is designated as ActD_Req (excluding the line delay
   between SDs, or between non-SD nodes, or between SD and non-SD
   nodes). During the transmission of Act_i (i = 1 to n), each node
   records the residence delay on the path of HN->CN at SD or non-SD
   nodes, denoted as ActD_Act_i (i = 1 to n) (excluding the line delay
   between SD, non-SD nodes, or between SD and non-SD nodes).

   HN records the reception timestamp th_R_1 for Req_1 and the
   transmission timestamps th_A_1 to th_A_n for the n response-type OAM
   messages Ack_1 to Ack_n. HN calculates n residence delays and uses
   them as the initial values for ActD_Act_1 to ActD_Act_n:

   ActD_Act_i = th_A_i - th_R_1 (i = 1 to n)

   Along with carrying ActD_Act_i (i = 1 to n), Act_i (i = 1 to n) also
   carries ActD_Req. CN records the reception timestamp tc_A_i (i = 1 to
   n) for receiving Act_i. CN can obtain ActD_Req and ActD_Ack_i from
   Act_i.

   Figure 9(b) shows a comprehensive delay model for multiple stages,
   using an example with two paths. In this figure, th_R_1 and th_A_1~2
   are timestamps at HN, while tc_R_1 and tc_A_1~2 are timestamps at CN.
   The tr represents the compensation delay reference point for all
   paths from HN to CN, calculated as the latest timestamp of receiving
   the response-type OAM in CN (tc_A_2 in the figure), with additional
   values determined by the user or controller based on engineering
   requirements. These additional values include extra link delay,
   variable residence delay upper limit, and any processing delay at CN
   before scheduling.

   The specific processing steps are described as follows:

   1. delay information collection. As shown in Figure 9(a), Path_1 to
   Path_n are n paths with the same HN and CN, used for service
   protection.

      a) CN sends Req_j message to HN via any path j (0<j<n+1), records
   the transmission time tc_R_j, and sets the accumulated residence
   delay of the SD or non-SD nodes that Req_j passes through as
   ActD_Req. In the request message, ActD_Req is individually carried in
   a field (Note: ActD_Req does not include the link delay between SDs,
   or between non-SD nodes, or between non-SD nodes, as there is no time
   synchronization).

      b) Upon receiving Req_j at HN, the reception time is recorded as
   th_R_j. HN constructs n response messages Ack_1 to Ack_n and sends


Guo, et al.             Expires June 11, 2024                [Page 22]

Internet-Draft     Compensation Reference for JRM        December 2023


   them to CN. The accumulated residence delay of the nodes that Ack_i
   (i=1 to n, same below) passes through is set as ActD_Ack_i. In each
   response message, in addition to carrying ActD_Req in a field, there
   is an additional field to carry ActD_Ack_i. The time for sending a
   response message to path i on HN is set to th_A_i, and the residence
   delay on HN (th_A_i - th_R_j) is used as the initial value of
   ActD_Ack_i;

      c) When CN receives Ack_i, the reception time is recorded as
   tc_A_i.

   2. Calculate the FixD_j for path j based on the information carried
   by Ack_j and the time it arrives at CN. The time tc_A_j - tc_R_j
   consists of: (1) the transmission delay of the request message Req_j
   from CN to HN; (2) processing delay at HN; (3) transmission delay of
   the response message Ack_j from HN to CN. Ack_j carries the
   accumulated residence delay ActD_Req and ActD_Ack_j of the nodes that
   the request and response messages pass through. Assuming the round-
   trip link delay is approximately equal, then:

      FixD_j= [(tc_A_j - tc_R_j) - (ActD_Req_j + ActD_Ack_j)]/2;

   3. Calculate the FixD_i for each path based on the information
   carried by the response messages Ack_i (0<i<n+1, i!=j) on each path,
   the time it arrives at CN, and the calculated FixD_j. Since (tc_A_i -
   tc_R_j) already contains the fixed transmission delay FixD_j of the
   request message, the fixed transmission delay for the other paths is:

      FixD_i = [(tc_A_i - tc_R_j) - (ActD_Req + ActD_Ack_i)] - FixD_j;

   4. Before deploying DetNet flow, calculate PthRefD. Taking two paths
   as an example as shown in Figure 9(b), the transmission delay t_Req_1
   from CN to HN is common and can be ignored. The delay from HN to CN
   on Path_1 is approximately from the time th_R_1 at HN to the time
   tc_A_1 at CN; similarly, the delay from HN to CN on Path_2 is
   approximately from the time th_R_1 at HN to the time tc_A_2 at CN.
   Select the latest arrival time tc_A_2 (approximating the transmission
   delay of the longest path) as the basis, and add the value tr_2,
   which is selected by the user or controller based on engineering
   requirements (including extra link delay, variable delay upper limit,
   and any processing delay before CN scheduling), as the reference
   point tr. Here, PthRefD is the time from th_R_1 at HN to the time tr
   at CN. Without time synchronization, direct subtraction cannot obtain
   the path delay. However, it can be indirectly obtained. Assuming that
   the response message via Path_2 arrives the latest, then:




Guo, et al.             Expires June 11, 2024                [Page 23]

Internet-Draft     Compensation Reference for JRM        December 2023


      PthRefD = FixD_2 + ActD_Ack_2 + Tr_2;

   5. Calculate the compensation reference delay value:

   Cap_i = PthRefD - FixD_i;

   6. Update Cap_i into CN or HN for forwarding compensation.

4.3.2. Data Plane

   To collect the compensation reference delay for AN by round-trip
   method, the data plane behaviors of each node are described as
   follows:

   1. CN's data plane receives the compensation reference delay
   collection message and converts it into an OAM packet.

   2. CN sends the request OAM packet for compensation reference delay
   collection to HN, with the path specified by the control plane. CN
   records the transmission time and forwards it to the control plane.

   3. Each node collects the residence delay of the request message
   passing through the path(if the node is SD's I-GW or E-GW, this is
   done by the SD's I-GW and E-GW). With coordination from the control
   plane, necessary identification information for identifying the path
   is generated. Similar requirements are specified in [I-D.ietf-detnet-
   pof]. When a CN node receives an OAM packet or an app-flow packet, it
   needs to obtain path-associated information.

   4. HN records the reception delay, constructs several response-type
   OAM messages, which include the residence delay of the request OAM
   message and the residence delay of the response OAM message
   (initially the residence delay on the HN), and sends them to the CN
   through different paths planned by the control plane.

   5. Each node collects the residence delay of the response messages
   passing through the path(if the node is SD's I-GW or E-GW, this is
   done by the SD's I-GW and E-GW). With coordination from the control
   plane, necessary identification information for identifying the path
   is generated.

   6. At each node implementing PEF on the HN->CN path, when identifying
   OAM for collecting compensation reference delay, the node does not
   eliminate duplicate copies and directly forwards the message.

   7. At the CN node, the reception time of each response message is
   recorded and sent to the control plane.


Guo, et al.             Expires June 11, 2024                [Page 24]

Internet-Draft     Compensation Reference for JRM        December 2023


4.3.3. Control Plane

   The control plane configures gateway identifiers, path information,
   and related operations to the nodes through signaling negotiation or
   controller configuration.

   For the collection of compensation reference delay for AN, the
   control plane needs to coordinate with the data plane to accomplish
   the following process:

   1. Plan network paths and identify the network as SN or AN.

   2. Configure the necessary information for generating path
   identifiers to each node. Similar to the requirements in [I-D.ietf-
   detnet-pof], HN nodes and CN nodes need to obtain path-associated
   information when receiving OAM messages or service data packets.

   3. Configure path identifier information on CN (for response-type
   OAM) and HN (for request-type OAM) for identifying the OAM messages
   transmitted along different paths and forwarding service data
   packets.

   4. Set a timeout mechanism. Reference can be made to the historical
   window mechanism on TSN in [IEEE 802.1 CB].

   5. Define and configure operations for collecting residence delays at
   nodes (including SD's I-GW and E-GW) along the path.

   6. Conduct integrity checks in the control plane.

   7. Allocate OAM message sequence numbers and send messages to CN to
   send compensation reference delay collection OAM messages.

   8. Upon receiving an OAM message, the CN control plane cooperates
   with the data plane to complete the measurement and calculate the
   reference delay. For each copy of the same OAM message received by
   the CN control plane, the maximum residence delay and fixed
   transmission delay within each node are calculated, based on which
   the compensation reference delay for each path is determined as
   described in section 4.3.1.

   9. Configure the compensation reference delay value to HN or CN.

   10. Runtime maintenance: periodically collect and calculate the full
   path reference delay and update the reference delay to CN or HN.




Guo, et al.             Expires June 11, 2024                [Page 25]

Internet-Draft     Compensation Reference for JRM        December 2023


   For the delay collection in each domain, the control plane
   coordinates to provide the relevant information required by the data
   plane and determines the providing method. The control plane can
   provide in the following ways:

   Method one: The control plane globally allocates node identifiers and
   distributes this identifier to each node (including SD's I-GW and E-
   GW). The control plane sends down the required information for delay
   collection to HN.

   Method two: The control plane configures operations on a per-flow
   basis at the intervening nodes (including SD's I-GW and E-GW).





5. Security Considerations

   TBD

6. IANA Considerations

   TBD

7. Acknowledgments

   TBD

8. Contributors

   The editor wishes to thank and acknowledge the following contributors
   for contributing text to this document.















Guo, et al.             Expires June 11, 2024                [Page 26]

Internet-Draft     Compensation Reference for JRM        December 2023


   Shenchao Xu
   New H3C Technologies Co., Ltd
   310052
   Email: xushenchao@h3c.com

   Ning Pan
   New H3C Technologies Co., Ltd
   100094
   Email: panning@h3c.com

   Xusheng Chen
   New H3C Technologies Co., Ltd
   100094
   Email: cxs@h3c.com

   Wei Wang
   New H3C Technologies Co., Ltd
   100094
   Email: david_wang@h3c.com

   Xinmin Liu
   New H3C Technologies Co., Ltd
   100094
   Email: liuxinmin@h3c.com

9. References

9.1. Informative References

   [I-D.guo-detnet-jitter-reduction-mechanism]

             Guo, D., Xu, S., "Jitter Reduction Mechanism for
             DetNet",Work in Progress, Internet-Draft, draft-guo-detnet-
             jitter-reduction-mechanism-01, 23 October 2023,    <
             https://datatracker.ietf.org/doc/draft-guo-detnet-jitter-
             reduction-mechanism/>.

   [I-D.ietf-detnet-scaling-requirements]

             Liu, P., Li, Y., Eckert, T., Xiong, Q., and J. Ryoo,
             "Requirements for Scaling Deterministic Networks",Work in
             Progress, Internet-Draft, draft-ietf-detnet-scaling-
             requirements-05, November 2023,
             <https://datatracker.ietf.org/doc/draft-ietf-detnet-
             scaling-requirements/>.

   [I-D.ietf-detnet-mpls-over-ip-preof]


Guo, et al.             Expires June 11, 2024                [Page 27]

Internet-Draft     Compensation Reference for JRM        December 2023


             Varga, B., Farkas, J., and A. G. Malis, "Deterministic
             Networking (DetNet): DetNet PREOF via MPLS over
             UDP/IP",Work in Progress, Internet-Draft, draft-ietf-
             detnet-mpls-over-ip-preof-07, 25 July 2023,
             <https://datatracker.ietf.org/doc/html/draft-ietf-detnet-
             mpls-over-ip-preof-07>.

   [I-D.ietf-detnet-pof]

             Varga, B., Farkas, J., Kehrer, S., Heer, T., "Deterministic
             Networking (DetNet): Packet Ordering Function", Work in
             Progress, Internet-Draft, draft-ietf-detnet-pof-06, 8
             November 2023, <https://datatracker.ietf.org/doc/draft-
             ietf-detnet-pof/>.

   [RFC9016]

             Varga, B., Farkas, J., Cummings, R., Jiang, Y., Fedyk, D.,
             "Flow and Service Information Model for Deterministic
             Networking (DetNet)", RFC 8964, DOI 10.17487/RFC9016, March
             2021, <https://www.rfc-editor.org/info/rfc9016>.

   [IEEE802.1CB]

             IEEE,"IEEE Standard for Local and metropolitan area
             networks-Frame Replication and Elimination for
             Reliability", IEEE Std 802.1CB-2017 (2017), 1--102.
             <https://doi.org/10.1109/IEEESTD.2017.8091139>.




















Guo, et al.             Expires June 11, 2024                [Page 28]

Internet-Draft     Compensation Reference for JRM        December 2023


Authors' Addresses

   Daorong Guo
   New H3C Technologies Co., Ltd
   Beijing
   100094
   China
   Email: guodaorong@h3c.com

   XUEJUN YOU
   New H3C Technologies Co., Ltd
   Beijing
   100094
   China
   Email: yoe@h3c.com

   Rubing Liu
   New H3C Technologies Co., Ltd
   Hangzhou
   310052
   China
   Email: liurubing@h3c.com

   Shiyin Zhu
   New H3C Technologies Co., Ltd
   Beijing
   100094
   China
   Email: zhushiyin@h3c.com



















Guo, et al.             Expires June 11, 2024                [Page 29]