Internet DRAFT - draft-schmutzer-pals-ple
draft-schmutzer-pals-ple
Network Working Group S. Gringeri
Internet-Draft J. Whittaker
Intended status: Standards Track Verizon
Expires: 17 November 2023 N. Leymann
Deutsche Telekom
C. Schmutzer, Ed.
Cisco Systems, Inc.
C. Brown
Ciena Corporation
16 May 2023
Private Line Emulation over Packet Switched Networks
draft-schmutzer-pals-ple-04
Abstract
This document describes a method for encapsulating high-speed bit-
streams as virtual private wire services (VPWS) over packet switched
networks (PSN) providing complete signal transport transparency.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 17 November 2023.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
Gringeri, et al. Expires 17 November 2023 [Page 1]
Internet-Draft PLE May 2023
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction and Motivation . . . . . . . . . . . . . . . . . 3
2. Requirements Notation . . . . . . . . . . . . . . . . . . . . 3
3. Terminology and Reference Model . . . . . . . . . . . . . . . 3
3.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4
3.2. Reference Models . . . . . . . . . . . . . . . . . . . . 5
4. Emulated Services . . . . . . . . . . . . . . . . . . . . . . 7
4.1. Generic PLE Service . . . . . . . . . . . . . . . . . . . 7
4.2. Ethernet services . . . . . . . . . . . . . . . . . . . . 7
4.2.1. 10GBASE-R and 25GBASE-R . . . . . . . . . . . . . . . 8
4.2.2. 40GBASE-R, 50GBASE-R and 100GBASE-R . . . . . . . . . 9
4.2.3. 200GBASE-R and 400GBASE-R . . . . . . . . . . . . . . 10
4.2.4. Energy Efficient Ethernet (EEE) . . . . . . . . . . . 11
4.3. SONET/SDH Services . . . . . . . . . . . . . . . . . . . 12
4.4. Fibre Channel Services . . . . . . . . . . . . . . . . . 12
4.4.1. 1GFC, 2GFC, 4GFC and 8GFC . . . . . . . . . . . . . . 13
4.4.2. 16GFC and 32GFC . . . . . . . . . . . . . . . . . . . 13
4.4.3. 64GFC and 4-lane 128GFC . . . . . . . . . . . . . . . 14
4.5. ODUk OTN Services . . . . . . . . . . . . . . . . . . . . 15
5. PLE Encapsulation Layer . . . . . . . . . . . . . . . . . . . 16
5.1. PSN and VPWS Demultiplexing Headers . . . . . . . . . . . 16
5.2. PLE Header . . . . . . . . . . . . . . . . . . . . . . . 17
5.2.1. PLE Control Word . . . . . . . . . . . . . . . . . . 17
5.2.2. RTP Header . . . . . . . . . . . . . . . . . . . . . 18
6. PLE Payload Layer . . . . . . . . . . . . . . . . . . . . . . 20
6.1. Basic Payload . . . . . . . . . . . . . . . . . . . . . . 20
6.2. Byte aligned Payload . . . . . . . . . . . . . . . . . . 20
7. PLE Operation . . . . . . . . . . . . . . . . . . . . . . . . 20
7.1. Common Considerations . . . . . . . . . . . . . . . . . . 20
7.2. PLE IWF Operation . . . . . . . . . . . . . . . . . . . . 20
7.2.1. PSN-bound Encapsulation Behavior . . . . . . . . . . 21
7.2.2. CE-bound Decapsulation Behavior . . . . . . . . . . . 21
7.3. PLE Performance Monitoring . . . . . . . . . . . . . . . 23
8. QoS and Congestion Control . . . . . . . . . . . . . . . . . 23
9. Security Considerations . . . . . . . . . . . . . . . . . . . 24
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 24
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 24
12. References . . . . . . . . . . . . . . . . . . . . . . . . . 24
12.1. Normative References . . . . . . . . . . . . . . . . . . 24
12.2. Informative References . . . . . . . . . . . . . . . . . 26
Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28
Gringeri, et al. Expires 17 November 2023 [Page 2]
Internet-Draft PLE May 2023
1. Introduction and Motivation
This document describes a method called Private Line Emulation (PLE)
for encapsulating high-speed bit-streams as Virtual Private Wire
Service (VPWS) over Packet Switched Networks (PSN). This emulation
suits applications where signal transparency is required and data or
framing structure interpretation of the PE would be counter
productive.
One example is two ethernet connected CEs and the need for
synchronous ethernet operation between them without the intermediate
PEs interfering or addressing concerns about ethernet control
protocol transparency for carrier ethernet services, beyond the
behavior definitions of MEF specifications.
Another example would be a Storage Area Networking (SAN) extension
between two data centers. Operating at a bit-stream level allows for
a connection between Fibre Channel switches without interfering with
any of the Fibre Channel protocol mechanisms.
Also SONET/SDH add/drop multiplexers or cross-connects can be
interconnected without interfering with the multiplexing structures
and networks mechanisms. This is a key distinction to CEP defined in
[RFC4842] where demultiplexing and multiplexing is desired in order
to operate per SONET Synchronous Payload Envelope (SPE) and Virtual
Tributary (VT) or SDH Virtual Container (VC). Said in another way,
PLE does provide an independent layer network underneath the SONET/
SDH layer network, whereas CEP does operate at the same level and
peer with the SONET/SDH layer network.
The mechanisms described in this document follow principals similar
to [RFC4553] but expanding the applicability beyond the narrow set of
PDH interfaces (T1, E1, T3 and E3) and allow the transport of signals
from many different technologies such as Ethernet, Fibre Channel,
SONET/SDH [GR253]/[G.707] and OTN [G.709] at gigabit speeds by
treating them as bit-stream payload defined in sections 3.3.3 and
3.3.4 of [RFC3985].
2. Requirements Notation
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
3. Terminology and Reference Model
Gringeri, et al. Expires 17 November 2023 [Page 3]
Internet-Draft PLE May 2023
3.1. Terminology
* ACH - Associated Channel Header
* AIS - Alarm Indication Signal
* CBR - Constant Bit Rate
* CE - Customer Edge
* CSRC - Contributing SouRCe
* ES - Errored Second
* FEC - Forward Error Correction
* IWF - InterWorking Function
* LDP - Label Distribution Protocol
* LF - Local Fault
* MPLS - Multi Protocol Label Switching
* NSP - Native Service Processor
* ODUk - Optical Data Unit k
* OTN - Optical Transport Network
* OTUk - Optical Transport Unit k
* PCS - Physical Coding Sublayer
* PE - Provider Edge
* PLE - Private Line Emulation
* PLOS - Packet Loss Of Signal
* PSN - Packet Switched Network
* P2P - Point-to-Point
* QOS - Quality Of Service
* RSVP-TE - Resource Reservation Protocol Traffic Engineering
Gringeri, et al. Expires 17 November 2023 [Page 4]
Internet-Draft PLE May 2023
* RTCP - RTP Control Protocol
* RTP - Realtime Transport Protocol
* SAN - Storage Area Network
* SES - Severely Errored Seconds
* SDH - Synchronous Digital Hierarchy
* SPE - Synchronous Payload Envelope
* SRTP - Secure Realtime Transport Protocol
* SRv6 - Segment Routing over IPv6 Dataplane
* SSRC - Synchronization SouRCe
* SONET - Synchronous Optical Network
* TCP - Transmission Control Protocol
* UAS - Unavailable Seconds
* VPWS - Virtual Private Wire Service
* VC - Virtual Circuit
* VT - Virtual Tributary
Similar to [RFC4553] and [RFC5086] the term Interworking Function
(IWF) is used to describe the functional block that encapsulates bit
streams into PLE packets and in the reverse direction decapsulates
PLE packets and reconstructs bit streams.
3.2. Reference Models
The generic reference model defined in Section 4.1 of [RFC4664] and
Section 4.1 of [RFC3985] does apply to PLE. Further the model
defined in Section 4.2 of [RFC3985] and in particular the concept of
a Native Service Processing (NSP) function defined in Section 4.2.2
of [RFC3985] does apply to PLE as well. The resulting reference
model for PLE is illustrated in Figure 1
Gringeri, et al. Expires 17 November 2023 [Page 5]
Internet-Draft PLE May 2023
|<--- p2p L2VPN service -->|
| |
| |<-PSN tunnel->| |
v v v v
+---------+ +---------+
| PE1 |==============| PE2 |
+---+-----+ +-----+---+
+-----+ | N | | | | N | +-----+
| CE1 |-----| S | IWF |.....VPWS.....| IWF | S |-----| CE2 |
+-----+ ^ | P | | | | P | ^ +-----+
| +---+-----+ +-----+---+ |
CE1 physical ^ ^ CE2 physical
interface | | interface
|<--- emulated service --->|
| |
attachment attachment
circuit circuit
Figure 1: PLE Reference Model
PLE embraces the minimum intervention principle outlined in
Section 3.3.5 of [RFC3985] whereas the data is flowing through the
PLE encapsulation layer as received without modifications.
For some service types the NSP function is responsible for performing
operations on the native data received from the CE. Examples are
terminating Forward Error Correction (FEC) or terminating the OTUk
layer for OTN. After the NSP the IWF is generating the payload of
the VPWS which is carried via a PSN tunnel.
J
| G
v |
+-----+ +-----+ v
+-----+ |- - -|=================|- - -| +-----+
| |<---------|.............................|<---------| |
| CE1 | | PE1 | VPWS | PE2 | | CE2 |
| |--------->|.............................|--------->| |
+-----+ |- - -|=================|- - -| +-----+
^ +-----+<-------+------->+-----+
| | ^
A +-+ |
|I| E
+-+
Figure 2: Relative Network Scenario Timing
Gringeri, et al. Expires 17 November 2023 [Page 6]
Internet-Draft PLE May 2023
To allow the clock of the transported signal to be carried across the
PLE domain in a transparent way the network synchronization reference
model and deployment scenario outlined in Section 4.3.2 of [RFC4197]
is applicable.
As illustrated in Figure 2, the attachment circuit clock E is
generated by PE2 via a differential clock recovery method in
reference to a common clock I. For this to work the difference
between clock I and clock A MUST be explicitly transferred between
the PE1 and PE2 using the timestamp inside the RTP header.
For the reverse direction PE1 does generate the clock J in reference
to clock I and the clock difference between I and G.
The way the common clock I is implemented is out of scope of this
document. Well established concepts for achieving frequency
synchronization in a PSN have already been defined in [G.8261] and
can be applied here as well.
4. Emulated Services
This specification does describe the emulation of services from a
wide range of technologies such as TDM, ethernet, fibre channel or
OTN as bit stream or structured bit stream as defined in
Section 3.3.3 of [RFC3985] and Section 3.3.4 of [RFC3985].
4.1. Generic PLE Service
The generic PLE service is an example of the bit stream defined in
Section 3.3.3 of [RFC3985].
Under the assumption that the CE-bound IWF is not responsible for any
service specific operation, a bit stream of any rate can be carried
using the generic PLE payload.
There is no NSP function present for this service.
4.2. Ethernet services
Ethernet services are special cases of the structured bit stream
defined in Section 3.3.4 of [RFC3985].
IEEE has defined several layers for ethernet in [IEEE802.3].
Emulation is operating at the physical (PHY) layer, more precisely at
the Physical Subcoding Layer (PCS).
Gringeri, et al. Expires 17 November 2023 [Page 7]
Internet-Draft PLE May 2023
Over time many differnet ethernet interface types have been specified
in [IEEE802.3] with a varying set of characteristcs such as optional
vs mandatory FEC and single-lane vs multi-lane transmission.
All ethernet services are leverging the basic PLE payload and
interface specific mechanisms are confined to the respective service
specific NSP functions.
4.2.1. 10GBASE-R and 25GBASE-R
The PCS layers of 10GBASE-R defined in clause 49 and 25GBASE-R
defined in clause 107 of [IEEE802.3] are based on a 64B/66B code.
[IEEE802.3] clauses 74 and 108 do define an optional FEC layer, if
present the PSN-bound NSP function MUST terminate the FEC and the CE-
bound NSP function must generate the FEC.
The PSN-bound NSP function is also responsible to detect 10GBASE-R
and 25GBASE-R specific attachment circuit faults such as LOS and sync
loss.
The PSN-bound IWF is mapping the scrambled 64B/66B code stream into
the basic PLE payload.
The CE-bound NSP function MUST perform
* PCS code sync
* descrambling
in order to properly
* transform invalid 66B code blocks into proper error control
characters /E/
* insert Local Fault (LF) ordered sets when the CE-bound IWF is in
PLOS state or when PLE packets are received with the L-bit being
set
Note: Invalid 66B code blocks typically are a consequence of the CE-
bound IWF inserting replacement data in case of lost PLE packets, or
if the farend PSN-bound NSP function did set sync headers to 11 due
to uncorrectable FEC errors.
Before sending the bit stream to the CE, the CE-bound NSP function
MUST also scramble the 64B/66B code stream.
Gringeri, et al. Expires 17 November 2023 [Page 8]
Internet-Draft PLE May 2023
4.2.2. 40GBASE-R, 50GBASE-R and 100GBASE-R
The PCS layers of 40GBASE-R and 100GBASE-R defined in clause 82 and
of 50GBASE-R defined in clause 133 of [IEEE802.3] are based on a
64B/66B code transmitted over multiple lanes.
[IEEE802.3] clauses 74 and 91 do define an optional FEC layer, if
present the PSN-bound NSP function MUST terminate the FEC and the CE-
bound NSP function must generate the FEC.
To gain access to the scrambled 64B/66B code stream the PSN-bound NSP
further MUST perform
* block synchronization
* PCS lane deskew
* PCS lane reordering
The PSN-bound NSP function is also responsible to detect 40GBASE-R,
50GBASE-R and 100GBASE-R specific attachment circuit faults such as
LOS and loss of alignment.
The PSN-bound IWF is mapping the serialized, scrambled 64B/66B code
stream including the alignment markers into the basic PLE payload.
The CE-bound NSP function MUST perform
* PCS code sync
* alignment marker removal
* descrambling
in order to properly
* transform invalid 66B code blocks into proper error control
characters /E/
* insert Local Fault (LF) ordered sets when the CE-bound IWF is in
PLOS state or when PLE packets are received with the L-bit being
set
Note: Invalid 66B code blocks typically are a consequence of the CE-
bound IWF inserting replacement data in case of lost PLE packets, or
if the farend PSN-bound NSP function did set sync headers to 11 due
to uncorrectable FEC errors.
Gringeri, et al. Expires 17 November 2023 [Page 9]
Internet-Draft PLE May 2023
When sending the bit stream to the CE, the CE-bound NSP function MUST
also perform
* scrambling of the 64B/66B code
* block distribution
* alignment marker insertion
4.2.3. 200GBASE-R and 400GBASE-R
The PCS layers of 200GBASE-R and 400GBASE-R defined in clause 119 of
[IEEE802.3] are based on a 64B/66B code transcoded to a 256B/257B
code to reduce the overhead and make room for a mandatory FEC.
To gain access to the 64B/66B code stream the PSN-bound NSP further
MUST perform
* alignment lock and deskew
* PCS Lane reordering and de-interleaving
* FEC decoding
* post-FEC interleaving
* alignment marker removal
* descrambling
* reverse transcoding from 256B/257B to 64B/66B
Further the PSN-bound NSP MUST perform rate compensation and
scrambling before the PSN-bound IWF is mapping the same into the
basic PLE payload.
Rate compensation is applied so that the rate of the 66B encoded bit
stream carried by PLE is 528/544 times the nominal bitrate of the
200GBASE-R or 400GBASE-R at the PMA service interface. X number of
66 byte long rate compensation blocks are inserted every X*20479
number of 66B client blocks. For 200GBASE-R the value of X is 16 and
for 400GBASE-R the value of X is 32. Rate compensation blocks are
special 66B control characters of type 0x00 that can easily be
searched for by the CE-bound IWF in order to remove them.
The PSN-bound NSP function is also responsible to detect 200GBASE-R
and 400GBASE-R specific attachment circuit faults such as LOS and
loss of alignment.
Gringeri, et al. Expires 17 November 2023 [Page 10]
Internet-Draft PLE May 2023
The CE-bound NSP function MUST perform
* PCS code sync
* descrambling
* rate compensation block removal
in order to properly
* transform invalid 66B code blocks into proper error control
characters /E/
* insert Local Fault (LF) ordered sets when the CE-bound IWF is in
PLOS state or when PLE packets are received with the L-bit being
set
Note: Invalid 66B code blocks typically are a consequence of the CE-
bound IWF inserting replacement data in case of lost PLE packets, or
if the farend PSN-bound NSP function did set sync headers to 11 due
to uncorrectable FEC errors.
When sending the bit stream to the CE, the CE-bound NSP function MUST
also perform
* transcoding from 64B/66B to 256B/257B
* scrambling
* aligment marker insertion
* pre-FEC distribution
* FEC encoding
* PCS Lane distribution
4.2.4. Energy Efficient Ethernet (EEE)
Section 78 of [IEEE802.3] does define the optional Low Power Idle
(LPI) capability for ethernet. Two modes are defined
* deep sleep
* fast wake
Gringeri, et al. Expires 17 November 2023 [Page 11]
Internet-Draft PLE May 2023
Deep sleep mode is not compatible with PLE due to the CE ceasing
transmission. Hence there is no support for LPI for 10GBASE-R
services across PLE.
When in fast wake mode the CE transmits /LI/ control code blocks
instead of /I/ control code blocks and therefore PLE is agnostic to
it. For 25GBASE-R and higher services across PLE, LPI is supported
as only fast wake mode is applicable.
4.3. SONET/SDH Services
SONET/SDH services are special cases of the structured bit stream
defined in Section 3.3.4 of [RFC3985].
SDH interfaces are defined in [G.707] and SONET interfaces are
defined in [GR253].
The PSN-bound NSP function does not modify the received data but is
responsible to detect SONET/SDH interface specific attachment circuit
faults such as LOS, LOF and OOF.
Data received by the PSN-bound IWF is mapped into the basic PLE
payload without any awareness of SONET/SDH frames.
When the CE-bound IWF is in PLOS state or when PLE packets are
received with the L-bit being set, the CE-bound NSP function is
responsible for generating the
* MS-AIS maintenance signal defined in clause 6.2.4.1.1 of [G.707]
for SDH services
* AIS-L maintenance signal defined in clause 6.2.1.2 of [GR253] for
SONET services
at client frame boundaries.
4.4. Fibre Channel Services
Fibre channel services are special cases of the structured bit stream
defined in Section 3.3.4 of [RFC3985].
The T11 technical committee of INCITS has defined several layers for
fibre channel. Emulation is operating at the FC-1 layer.
Over time many different fibre channel interface types have been
specified in FC-PI-x and FC-FS-x standards with a varying set of
characteristcs such as optional vs mandatory FEC and single-lane vs
multi-lane transmission.
Gringeri, et al. Expires 17 November 2023 [Page 12]
Internet-Draft PLE May 2023
All fibre channel services are leveraging the basic PLE payload and
interface specific mechanisms are confined to the respective service
specific NSP functions.
4.4.1. 1GFC, 2GFC, 4GFC and 8GFC
The PSN-bound NSP function is responsible to detect fibre channel
specific attachment circuit faults such as LOS and sync loss.
The PSN-bound IWF is mapping the received 8B/10B code stream as is
into the basic PLE payload.
The CE-bound NSP function MUST perform transmission word sync in
order to properly
* replace invalid transmission words with the special character
K30.7
* insert Not Operational (NOS) ordered sets when the CE-bound IWF is
in PLOS state or when PLE packets are received with the L-bit
being set
Note: Invalid transmission words typically are a consequence of the
CE-bound IWF inserting replacement data in case of lost PLE packets.
FC-FS-2 amendment 1 does define the use of scrambling for 8GFC, in
this case the CE-bound NSP MUST also perform descrambling before
replacing invalid transmission words or inserting NOIS ordered sets.
And before sending the bit stream to the, the CE-bound NSP function
MUST scramble the 8B/10B code stream.
4.4.2. 16GFC and 32GFC
FC-PI-5 amendment 1 defines a optional FEC layer for 16GFC. For
32GFC the FEC layer is, as defined in FC-PI-6, mandatory. If FEC is
present, the PSN-bound NSP function MUST terminate the FEC and the
CE-bound NSP function must generate the FEC.
The PSN-bound NSP function is responsible to detect fibre channel
specific attachment circuit faults such as LOS and sync loss.
The PSN-bound IWF is mapping the received 64B/66B code stream as is
into the basic PLE payload.
The CE-bound NSP function MUST perform
* transmission word sync
Gringeri, et al. Expires 17 November 2023 [Page 13]
Internet-Draft PLE May 2023
* descrambling
in order to properly
* replace invalid transmission words with the error transmission
word 1Eh
* insert Not Operational (NOS) ordered sets when the CE-bound IWF is
in PLOS state or when PLE packets are received with the L-bit
being set
Note: Invalid transmission words typically are a consequence of the
CE-bound IWF inserting replacement data in case of lost PLE packets,
or if the farend PSN-bound NSP function did set sync headers to 11
due to uncorrectable FEC errors.
Before sending the bit stream to the CE, the CE-bound NSP function
MUST also scramble the 64B/66B code stream.
4.4.3. 64GFC and 4-lane 128GFC
Both FC-PI-7 for 64GFC and FC-PI-6P for 4-lane 128GFC define a
mandatory FEC layer. The PSN-bound NSP function MUST terminate the
FEC and the CE-bound NSP function must generate the FEC.
To gain access to the 64B/66B code stream the PSN-bound NSP further
MUST perform
* alignment lock and deskew
* Lane reordering and de-interleaving
* FEC decoding
* post-FEC interleaving
* alignment marker removal
* descrambling
* reverse transcoding from 256B/257B to 64B/66B
Further the PSN-bound NSP MUST perform scrambling before the PSN-
bound IWF is mapping the same into the basic PLE payload.
Note : the use of rate compensation is for further study.
Gringeri, et al. Expires 17 November 2023 [Page 14]
Internet-Draft PLE May 2023
The PSN-bound NSP function is also responsible to detect fibre
channel specific attachment circuit faults such as LOS and sync loss.
The CE-bound NSP function MUST perform
* tranmission word sync
* descrambling
in order to properly
* replace invalid transmission words with the error transmission
word 1Eh
* insert Not Operational (NOS) ordered sets when the CE-bound IWF is
in PLOS state or when PLE packets are received with the L-bit
being set
Note: Invalid transmission words typically are a consequence of the
CE-bound IWF inserting replacement data in case of lost PLE packets,
or if the farend PSN-bound NSP function did set sync headers to 11
due to uncorrectable FEC errors.
When sending the bit stream to the CE, the CE-bound NSP function MUST
also perform
* transcoding from 64B/66B to 256B/257B
* scrambling
* aligment marker insertion
* pre-FEC distribution
* FEC encoding
* Lane distribution
4.5. ODUk OTN Services
ODUk services are special cases of the structured bit stream defined
in Section 3.3.4 of [RFC3985].
OTN interfaces are defined in [G.709].
The PSN-bound NSP function MUST terminate the FEC and present to the
IWF an extended ODUk including a valid frame alignment overhead.
Gringeri, et al. Expires 17 November 2023 [Page 15]
Internet-Draft PLE May 2023
The PSN-bound NSP function is also responsible to detect OTUk
specific attachment circuit faults such as LOS, LOF, LOM and AIS.
For ODUk services the PSN-bound IWF MUST use the byte aligned PLE
payload.
The CE-bound NSP function will recover the ODUk by searching for the
frame alignment overhead and generate the FEC.
When the CE-bound IWF is in PLOS state or when PLE packets are
received with the L-bit being set, the CE-bound NSP function is
responsible for generating the ODUk-AIS maintenance signal defined in
clause 16.5.1 of [G.709] at client frame boundaries.
5. PLE Encapsulation Layer
The basic packet format used by PLE is shown in the Figure 3.
+-------------------------------+ -+
| PSN and VPWS Demux | \
| (MPLS/SRv6) | > PSN and VPWS
| | / Demux Headers
+-------------------------------+ -+
| PLE Control Word | \
+-------------------------------+ > PLE Header
| RTP Header | /
+-------------------------------+ --+
| Bit Stream | \
| Payload | > Payload
| | /
+-------------------------------+ --+
Figure 3: PLE Encapsulation Layer
5.1. PSN and VPWS Demultiplexing Headers
This document does not imply any specific technology to be used for
implementing the VPWS demultiplexing and PSN layers.
When a MPLS PSN layer is used. A VPWS label provides the
demultiplexing mechanism as described in Section 5.4.2 of [RFC3985].
The PSN tunnel can be a simple best path Label Switched Path (LSP)
established using LDP [RFC5036] or Segment Routing [RFC8402] or a
traffic engineered LSP established using RSVP-TE [RFC3209] or SR-TE
[RFC9256].
Gringeri, et al. Expires 17 November 2023 [Page 16]
Internet-Draft PLE May 2023
When PLE is applied to a SRv6 based PSN, the mechanisms defined in
[RFC8402] and the End.DX2 endpoint behavior defined in [RFC8986] do
apply.
5.2. PLE Header
The PLE header MUST contain the PLE control word (4 bytes) and MUST
include a fixed size RTP header [RFC3550]. The RTP header MUST
immediately follow the PLE control word.
5.2.1. PLE Control Word
The format of the PLE control word is in line with the guidance in
[RFC4385] and is shown in Figure 4.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0|L|R|RSV|FRG| LEN | Sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: PLE Control Word
The bits 0..3 of the first nibble are set to 0 to differentiate a
control word or Associated Channel Header (ACH) from an IP packet or
ethernet frame. The first nibble MUST be set to 0000b to indicate
that this header is a control word as defined in Section 3 of
[RFC4385].
The other fields in the control word are used as defined below:
* L
Set by the PE to indicate that data carried in the payload is
invalid due to an attachment circuit fault. The downstream PE
MUST play out appropriate replacement data. The NSP MAY inject an
appropriate native fault propagation signal.
* R
Set by the downstream PE to indicate that the IWF experiences
packet loss from the PSN or a server layer backward fault
indication is present in the NSP. The R bit MUST be cleared by
the PE once the packet loss state or fault indication has cleared.
* RSV
Gringeri, et al. Expires 17 November 2023 [Page 17]
Internet-Draft PLE May 2023
These bits are reserved for future use. This field MUST be set to
zero by the sender and ignored by the receiver.
* FRG
These bits MUST be set to zero by the sender and ignored by the
receiver.
* LEN
In accordance to Section 3 of [RFC4385] the length field MUST
always be set to zero as there is no padding added to the PLE
packet. To detect malformed packets the default, preconfigured or
signaled payload size MUST be assumed.
* Sequence number
The sequence number field is used to provide a common PW
sequencing function as well as detection of lost packets. It MUST
be generated in accordance with the rules defined in Section 5.1
of [RFC3550] and MUST be incremented with every PLE packet being
sent.
5.2.2. RTP Header
The RTP header MUST be included and is used for explicit transfer of
timing information. The RTP header is purely a formal reuse and RTP
mechanisms, such as header extensions, contributing source (CSRC)
list, padding, RTP Control Protocol (RTCP), RTP header compression,
Secure Realtime Transport Protocol (SRTP), etc., are not applicable
to PLE VPWS.
The format of the RTP header is as shown in Figure 5.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Synchronization Source (SSRC) Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5: RTP Header
* V: Version
Gringeri, et al. Expires 17 November 2023 [Page 18]
Internet-Draft PLE May 2023
The version field MUST be set to 2.
* P: Padding
The padding flag MUST be set to zero by the sender and ignored by
the receiver.
* X: Header extension
The X bit MUST be set to zero by sender and ignored by receiver.
* CC: CSRC count
The CC field MUST be set to zero by the sender and ignored by the
receiver.
* PT: Payload type
A PT value MUST be allocated from the range of dynamic values
defined by [RFC3551] for each direction of the VPWS. The same PT
value MAY be reused both for direction and between different PLE
VPWS.
* Sequence number
The Sequence number in the RTP header MUST be equal to the
sequence number in the PLE control word. The sequence number of
the RTP header MAY be used to extend the sequence number of the
PLE control word from 16 to 32 bits. If so, the initial value of
the RTP sequence number MUST be 0 and incremented whenever the PLE
control word sequence number cycles through from 0xFFFF to 0x0000.
* Timestamp
Timestamp values are used in accordance with the rules established
in [RFC3550]. For bit-streams up to 200 Gbps the frequency of the
clock used for generating timestamps MUST be 125 MHz based on a
the common clock I. For bit-streams above 200 Gbps the frequency
MUST be 250 MHz.
* SSRC: Synchronization source
The SSRC field MAY be used for detection of misconnections.
Gringeri, et al. Expires 17 November 2023 [Page 19]
Internet-Draft PLE May 2023
6. PLE Payload Layer
A bit-stream is mapped into a PLE packet with a fixed payload size
which MUST be defined during VPWS setup, MUST be the same in both
directions of the VPWS and MUST remain unchanged for the lifetime of
the VPWS.
All PLE implementations MUST be capable of supporting the default
payload size of 1024 bytes.
6.1. Basic Payload
The PLE payload is filled with incoming bits of the bit-stream
starting from the most significant to the least significant bit
without considering any structure of the bit-stream.
6.2. Byte aligned Payload
The PLE payload is filled in a byte aligned manner, where the order
of the payload bytes corresponds to their order on the attachment
circuit. Consecutive bits coming from the attachment circuit fill
each payload byte starting from most significant bit to least
significant.
All PLE implementations MUST support the transport of OTN bit-streams
using the byte aligned payload.
7. PLE Operation
7.1. Common Considerations
A PLE VPWS can be established using manual configuration or
leveraging mechanisms of a signaling protocol.
Furthermore emulation of bit-stream signals using PLE is only
possible when the two attachment circuits of the VPWS are of the same
type (OC192, 10GBASE-R, ODU2, etc) and are using the same PLE payload
type and payload size. This can be ensured via manual configuration
or via a signaling protocol
PLE related control protocol extensions to PWE3 [RFC4447] and EVPN-
VPWS [RFC8214] are out of scope of this document and are described in
[I-D.schmutzer-bess-ple-vpws-signalling].
7.2. PLE IWF Operation
Gringeri, et al. Expires 17 November 2023 [Page 20]
Internet-Draft PLE May 2023
7.2.1. PSN-bound Encapsulation Behavior
After the VPWS is set up, the PSN-bound IWF does perform the
following steps:
* Packetize the data received from the CE is into a fixed size PLE
payloads
* Add PLE control word and RTP header with sequence numbers, flags
and timestamps properly set
* Add the VPWS demultiplexer and PSN headers
* Transmit the resulting packets over the PSN
* Set L bit in the PLE control word whenever attachment circuit
detects a fault
* Set R bit in the PLE control word whenever the local CE-bound IWF
is in packet loss state
7.2.2. CE-bound Decapsulation Behavior
The CE-bound IWF is responsible for removing the PSN and VPWS
demultiplexing headers, PLE control word and RTP header from the
received packet stream and play-out of the bit-stream to the local
attachment circuit.
A de-jitter buffer MUST be implemented where the PLE packets are
stored upon arrival. The size of this buffer SHOULD be locally
configurable to allow accommodation of specific PSN packet delay
variation expected.
The CE-bound IWF SHOULD use the sequence number in the control word
to detect lost and mis-ordered packets. It MAY use the sequence
number in the RTP header for the same purposes.
The payload of a lost packet MUST be replaced with equivalent amount
of replacement data. The contents of the replacement data MAY be
locally configurable. All PLE implementations MUST support
generation of "0xAA" as replacement data. The alternating sequence
of 0s and 1s of the "0xAA" pattern does ensure clock synchronization
is maintained. While playing out the replacement data, the IWF will
apply a holdover mechanism to maintain the clock.
Whenever the VPWS is not operationally up, the CE-bound NSP function
MUST inject the appropriate native downstream fault indication
signal.
Gringeri, et al. Expires 17 November 2023 [Page 21]
Internet-Draft PLE May 2023
Whenever a VPWS comes up, the CE-bound IWF enters the intermediate
state, will start receiving PLE packets and will store them in the
jitter buffer. The CE-bound NSP function will continue to inject the
appropriate native downstream fault indication signal until a pre-
configured amount of payloads is stored in the jitter buffer.
After the pre-configured amount of payload is present in the jitter
buffer the CE-bound IWF transitions to the normal operation state and
the content of the jitter buffer is played out to the CE in
accordance with the required clock. In this state the CE-bound IWF
MUST perform egress clock recovery.
The recovered clock MUST comply with the jitter and wander
requirements applicable to the type of attachment circuit, specified
in:
* [G.825] and [G.823] for SDH
* [GR253] for SONET
* [G.8261] for synchronous ethernet
* [G.8251] for OTN
Whenever the L bit is set in the PLE control word of a received PLE
packet the CE-bound NSP function SHOULD inject the appropriate native
downstream fault indication signal instead of playing out the
payload.
If the CE-bound IWF detects loss of consecutive packets for a pre-
configured amount of time (default is 1 millisecond), it enters
packet loss (PLOS) state and a corresponding defect is declared.
If the CE-bound IWF detects a packet loss ratio (PLR) above a
configurable signal-degrade (SD) threshold for a configurable amount
of consecutive 1-second intervals, it enters the degradation (DEG)
state and a corresponding defect is declared. Possible values for
the SD-PLR threshold are between 1..100% with the default being 15%.
Possible values for consecutive intervals are 2..10 with the default
7.
While either a PLOS or DEG defect is declared the CE-bound NSP
function SHOULD inject the appropriate native downstream fault
indication signal. Also the PSN-bound IWF SHOULD set the R bit in
the PLE control word of every packet transmitted.
Gringeri, et al. Expires 17 November 2023 [Page 22]
Internet-Draft PLE May 2023
The CE-bound IWF does change from the PLOS to normal state after the
pre-configured amount of payload has been received similarly to the
transition from intermediate to normal state.
Whenever the R bit is set in the PLE control word of a received PLE
packet the PLE performance monitoring statistics SHOULD get updated.
7.3. PLE Performance Monitoring
PLE SHOULD provide the following functions to monitor the network
performance to be inline with expectations of transport network
operators.
The near-end performance monitors defined for PLE are as follows:
* ES-PLE : PLE Errored Seconds
* SES-PLE : PLE Severely Errored Seconds
* UAS-PLE : PLE Unavailable Seconds
Each second with at least one packet lost or a PLOS/DEG defect SHALL
be counted as ES-PLE. Each second with a PLR greater than 15% or a
PLOS/DEG defect SHALL be counted as SES-PLE.
UAS-PLE SHALL be counted after configurable number of consecutive
SES-PLE have been observed, and no longer counted after a
configurable number of consecutive seconds without SES-PLE have been
observed. Default value for each is 10 seconds.
Once unavailability is detected, ES and SES counts SHALL be inhibited
up to the point where the unavailability was started. Once
unavailability is removed, ES and SES that occurred along the
clearing period SHALL be added to the ES and SES counts.
A PLE far-end performance monitor is providing insight into the CE-
bound IWF at the far end of the PSN. The statistics are based on the
PLE-RDI indication carried in the PLE control word via the R bit.
The PLE VPWS performance monitors are derived from the definitions in
accordance with [G.826]
8. QoS and Congestion Control
The PSN carrying PLE VPWS may be subject to congestion, but PLE VPWS
representing constant bit-rate (CBR) flows cannot respond to
congestion in a TCP-friendly manner as described in [RFC2913].
Gringeri, et al. Expires 17 November 2023 [Page 23]
Internet-Draft PLE May 2023
Hence the PSN providing connectivity for the PLE VPWS between PE
devices MUST be Diffserv [RFC2475] enabled and MUST provide a per
domain behavior [RFC3086] that guarantees low jitter and low loss.
To achieve the desired per domain behavior PLE VPWS SHOULD be carried
over traffic-engineering paths through the PSN with bandwidth
reservation and admission control applied.
9. Security Considerations
As PLE is leveraging VPWS as transport mechanism the security
considerations described in [RFC7432] and [RFC3985] are applicable.
10. IANA Considerations
Applicable signaling extensions are out of the scope of this
document, hence there are no new requirements from IANA.
11. Acknowledgements
The authors would like to thank Andreas Burk for reviewing this
document and providing useful comments and suggestions.
12. References
12.1. Normative References
[G.707] International Telecommunication Union (ITU), "Network node
interface for the synchronous digital hierarchy (SDH)",
January 2007, <https://www.itu.int/rec/T-REC-G.707>.
[G.709] International Telecommunication Union (ITU), "Interfaces
for the optical transport network", June 2020,
<https://www.itu.int/rec/T-REC-G.709>.
[G.823] International Telecommunication Union (ITU), "The control
of jitter and wander within digital networks which are
based on the 2048 kbit/s hierarchy", March 2000,
<https://www.itu.int/rec/T-REC-G.823>.
[G.825] International Telecommunication Union (ITU), "The control
of jitter and wander within digital networks which are
based on the synchronous digital hierarchy (SDH)", March
2000, <https://www.itu.int/rec/T-REC-G.825>.
Gringeri, et al. Expires 17 November 2023 [Page 24]
Internet-Draft PLE May 2023
[G.8251] International Telecommunication Union (ITU), "The control
of jitter and wander within the optical transport network
(OTN)", November 2022,
<https://www.itu.int/rec/T-REC-G.8251>.
[G.826] International Telecommunication Union (ITU), "End-to-end
error performance parameters and objectives for
international, constant bit-rate digital paths and
connections", December 2002,
<https://www.itu.int/rec/T-REC-G.826>.
[G.8261] International Telecommunication Union (ITU), "Timing and
synchronization aspects in packet networks", August 2019,
<https://www.itu.int/rec/T-REC-G.8261>.
[GR253] Telcordia, "SONET Transport Systems : Common Generic
Criteria", October 2009.
[IEEE802.3]
IEEE, "IEEE Standard for Ethernet", May 2022,
<https://standards.ieee.org/ieee/802.3/10422/>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
and W. Weiss, "An Architecture for Differentiated
Services", RFC 2475, DOI 10.17487/RFC2475, December 1998,
<https://www.rfc-editor.org/rfc/rfc2475>.
[RFC3086] Nichols, K. and B. Carpenter, "Definition of
Differentiated Services Per Domain Behaviors and Rules for
their Specification", RFC 3086, DOI 10.17487/RFC3086,
April 2001, <https://www.rfc-editor.org/rfc/rfc3086>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/rfc/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003,
<https://www.rfc-editor.org/rfc/rfc3551>.
Gringeri, et al. Expires 17 November 2023 [Page 25]
Internet-Draft PLE May 2023
[RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
Edge-to-Edge (PWE3) Architecture", RFC 3985,
DOI 10.17487/RFC3985, March 2005,
<https://www.rfc-editor.org/rfc/rfc3985>.
[RFC4197] Riegel, M., Ed., "Requirements for Edge-to-Edge Emulation
of Time Division Multiplexed (TDM) Circuits over Packet
Switching Networks", RFC 4197, DOI 10.17487/RFC4197,
October 2005, <https://www.rfc-editor.org/rfc/rfc4197>.
[RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson,
"Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for
Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385,
February 2006, <https://www.rfc-editor.org/rfc/rfc4385>.
[RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer
2 Virtual Private Networks (L2VPNs)", RFC 4664,
DOI 10.17487/RFC4664, September 2006,
<https://www.rfc-editor.org/rfc/rfc4664>.
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, <https://www.rfc-editor.org/rfc/rfc7432>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.
12.2. Informative References
[I-D.schmutzer-bess-ple-vpws-signalling]
Gringeri, S., Whittaker, J., Schmutzer, C., and P.
Brissette, "Private Line Emulation VPWS Signalling", Work
in Progress, Internet-Draft, draft-schmutzer-bess-ple-
vpws-signalling-02, 3 May 2021,
<https://datatracker.ietf.org/doc/html/draft-schmutzer-
bess-ple-vpws-signalling-02>.
[RFC1925] Callon, R., "The Twelve Networking Truths", RFC 1925,
DOI 10.17487/RFC1925, April 1996,
<https://www.rfc-editor.org/rfc/rfc1925>.
[RFC2913] Klyne, G., "MIME Content Types in Media Feature
Expressions", RFC 2913, DOI 10.17487/RFC2913, September
2000, <https://www.rfc-editor.org/rfc/rfc2913>.
Gringeri, et al. Expires 17 November 2023 [Page 26]
Internet-Draft PLE May 2023
[RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001,
<https://www.rfc-editor.org/rfc/rfc3209>.
[RFC4447] Martini, L., Ed., Rosen, E., El-Aawar, N., Smith, T., and
G. Heron, "Pseudowire Setup and Maintenance Using the
Label Distribution Protocol (LDP)", RFC 4447,
DOI 10.17487/RFC4447, April 2006,
<https://www.rfc-editor.org/rfc/rfc4447>.
[RFC4553] Vainshtein, A., Ed. and YJ. Stein, Ed., "Structure-
Agnostic Time Division Multiplexing (TDM) over Packet
(SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006,
<https://www.rfc-editor.org/rfc/rfc4553>.
[RFC4842] Malis, A., Pate, P., Cohen, R., Ed., and D. Zelig,
"Synchronous Optical Network/Synchronous Digital Hierarchy
(SONET/SDH) Circuit Emulation over Packet (CEP)",
RFC 4842, DOI 10.17487/RFC4842, April 2007,
<https://www.rfc-editor.org/rfc/rfc4842>.
[RFC5036] Andersson, L., Ed., Minei, I., Ed., and B. Thomas, Ed.,
"LDP Specification", RFC 5036, DOI 10.17487/RFC5036,
October 2007, <https://www.rfc-editor.org/rfc/rfc5036>.
[RFC5086] Vainshtein, A., Ed., Sasson, I., Metz, E., Frost, T., and
P. Pate, "Structure-Aware Time Division Multiplexed (TDM)
Circuit Emulation Service over Packet Switched Network
(CESoPSN)", RFC 5086, DOI 10.17487/RFC5086, December 2007,
<https://www.rfc-editor.org/rfc/rfc5086>.
[RFC8214] Boutros, S., Sajassi, A., Salam, S., Drake, J., and J.
Rabadan, "Virtual Private Wire Service Support in Ethernet
VPN", RFC 8214, DOI 10.17487/RFC8214, August 2017,
<https://www.rfc-editor.org/rfc/rfc8214>.
[RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L.,
Decraene, B., Litkowski, S., and R. Shakir, "Segment
Routing Architecture", RFC 8402, DOI 10.17487/RFC8402,
July 2018, <https://www.rfc-editor.org/rfc/rfc8402>.
[RFC8986] Filsfils, C., Ed., Camarillo, P., Ed., Leddy, J., Voyer,
D., Matsushima, S., and Z. Li, "Segment Routing over IPv6
(SRv6) Network Programming", RFC 8986,
DOI 10.17487/RFC8986, February 2021,
<https://www.rfc-editor.org/rfc/rfc8986>.
Gringeri, et al. Expires 17 November 2023 [Page 27]
Internet-Draft PLE May 2023
[RFC9256] Filsfils, C., Talaulikar, K., Ed., Voyer, D., Bogdanov,
A., and P. Mattes, "Segment Routing Policy Architecture",
RFC 9256, DOI 10.17487/RFC9256, July 2022,
<https://www.rfc-editor.org/rfc/rfc9256>.
Contributors
Faisal Dada
Xilinx
Email: faisald@xilinx.com
Gerald Smallegange
Ciena Corporation
Email: gsmalleg@ciena.com
Luca Della Chiesa
Cisco Systems, Inc.
Email: ldellach@cisco.com
Nagendra Kumar Nainar
Cisco Systems, Inc.
Email: naikumar@cisco.com
Carlos Pignataro
Cisco Systems, Inc.
Email: cpignata@cisco.com
Authors' Addresses
Steven Gringeri
Verizon
Email: steven.gringeri@verizon.com
Jeremy Whittaker
Verizon
Email: jeremy.whittaker@verizon.com
Nicolai Leymann
Deutsche Telekom
Email: N.Leymann@telekom.de
Gringeri, et al. Expires 17 November 2023 [Page 28]
Internet-Draft PLE May 2023
Christian Schmutzer (editor)
Cisco Systems, Inc.
Email: cschmutz@cisco.com
Chris Brown
Ciena Corporation
Email: cbrown@ciena.com
Gringeri, et al. Expires 17 November 2023 [Page 29]
