| Common Control and Measurment Plane | I. Hussain, Ed. |
| Internet-Draft | R. Valiveti |
| Intended status: Informational | Infinera Corp |
| Expires: August 18, 2017 | Q. Wang, Ed. |
| ZTE | |
| L. Andersson, Ed. | |
| M. Chen | |
| H. Zheng | |
| Huawei | |
| February 14, 2017 |
GMPLS Routing and Signaling Framework for Flexible Ethernet (FlexE)
draft-izh-ccamp-flexe-fwk-01
Traditionally, Ethernet MAC rates were constrained to match the rates of the Ethernet PHY(s). OIF's implementation agreement [OIFMLG3] was the first step in allowing MAC rates to be different than the PHY rates. OIF has recently approved another implementation agreement [OIFFLEXE1] which allows complete decoupling of the MAC data rates and the Ethernet PHY(s) that support them. This includes support for (a) MAC rates which are greater than the rate of a single PHY (satisfied by bonding of multiple PHY(s)), (b) MAC rates which are less than the rate of a PHY (sub-rate), (c) support of multiple FlexE client signals carried over a single PHY, or over a collection of bonded PHY(s). The FlexE Shim functions which bond multipe Ethernet PHY(s) to realize a large "pipe" view of the connectivity between two FlexE aware devices as a collection of multiple point-to-point links (one link per Ethernet PHY). These logical point-to-point links can either be direct links (without an intervening transport network), or realized via an Optical transport network. This document describes usecases that reflect FlexE deployment scenarios and specifies the technical requirements for extensions to GMPLS control plane protocols to support these scenarios.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 18, 2017.
Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Traditionally, Ethernet MAC rates were constrained to match the rates of the Ethernet PHY(s). OIF's implementation agreement [OIFMLG3] was the first step in allowing MAC rates to be different than the PHY rates standardized by IEEE. OIF has recently approved another implementation agreement [OIFFLEXE1] which allows complete decoupling of the MAC data rates and the Ethernet PHY(s) that support them. This includes support for (a) MAC rates which are greater than the rate of a single PHY (satisfied by bonding of multiple PHY(s)), (b) MAC rates which are less than the rate of a PHY (sub-rate), (c) support of multiple FlexE client signals carried over a single PHY, or over a collection of bonded PHY(s). The capabilities supported by the OIF FlexE implementation agreement version 1.0 are:
All networks which support the bonding of Ehernet interfaces (as per [OIFFLEXE1]) include a basic building block -- which consists of two FlexE Shim functions (located at opposite ends of a link) and the (logical) point to point links that carry the Ethernet PHY signals between the two FlexE Shim Functions. These logical point-to-point PHY links can be realized in a variety of ways:
This Internet-draft examines the usescases that arise when the logical links between FlexE capable devices are (a) point-to-point links without any intervening network (b) realized via Optical transport networks. This draft considers the variants in which fhe two peer FlexE devices are both customer-edge devices, or customer-edge/provider edge devices. This list of usecases will help identify the Control Plane (i.e. Routing and Signaling) extensions that may be required.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
The FlexE Shim layer in a router maps the FlexE client(s) over the FlexE group. The transport network is unware of the FlexE. Each of the FlexE group PHY is carried independently across the transport network over the same fiber route. The FlexE Shim in the router tolerates end-to-end skew across the network. In this usecase, the router makes flexible use of the full capacity of the FlexE group, and depends on legacy transport equipment to realize PCS-codeword-transparent transport of 100GbE. It allows striping of PHYs in the FlexE group over multiple line cards in the transport equipment. It is worth mentioning that in this case, the FlexE Shim layer is terminated at the routers, and the coordination of operations related to FlexE clients, e.g. creating new FlexE clients, deleting existing FlexE clients, and resizing the bandwidth of existing FlexE clients (if desired) happens between the two routers. Note that the transport network is completely transparent to the FlexE signals, and doesn't participate in any FlexE protocols.
==================================================================
+ FlexE Ethernet Client(s) +
+-----------------------------------------------------------+
+ +
+ FlexE skew tolerance
+----------------------------------------+
+ for end-to-distance +
+-----------+ 2x100GE +---------+ +----------+ +------------+
| | | | | | | |
| Router1 | | | | | | |
|FlexE Shim +---------+ A-end | | Z-end +-----+Router 2 |
| | | (FlexE | | (FlexE | |(FlexE Shim)|
| +---^-----+ unaware)| | unaware)+-----+ |
| | | | | | | | |
| | | | | | | | |
+-----------+ + +---------+ +----------+ +------------+
FlexE Group
\----------Transport----------/
network
+--------------+ +----------------+
| FlexE Clients| | FlexE Client(s)|
+--------------+ +----------------+
| FlexE Shim | | FlexE Shim |
+----+----+----+ +----+------+----+
|PHY | | PHY | | PHY | | PHY |
+---+---+--+---+ +---+--+ +--+--+
| | +-----+ +-----+ | |
| +----------+ PHY | | PHY |-------+ |
| +-----+ +-----+ |
| | ODU4+-----------+ ODU4| |
| +-----+ +-----+ |
| |
| +-----+ +-----+ |
+-----------------+ PHY | | PHY +-----------------+
+-----+ +-----+
| ODU4+-----------+ ODU4|
+-----+ +-----+
==================================================================
Figure 1: FlexE unaware transport
This scenario represents an optimization of the FlexE unaware transport presented in Section 3.1, and illustrated in Figure 1. In this application (see Figure 2), the devices at the edge of the transport network do not terminate the FlexE shim layer, but are aware of the (a) composition of the FlexE group (i.e. set of all contained Ethernet PHYs) and (b) format of the FlexE overhead. At the ingress to the transport network, the transport network edge removes the unavailable calendar slots, and retains all available calendar slots (whether they are allocated or not). At the egress point of the transport network, the edge device adds the unavailable calendar slots back. The result is that the FlexE Shim layers at both routers see exactly the same input that they saw in the FlexE unware scenario -- with the added benefit that the line (or DWDM) side bandwidth has been optimized to be sufficient to carry only the available calendar slots in all of the Ethernet PHY(s) in the FlexE group.
The transport network edge device could learn of the set of unavailable calendar slots in a variety of ways; a few examples are listed below:
Note that the process of removing unavailable calendar slots from a FlexE PHY is called "crunching" (see [OIFFLEXE1]). The following additional notes apply to Figure 2:
[N1]RV: The figure may need further editing to accurately depict the signal hierarchy.
================================================================
FlexE Ethernet Client(s) +-----------------------------------------------------+ FlexE skew tolerance +---------------------------------------------+ for end+to+distance +--------+ 2 x 100GE +---------+ +---------+ +------+ | R1 | | | | +----+ R2 | | (FlexE+-----------+ NE A | | NE Z | |(FlexE| | Shim) | | (FlexE | | (FlexE +----+ Shim | | +-----^-----+ aware) | | aware) | | | | | | | | | | | | +--------+ + +---------+ +---------+ +------+ FlexE Group \+--------+Transport+--------+/ network +-------------+ +-------------+ |FlexE clients| |FlexE clients| +-------------+ +-------------+ | FlexE Shim | | FlexE Shim | +-------------+ +-------------+ | PHY | PHY | | PHY | PHY | +-------------+ +-------------+ | | | | | | +-------------+ +------------+ | | | | | FlexE-psg | | FlexE-psg | | | | | +-------------+ +------------+ | | | +--+ PHY|ODUflex +------- |ODUFlex|PHY +--+ | | +-------------+ +------------+ | | | | +-------------+ +------------+ | | | FlexE|psg | | FlexE|psg | | | +-------------+ +------------+ | +--------+ PHY|ODUflex +------- |ODUFlex|PHY +--------+ +-------------+ +------------+ + Legend: | R1, R2 + Routers (supporting the FlexE clients) | NE A, Z + Transport Network Edge nodes + FlexE-psg: FlexE partial rate (sub) group signal (per G.709:17.12)
===============================================================
Figure 2: FlexE Aware Transport
These usecases build upon the basic router-transport equipment connectivity illustrated in Figure 1. The FlexE shim layer at the router maps to the set of FlexE clients over the FlexE group, as usual. This section considers various usecases in which the equipment located at the edge of the transport network instantiates the FlexE Shim function which peers with the FlexE shim on the customer device. In the router to network direction, the transport edge node terminates the FlexE shim layer, and extracts one or more FlexE client signals, and transports them through the network. That is, these usecases are distinguished from the FlexE unaware cases in that the FlexE group, and the FlexE shim layer end at the transport network edge, and only the extracted FlexE client signals transit the optical network. In the network to router direction, the transport edge node maps a set of FlexE clients to the FlexE group (i.e. performing the same functions as the router which connects to the transport network).The various usecases differ in the combination of service endpoints in the transport network. In the FlexE termination scenarios, the distance between the FlexE Shims is limited the normal Ethernet link distance. The FlexE shims in the router, and the equipment need to support a small amount skew.
In this scenario, service consists of transporting a FlexE client through the transport network, and possibly combining this FlexE client with other FlexE clients into a FlexE group at the endpoints. The FlexE client signal BMP mapped into an ODUflex (of the appropriate rate) and then switched across the OTN. Figure 3 illustrates the scenario involving the mapping of a FlexE client to an ODUflex envelope; this figure only shows the signal "stack" at the service endpoints, and doesn't illustrate the switching of the ODUflex entity through the OTN. The ODUflex signal then carried over a sequence of OTUk links (with a maximum rate of 100G), and/or OTUCn (with rates of n X 100G). Although Figure 3 illustrates the scenario in which one FlexE client is transported within the OTN, the following points should be noted:
==================================================================
+--------+ 2 x 100GE +---------+ +----------+ +--------+
| | | | | | | |
| Router1| | | | | | |
| FlexE +-----------+ A-end | | Z-end +------+Router2 |
| Shim | | (FlexE | | (FlexE | |FlexE |
| +-----^-----+ term) | term) +------+ Shim |
| | | | | | | | |
| | | | | | | | |
+--------+ + +---------+ +----------+ +--------+
FlexE Group
\+--------+Transport+--------+/
network
+-----------+ +--------------+ +-------------+ +-----------+
| Client(s) | | Client | | Client | | Client(s) |
+-----------+ +--------+-----+ +------+------+ +-----------+
| FlexE Shim| | Shim | | | | Shim | | FlexE Shim|
+-----------+ +--------+ ODU | | ODU +------+ +-----------+
| PHY(s) | | PHY(s) | flex| | flex |PHY(s)| | PHY(s) |
+---+-------+ +---+----+--+--+ +---+--+---+--+ +---+-------+
| | | | | |
+---------------+ +-----------+------+----------+
=================================================================
Figure 3: FlexE termination: FlexE clients at both endpoints
The OIF implementation agreement [OIFFLEXE1] currently supports FlexE client signals carried over one or more 100GBASE-R PHY(s). There is a calendar of 5G timeslots associated with each PHY, and each FlexE client can make use of a number of timeslots (possibly distributed across the members of the FlexE group). This implies that the FlexE client rates are multiples of 5Gbps. When the rates of the FlexE client signals matches the MAC rates corresponding to existing Ethernet PHYs, i.e. 10GBASE-R/40GBASE-R/100GBASE-R, there is a need for the FlexE client signal to interwork with the native Ethernet client received from a single (non-FlexE capable) Ethernet PHY. This capability is expected to be extended to any future Ethernet PHY rates that the IEEE may define in future (e.g. 25G, 50G, 200G etc.). In these cases, although the bit rate of the FlexE client matches the MAC rate of other endpoint, the 64B66B PCS codewords for the FlexE client need to be transformed (via ordered set translation) to match the specification for the specific Ethernt PHY. These details are described in Section 7.2.2 of [OIFFLEXE1] and are not eloborated any further in this document.
Figure 4 illustrates a scenario involving the interworking of a 10G FlexE client with a 10GBASE-R native Ethernet signal. In this example, the network wrapper is ODU2e.
==================================================================
+--------+ 2 x 100GE +-------+ +-------+ +--------+
| | | | | | | |
| Router1| | | | | | |
|(FlexE +-----------+ A-end | | Z-end | 10GE |Router 2|
| Shim) | |(FlexE | | +------+ |
| +-----^-----+ term) | | | | |
| | | | | | | | |
| | | | | | | | |
+--------+ + +-------+ +-------+ +--------+
FlexE Group
\+---------Transport---------+/
network
+-----------+ +---------------+
| Client(s) | | Client | +------------+ +---------+
+-----------+ +-------+-------+ | 10GE PCS | | 10GE PCS|
| FlexE Shim| | Shim | | +-------+----+ +---------+
+-----------+ +-------+ ODU | | ODU2e | PHY| | PHY |
| PHY(s) | | PHY(s)| 2e | +---+---+--+-+ +-----+---+
+---+-------+ +---+-------+---+ | | |
| | | | | |
| | | | | |
+---------------+ +-------------+ +------------+
=================================================================
Figure 4: FlexE client interop with Native Ethernet Client
As explained in the Introduction section ( Section 1 OIFMLG3 [OIFMLG3] introduced support for carrying 10GE and 40GE client signals over a group of 100GBASE-R Ethernet PHY(s). While the most recent implementation agreement doesn't call it out explicitly, it is expected that the FlexE clients (as defined in [OIFFLEXE1]), and 10GBASE-R/40GBASE-R clients supported by OIFMLG3 [OIFMLG3]) will interoperate.
Figure 5 illustrates a scenario involving the interworking of a 10G FlexE client with a 10GBASE-R client supported by an OIFMLG3 interface. In this example, the network wrapper is ODU2e.
==================================================================
+--------+ 2 x 100GE +---------+ +---------+ +---------+
| | | | | | | |
| Router1| | | | | | |
| FlexE +-----------+ A-end | | Z-end +------+Router 2 |
| Shim | | (FlexE | | | |(MLG-3.0)|
| +-----^-----+ term) | | +------+ |
| | | | | | | | |
| | | | | | | | |
+--------+ + +---------+ +---------+ +---------+
FlexE Group
\+--------+Transport+--------+/
network
+-----------+ +-------------+ +--------------+ +----------+
| Client(s) | | Client | | 10GE PCS | | 10GE Cl. |
+-----------+ +--------+----+ +------+-------+ +----------+
| FlexE Shim| | Shim | | | | MLG3 | | MLG3 |
+-----------+ +--------+ ODU| | ODU +-------+ +----------+
| PHY(s) | | PHY(s) | 2e | | 2e | PHY(s)| | PHY(s) |
+---+-------+ +---+----+--+-+ +---+--+---+---+ +---+------+
| | | | | |
+---------------+ +------------+ +------------+
=================================================================
Figure 5: FlexE client interop with Ethernet Client supported by MLG3
This section covers a degenerate FlexE termination scenario in which Router1, Router2, and Router3 are interconnected through back-to-back FlexE groups without an intermediate transport network (see Figure 6). Even in scenarios where there is a transport network providing FlexE unaware/aware transport services for this pair of FlexE groups, the FlexE layer network can be viewed as an overlay on top of the underlying transport network. As such, all of the FlexE Shim operations (e.g. adding/deleting FlexE clients, resizing existing clients) proceed in the same manner -- regardless of whether the routers are directly connected or not.
In this example, the FlexE Shim at Router2 extracts one or more FlexE client signals from the FlexE group connected to Router1, and mutliplexes these extracted FlexE signals into the FlexE group towards the appropriate router (e.g. Router3). Note that each of the extracted FlexE client signals can be independently routed towards its respective FlexE group.
==================================================================
+--------+ 2 x 100GE +---------+ 3 x 100GE +---------+
| | | | | |
| Router1| | | | |
| FlexE +-----------+ Router2 +-----------+ Router3 |
| Shim | | FlexE +-----------+ FlexE |
| +-----^-----+ Shim +-----^-----+ Shim |
| | | | | | | |
| | | | | | | |
+--------+ + +---------+ + +---------+
FlexE Group FlexE Group
=================================================================
Figure 6: Back-to-Back FlexE
The hop-by-hop (a hop is delimited by two FlexE Shim functions) resizing of a FlexE client signal operates by maintaining two sets of calendar slots for each client: the present and the future. Once the configuration of both calendar slots for a specific client is complete, the node signals to its peer to switch to from the present set to the new set of calendar slots. Note that the switch to the new set of calendar slots is unidirectional, and the process is executed independently for both directions of transfer. This process makes use of the following FlexE overhead (as per [OIFFLEXE1]
FlexE client resizing operations are supported and can be achieved via the configuration of Calendar A and Calendar B. It is worth noting that there is no guarantee that such resizing will be hitless. Table 1 provides a summary of client bandwidth resize applicability in various use cases presented in this document.
| FlexE Shim endpoint 1 | FlexE Shim endpoint 2 | Usecase | Transport Network Function | Resizing supported? |
|---|---|---|---|---|
| CE (e.g. router) | CE | Section 3.1 | FlexE unaware transport | Yes. Done at endpoints. The OTN pipes are configured for the maximum number of calendar slots across each PHY in the FlexE group. Therefore, no resizing is required in the OTN layer. |
| CE (e.g. router) | CE | Section 3.2 | FlexE aware transport | Supported at the endpoints only if the set of available/unavailable calendar slots is constant. Not supported otherwise (see notes at the end of Section 3.2). |
| CE (e.g. router) | Transport Network Edge | Section 3.3.1 | FlexE Termination in Transport | Not supported due to lack to lack of a general (i.e. one that works regardless of the ODUflex bandwidth) hitless ODUflex resizing in G.709. |
| CE (e.g. router) | CE | Section 3.4 | No transport network | Yes. Done at endpoints by CE(s). Thus, for example, in Figure 6, the resizing of the end-to-end FlexE client circuit with a scope of Router1-Router2-Router3 is accomplished by correctly coordinating the resizing operations across these two segments: Router1-Router2, Router2-Router3. It is expected that the exact sequence of hop-by-hop resize operations is different between bandwidth increase/decrease scenarios. |
This section summarizes the requirements for FlexE Group and FlexE Client signaling and routing. The requirements are derived from the usecases described in Section 3 of this document. Data plane requirements (and/or solutions) (e.g. crunching of tributary slots, adding unavailable tributary slots etc.) are not explicitly mentioned in the following text. Given that the control plane sets up circuits that transport client streams, there are no implications for the control plane in matters of delay, jitter tolerance etc. The requirements listed in this section will be used to identify the Control Plane (i.e. Routing and Signaling) extensions that will be required to support FlexE services in an OTN.
A Control Plane solution will be compliant to the specification in Section 7 if it meets all the mandatory (MUST, SHALL) requirments, the solution may also meet the optional (SHOULD, MAY) requirments.
This section discusses the environment where FlexE operates, this should include both what FlexE runs over and what applications run on top of FlexE.
This section discusses the different parts of FlexE signaling and routing and how these parts interoperte.
This memo includes no request to IANA.
Note to the RFC Editor: This section should be removed before publishing.
None.
Khuzema Pithewan, Infinera Corp, kpithewan@infinera.com
Fatai Zhang, Huawei, zhangfatai@huawei.com
Jie Dong, Huawei, jie.dong@huawei.com
Zongpeng Du, Huawei, duzongpeng@huawei.com
Xian Zhang, Huawei, zhang.xian@huawei.com
James Huang, Huawei, james.huang@huawei.com
Qiwen Zhong, Huawei, zhongqiwen@huawei.com
| [G7044] | ITU, "Hitless adjustment of ODUflex(GFP) (https://www.itu.int/rec/T-REC-G.7044-201110-I/en)", Cctober 2011. |
| [G709] | ITU, "Optical Transport Network Interfaces (http://www.itu.int/rec/T-REC-G.709-201606-P/en)", July 2016. |
| [G798] | ITU, "Characteristics of optical transport network hierarchy equipment functional blocks (http://www.itu.int/rec/T-REC-G.798-201212-I/en)", February 2014. |
| [OIFFLEXE1] | OIF, "FLex Ethernet Implementation Agreement Version 1.0 (OIF-FLEXE-01.0)", March 2016. |
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997. |
| [OIFMLG3] | OIF, "Multi-Lane Gearbox Implementation Agreement Version 3.0 (OIF-MLG-3.0)", April 2016. |
This becomes an Appendix.