0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                   Label Stack Element                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                   Label Stack Element                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              BIER-MPLS label          |     |1|               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 1 0 1|  Ver  |  Len  |              Entropy                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                BitString  (first 32 bits)                     ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~                                                               ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~                BitString  (last 32 bits)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|OAM|     Reserved    | Proto |             BFIR-id             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ver | Message Type  | Proto |          Reserved               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~                  BFD control packet                           ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
          

Figure 2: BFD over BIER with G-ACh format

3.1.1.2. Proactive CC/CV in NVO3

There is currently no WG document on proactive CC/CV. The individual requirements document [I-D.ashwood-nvo3-oam-requirements] covers this and there is a related proposal for BFD over VXLAN in [I-D.spallagatti-bfd-vxlan].

3.1.1.3. Proactive CC/CV over SFP

3.1.2. On-demand Continuity Check and Connectivity Verification

On-demand Continuity Check and Connectivity Verification protocols include:

• MPLS Echo Request/Reply, a.k.a. LSP Ping, as defined in [RFC4379] and its numerous extensions;
• LSP Self-ping, as defined in [RFC7746];
• [I-D.kumarzheng-bier-ping] is a good example of generic troubleshooting and defect localization tool that can be extended and suited for more specific requirements of the particular type of an overlay network;

3.1.2.1. On-demand CC/CV in BIER

[I-D.kumarzheng-bier-ping] defines format of Echo Request/Reply control packet and set of TLVs that can be used to perform failure detection and isolation in BIER domain over MPLS network.

3.1.2.2. On-demand CC/CV in NVO3

There is currently no WG document for on-demand CC/CV.

Individual documents exist for tracing such as [I-D.pang-nvo3-vxlan-path-detection], and [I-D.nordmark-nvo3-transcending-traceroute].

3.1.2.3. On-demand CC/CV over SFP

3.1.3. Alarm Indication Signal

3.1.3.1. AIS in BIER

3.1.3.2. AIS in NVO3

There is currently no WG document on Alarm Indication Signal.

The individual draft [I-D.nordmark-nvo3-transcending-traceroute] suggests reusing ICMP errors for defect indications.

3.1.3.3. AIS over SFP

3.2. Overlay OAM Performance Measurement

These protocols may be considered for Overlay Performance Measurement (PM) OAM:

• packet loss and delay measurement in MPLS networks, as defined in [RFC6374] with ability to export measurement results for post-processing [I-D.ietf-mpls-rfc6374-udp-return-path];
• One-Way Active Measurement Protocol (OWAMP), as defined in [RFC4656], and Two-Way Active Measurement Protocol (TWAMP), as defined in [RFC5357], [RFC6038], and [RFC7750];
• use of the Marking Method [I-D.ietf-ippm-alt-mark] that, if accordingly supported by the overlay layer, can behave as close as technically possible to a passive method to measure performance, e.g. [I-D.mirsky-bier-pmmm-oam].

3.2.1. Overlay OAM PM Active

Requirements towards PM OAM for overlay networks are listed in the Section 4.2 [I-D.ooamdt-rtgwg-ooam-requirement]. Two sets of performance measurement protocols had been developed at IETF so far:

• OWAMP [RFC4656] and TWAMP [RFC5357] each includes the control protocol to negotiate required parameters and control a test session as well as the test protocol itself that specify format and processing of a test packet. Historically, TWAMP, that enables measurement of the latency, packet loss both as one-way and round trip performance metric, has been implemented more often and thus gained wider deployment than OWAMP. There are several properties of the test protocol that may not be suitable for its use in overlay networks:

  -

  the test protocol is targetted to IP layer and carries some IP-specific information;

  -

  the format of the sent test and the reflected packets differ significantly and that complicates efficient HW-based implementation;

  -

  latency and packet loss measurement operations are bundled together and that causes certain overhead when only one of performance metrics is to be measured;

  -

  only Network Time Protocol (NTP) format of timestamp is currently supported that requires additional processing to convert from IEEE-1588 time format that broadly supported in many current packet forwarding engines.

• [RFC6374] defines the test protocol that enables measurement of the latency and paket loss as one-way and round-trip perfomance metrics. Comparing to OWAMP/TWAMP RFC 6374 has certain advantages:

  -

  the test protocol is flexible and these performance metrics can be measured independently or in the single test session;

  -

  the protocol does not carry transport layer specific information;

  -

  there's no difference between format of the packet transmitted by the sender and reflected by the responder as the test packets preallocates space for all necesary data it collects;

  -

  both NTP and PTP time formats allowed to be used to record time for latency measurement.

[RFC6374] can be used as foundation of active PM OAM in overlay networks. The YANG data model [RFC6020] of the packet loss and delay measurement based on [RFC6374] can improve control and increase operational value of active performance measurement in overlay networks.

3.2.1.1. Active PM in BIER

Currently there is no draft related to active PM OAM in the WG.

3.2.1.2. Active PM in NVO3

Performance management has been discussed in NVO3 but there is currently no draft in the WG.

3.2.1.3. Active PM over SFP

3.2.2. Overlay OAM PM Passive

3.2.2.1. Passive PM in BIER

[I-D.mirsky-bier-pmmm-oam] describes how the Marking Method can be used in BIER domain over MPLS networks.

3.2.2.2. Passive PM in NVO3

Marking has been discussed in NVO3 sessions, but there is no draft in the working group.

The Generic Protocol Extension for VXLAN [I-D.ietf-nvo3-vxlan-gpe], Generic Network Virtualization Encapsulation [I-D.ietf-nvo3-geneve], Generic UDP Encapsulation [I-D.ietf-nvo3-gue] are just some examples of the new encapsulations to support network virtualization. NVO3 PM would be used to probe the NV Edge to NV Edge tunnels and NV Edge entity status for a DC network. The main requirement for Performance Management is to be able to support measurement of the frame loss, delay and delay variation between two NV Edge devices that support the same VNI within a given NVO3 domain on per VNI basis. Alternate Marking Method [I-D.ietf-ippm-alt-mark] enables calculation of these metrics but sets forth requirements toward overlay encapsulation to make use of the AMM behave in the network as passive OAM per definition in [RFC7799].

3.2.2.3. Passive PM over SFP

In the SFC architecture SF, SFF, Classifier and NSH Proxy Agent are the elements that can incorporate the measurement agent functionality to support SFC performance measurement. The required OAM Performance Measurement, as described in [I-D.ietf-sfc-oam-framework] highlight the capability to assess the monitoring at SF and SFF or a Set of SF/SFF, both in case of SFC-aware SF and SFC-unaware SF; the monitoring of SFP (and RSP) that comprises a set of SFs that may be ordered or unordered; the monitoring of the Classifiers operation and the monitoring of the SFC as a whole.

Performance measurement includes measuring of packet loss, delay, delay variation and could be performed by the marking method proposed in [I-D.ietf-ippm-alt-mark]. To make use of the marking method behave as passive OAM, as defined in [RFC7799], the overlay network encapsulation should allocate the field, preferrably two bits long, whose value does not affect how a packet is treated by the overlay network.

3.3. Telemetry in Overlay OAM

Excessive use of the in-band OAM channel may affect user flow and thus change network behavior. For example, if operator uses passive measurement exporting massive amount of data over the OAM channel may affect network. I think that a management channel should be used in such case. Obviously it may traverse the same nodes and links but may not require the same QoS. We can refer to LMAP Reference Model [RFC7594] with Controller, Measurement Agent and Data Collector.

[I-D.lapukhov-dataplane-probe] proposes transport independent generic telemetry probe structure.

3.4. Conclusions

• A common Overlay OAM header should be defined to support demultiplexing of OAM protocols.
• Existing modes of BFD protocol, primarily its Async mode, can be used either in IP/UDP or ACH format, as proactive continuity check mechanism in overlay networks.
• A new control packet to be used for on-demand CC/CV in overlay networks should be defined. Set of common TLVs may be defined while more specific TLVs to be defined by respective groups of experts.
• [RFC6374] can be used as the foundation of active performance measurement OAM in overlay networks.
• YANG data model of the active PM OAM in overlay networks would improve control and increase operational value of the test methods.
• Performance measurement includes measuring of packet loss, delay, delay variation and could be performed by the marking method, for example as proposed in [I-D.ietf-ippm-alt-mark]. To make use of the marking method behave as passive OAM, as defined in [RFC7799], the overlay network encapsulation should allocate the field, preferrably two bits long, whose value does not affect how a packet is treated by the overlay network.

4. IANA Considerations

This document does not propose any IANA consideration. This section may be removed.

5. Security Considerations

6. Acknowledgement

TBD

7. References

7.1. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.

7.2. Informative References

[I-D.ashwood-nvo3-oam-requirements]	Chen, H., Ashwood-Smith, P., Xia, L., Iyengar, R., Tsou, T., Sajassi, A., Boucadair, M., Jacquenet, C., Daikoku, M., Ghanwani, A. and R. Krishnan, "NVO3 Operations, Administration, and Maintenance Requirements", Internet-Draft draft-ashwood-nvo3-oam-requirements-04, October 2015.
[I-D.ietf-bfd-multipoint]	Katz, D., Ward, D. and J. Networks, "BFD for Multipoint Networks", Internet-Draft draft-ietf-bfd-multipoint-08, April 2016.
[I-D.ietf-bfd-multipoint-active-tail]	Katz, D., Ward, D. and J. Networks, "BFD Multipoint Active Tails.", Internet-Draft draft-ietf-bfd-multipoint-active-tail-02, May 2016.
[I-D.ietf-bfd-seamless-base]	Pignataro, C., Ward, D., Akiya, N., Bhatia, M. and J. Networks, "Seamless Bidirectional Forwarding Detection (S-BFD)", Internet-Draft draft-ietf-bfd-seamless-base-11, May 2016.
[I-D.ietf-bfd-seamless-ip]	Pignataro, C., Ward, D. and N. Akiya, "Seamless Bidirectional Forwarding Detection (S-BFD) for IPv4, IPv6 and MPLS", Internet-Draft draft-ietf-bfd-seamless-ip-06, May 2016.
[I-D.ietf-ippm-alt-mark]	Fioccola, G., Capello, A., Cociglio, M., Castaldelli, L., Chen, M., Zheng, L., Mirsky, G. and T. Mizrahi, "Alternate Marking method for passive performance monitoring", Internet-Draft draft-ietf-ippm-alt-mark-01, July 2016.
[I-D.ietf-mpls-rfc6374-udp-return-path]	Bryant, S., Sivabalan, S. and S. Soni, "RFC6374 UDP Return Path", Internet-Draft draft-ietf-mpls-rfc6374-udp-return-path-05, April 2016.
[I-D.ietf-nvo3-geneve]	Gross, J. and I. Ganga, "Geneve: Generic Network Virtualization Encapsulation", Internet-Draft draft-ietf-nvo3-geneve-01, January 2016.
[I-D.ietf-nvo3-gue]	Herbert, T., Yong, L. and O. Zia, "Generic UDP Encapsulation", Internet-Draft draft-ietf-nvo3-gue-04, July 2016.
[I-D.ietf-nvo3-vxlan-gpe]	Kreeger, L. and U. Elzur, "Generic Protocol Extension for VXLAN", Internet-Draft draft-ietf-nvo3-vxlan-gpe-02, April 2016.
[I-D.ietf-rtgwg-dt-encap]	Nordmark, E., Tian, A., Gross, J., Hudson, J., Kreeger, L., Garg, P., Thaler, P. and T. Herbert, "Encapsulation Considerations", Internet-Draft draft-ietf-rtgwg-dt-encap-01, March 2016.
[I-D.ietf-sfc-oam-framework]	Aldrin, S., Krishnan, R., Akiya, N., Pignataro, C. and A. Ghanwani, "Service Function Chaining Operation, Administration and Maintenance Framework", Internet-Draft draft-ietf-sfc-oam-framework-01, February 2016.
[I-D.kumarzheng-bier-ping]	Kumar, N., Pignataro, C., Akiya, N., Zheng, L., Chen, M. and G. Mirsky, "BIER Ping and Trace", Internet-Draft draft-kumarzheng-bier-ping-03, July 2016.
[I-D.lapukhov-dataplane-probe]	Lapukhov, P. and r. remy@barefootnetworks.com, "Data-plane probe for in-band telemetry collection", Internet-Draft draft-lapukhov-dataplane-probe-01, June 2016.
[I-D.mirsky-bier-pmmm-oam]	Mirsky, G., Zheng, L., Chen, M. and G. Fioccola, "Performance Measurement (PM) with Marking Method in Bit Index Explicit Replication (BIER) Layer", Internet-Draft draft-mirsky-bier-pmmm-oam-01, March 2016.
[I-D.nordmark-nvo3-transcending-traceroute]	Nordmark, E., Appanna, C. and A. Lo, "Layer-Transcending Traceroute for Overlay Networks like VXLAN", Internet-Draft draft-nordmark-nvo3-transcending-traceroute-02, March 2016.
[I-D.ooamdt-rtgwg-ooam-requirement]	Kumar, N., Pignataro, C., Kumar, D., Mirsky, G., Chen, M., Nordmark, E., Networks, J. and D. Mozes, "Overlay OAM Requirements", Internet-Draft draft-ooamdt-rtgwg-ooam-requirement-01, July 2016.
[I-D.pang-nvo3-vxlan-path-detection]	<>, J., Liu, D., Liu, D., Zhang, D., Yizhou, L., Chen, H., <>, D., <>, B., Kumar, D., Gao, R. and Y. Qiao, "Path Detection in VXLAN Overlay Network", Internet-Draft draft-pang-nvo3-vxlan-path-detection-02, March 2016.
[I-D.saum-nvo3-pmtud-over-vxlan]	Dikshit, S. and A. Nayak, "PMTUD Over Vxlan", Internet-Draft draft-saum-nvo3-pmtud-over-vxlan-03, June 2016.
[I-D.singh-nvo3-vxlan-router-alert]	Singh, K., Jain, P., Rio, D., Henderickx, W., Shekhar, R. and R. Rahman, "VxLAN Router Alert Option", Internet-Draft draft-singh-nvo3-vxlan-router-alert-02, September 2015.
[I-D.spallagatti-bfd-vxlan]	Networks, J., sajibasil@gmail.com, s., Paragiri, S., Govindan, V., Mudigonda, M. and G. Mirsky, "BFD for VXLAN", Internet-Draft draft-spallagatti-bfd-vxlan-03, April 2016.
[RFC4379]	Kompella, K. and G. Swallow, "Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures", RFC 4379, DOI 10.17487/RFC4379, February 2006.
[RFC4656]	Shalunov, S., Teitelbaum, B., Karp, A., Boote, J. and M. Zekauskas, "A One-way Active Measurement Protocol (OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006.
[RFC5357]	Hedayat, K., Krzanowski, R., Morton, A., Yum, K. and J. Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", RFC 5357, DOI 10.17487/RFC5357, October 2008.
[RFC5880]	Katz, D. and D. Ward, "Bidirectional Forwarding Detection (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010.
[RFC5882]	Katz, D. and D. Ward, "Generic Application of Bidirectional Forwarding Detection (BFD)", RFC 5882, DOI 10.17487/RFC5882, June 2010.
[RFC5883]	Katz, D. and D. Ward, "Bidirectional Forwarding Detection (BFD) for Multihop Paths", RFC 5883, DOI 10.17487/RFC5883, June 2010.
[RFC5884]	Aggarwal, R., Kompella, K., Nadeau, T. and G. Swallow, "Bidirectional Forwarding Detection (BFD) for MPLS Label Switched Paths (LSPs)", RFC 5884, DOI 10.17487/RFC5884, June 2010.
[RFC5885]	Nadeau, T. and C. Pignataro, "Bidirectional Forwarding Detection (BFD) for the Pseudowire Virtual Circuit Connectivity Verification (VCCV)", RFC 5885, DOI 10.17487/RFC5885, June 2010.
[RFC6020]	Bjorklund, M., "YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)", RFC 6020, DOI 10.17487/RFC6020, October 2010.
[RFC6038]	Morton, A. and L. Ciavattone, "Two-Way Active Measurement Protocol (TWAMP) Reflect Octets and Symmetrical Size Features", RFC 6038, DOI 10.17487/RFC6038, October 2010.
[RFC6374]	Frost, D. and S. Bryant, "Packet Loss and Delay Measurement for MPLS Networks", RFC 6374, DOI 10.17487/RFC6374, September 2011.
[RFC6428]	Allan, D., Swallow, G. and J. Drake, "Proactive Connectivity Verification, Continuity Check, and Remote Defect Indication for the MPLS Transport Profile", RFC 6428, DOI 10.17487/RFC6428, November 2011.
[RFC7276]	Mizrahi, T., Sprecher, N., Bellagamba, E. and Y. Weingarten, "An Overview of Operations, Administration, and Maintenance (OAM) Tools", RFC 7276, DOI 10.17487/RFC7276, June 2014.
[RFC7594]	Eardley, P., Morton, A., Bagnulo, M., Burbridge, T., Aitken, P. and A. Akhter, "A Framework for Large-Scale Measurement of Broadband Performance (LMAP)", RFC 7594, DOI 10.17487/RFC7594, September 2015.
[RFC7726]	Govindan, V., Rajaraman, K., Mirsky, G., Akiya, N. and S. Aldrin, "Clarifying Procedures for Establishing BFD Sessions for MPLS Label Switched Paths (LSPs)", RFC 7726, DOI 10.17487/RFC7726, January 2016.
[RFC7746]	Bonica, R., Minei, I., Conn, M., Pacella, D. and L. Tomotaki, "Label Switched Path (LSP) Self-Ping", RFC 7746, DOI 10.17487/RFC7746, January 2016.
[RFC7750]	Hedin, J., Mirsky, G. and S. Baillargeon, "Differentiated Service Code Point and Explicit Congestion Notification Monitoring in the Two-Way Active Measurement Protocol (TWAMP)", RFC 7750, DOI 10.17487/RFC7750, February 2016.
[RFC7799]	Morton, A., "Active and Passive Metrics and Methods (with Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, May 2016.

Authors' Addresses