Internet DRAFT - draft-krishnan-opsawg-in-band-pro-sla
draft-krishnan-opsawg-in-band-pro-sla
OPSAWG Working Group Ram(Ramki) Krishnan
Internet Draft Support Vectors
Category: Experimental
Expires: April 2017 December 30, 2016
In-band Telemetry for a Proactive SLA Monitoring Framework
draft-krishnan-opsawg-in-band-pro-sla-03
Abstract
The goal of in-band telemetry is to drive per packet, per hop real-
time monitoring for the infrastructure towards achieving a
programmable proactive SLA monitoring framework. Some of the key
aspects from a switch/NIC perspective are - ingress/egress timestamp
(latency), queue depth, bandwidth etc. Some of the key aspects from
a server perspective are - cache/memory statistics etc. This
document summarizes the current work in the industry in this area
and identifies key requirements for a comprehensive solution.
Towards addressing the requirements, this document describes uses
cases and defines reusable monitoring packet formats across all
layers in the OAM hierarchy.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with
the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
Krishnan Expires April 2014 [Page 1]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on April, 2014.
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document.
Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119 [RFC 2119].
Table of Contents
1. Introduction...................................................4
1.1. Acronyms..................................................5
2. In-band Telemetry for IPSEC tunnel packets.....................5
2.1. Packet Format 1 - Geneve..................................6
2.2. Packet Format 2 - VXLAN GPE...............................7
2.3. Packet Format 3 - IP options..............................8
3. In-band Telemetry for Service Chaining.........................8
3.1. NSH for service chaining Packet Format...................10
3.2. VXLAN-GPE for overlay and NSH for service chaining Packet
Format........................................................10
3.3. VXLAN-GPE for overlay and NSH for service chaining Packet
Format........................................................11
4. Pre-construction/Minimizing of In-band Telemetry Header.......12
4.1. Pre-construction of In-band Telemetry Header.............12
4.2. Minimizing of In-band Telemetry Header for latency
measurement...................................................12
5. Summarizing information in Telemetry header...................13
6. IANA Considerations...........................................14
7. Security Considerations.......................................14
8. Acknowledgements..............................................14
9. References....................................................15
Krishnan Expires April 2014 [Page 2]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
9.1. Normative References.....................................15
9.2. Informative References...................................15
Krishnan Expires April 2014 [Page 3]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
1. Introduction
Proactive SLA monitoring is key for enabling DevOps in a converged
infrastructure. As new services are continuously enabled using
DevOps methodologies, it is critical to make sure that the users are
delivered the promised SLAs through proactive SLA monitoring; in the
case where SLAs are violated, the system should be able to
automatically fix the issue or revert back to the old configuration
as needed.
Standards-based monitoring schemes [ietf-twamp] are coarse grained -
first, based on injected packets and not on customer data packets
and next, lack of per hop visibility while monitoring end-to-end and
last, lack of coverage for network functions.
New proposed monitoring schemes focus on switches/routers end-to-end
in the DC - in-band network telemetry [p4-in-band] is to enable per
packet, per hop monitoring for timestamp (latency), queue depth,
bandwidth etc., Data-plane probe for in-band telemetry collection
[ietf-in-band-dpp] is to enable the above per injected packet,
[ietf-sfc-monitor] describes one-way latency monitoring for service
chaining nodes using timestamps.
Given the above landscape, the key requirements for a comprehensive
proactive SLA monitoring framework are as follows
. Ability to monitor selective flows, e.g. monitor only low latency
traffic
. Ability to mirror selective flows which are monitored, e.g.
mirror only low latency traffic (mirroring all flows may not
scale)
. Ability to convey summarized information to the central
management entity, e.g. alert the central management system only
when a programmable percentile (for example 99.9th) latency
exceeds a high threshold for a flow since mirroring entire flow
may not scale
. Ability to strip monitoring information in the network edge
since the application network stacks may not be able to process
the additional monitoring information
. Ability to handle encrypted packets, e.g. enterprise cloud VPN
across WAN, secure IaaS tunnels within a DC
Krishnan Expires April 2014 [Page 4]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
. Ability to monitor individual network function paths, e.g. VNF
service chaining where several VNFs/VMs are sharing the same
physical server
. Ability to address each layer in the OAM hierarchy in a
generic way using a common monitoring format. Within a DC,
the various OAM layers could be Service Function, Overlay and
Underlay.
. Ability to pre-construct the space for monitoring headers
(described in Section 4) to guarantee deterministic performance
especially for virtual network functions which are subject to a
cache hierarchy in an industry standard server
. Ability to programmably select the hops being monitored
to make sure the monitoring header size is bounded
Towards addressing the key requirements, this document describes
uses cases and packet formats for handling encrypted data packets
(e.g. IPSEC for IaaS deployment) and service chaining and also
describes options for maintaining deterministic application
performance while performing elaborate monitoring.
1.1. Acronyms
DPI: Deep Packet Inspection
MPLS: Multiprotocol Label Switching
NVGRE: Network Virtualization using Generic Routing Encapsulation
OAM: Operations, Administration, and Maintenance
SF: Service Function
SFC: Service Function Chain
SFP: Service Function Path
VXLAN: Virtual Extensible LAN
2. In-band Telemetry for IPSEC tunnel packets
The following describes in-band telemetry for IPSEC tunnels which is
the most popular WAN tunneling protocol for secure communication.
Use Cases:
Krishnan Expires April 2014 [Page 5]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
. Cloud VPN: IPSEC tunnel between Enterprise branch and
Enterprise/Cloud DC
oPrimary use case for IPSEC is inter-domain, for example
enterprise branch office to PoP could be one network domain
(operator A) and PoP to Enterprise/Cloud DC could be another
network domain (Operator B), e.g. Google Cloud Interconnect
oValue proposition:
. Real-time visibility/Service assurance for high priority
tunnels carrying applications such as real-time
voice/video
. Minimal WAN switch/router buffer overprovisioning for
all classes of traffic and maximizing WAN link
utilization
. Intra-DC: IPSEC tunnel between overlay end points for a private
multi-tenant environment in a converged infrastructure (vlan,
VXLAN provide isolation but not privacy)
oValue proposition:
. Real-time visibility/Service assurance for high priority
tunnels carrying applications such as transactional
storage, real-time big data
. Minimal DC switch/router buffer overprovisioning for all
classes of traffic
There are several possible packet formats for achieving the above use
cases. They are described below.
2.1. Packet Format 1 - Geneve
. Outer MAC Header
. Outer IP Header
oIP protocol - UDP
oDestination IP, Source IP, other fields
. Outer UDP Header
oDestination UDP port - Geneve (6081)
Krishnan Expires April 2014 [Page 6]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
. Outer Geneve Header
oProtocol type - 0x6558 (RFC 1701- trans ethernet bridging)
oOption Length - greater than zero
oOption "INT"
. Option Class (16 bits) - INT
. Option Class needs to sync up with [ietf-geneve]
oOption "Next Protocol" - new option (total length including
data is 8 bytes)
. Option class (16 bits) - Next Protocol
. Overrides protocol type in base Geneve header
. Type (8 bits) - Critical bit is set, Lower 8 bit byte in
4 bytes of data is protocol
. Reserved (3 bits)
. Length (5 bits) - set to 0x1 (4 bytes of data)
. Data (32 bits) - for IPSEC - set to 0x0000032 (ESP) or
0x00000033 (AH)
. Encrypted or Authenticated payload
2.2. Packet Format 2 - VXLAN GPE
. Outer MAC Header
. Outer IP Header
oIP protocol - UDP
oDestination IP, Source IP, other fields
. Outer UDP Header
oDestination UDP port - VXLAN GPE (4790)
. Outer VXLAN GPE Header
Krishnan Expires April 2014 [Page 7]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
oNext Protocol - 0x5 - INT
. Outer INT Header(s)
oNext Protocol - ESP (0x7) or AH (0x8)
. Need to create two new next protocols, aligning with
[ietf-nsh] and [p4-in-band]
. Encrypted or Authenticated payload
2.3. Packet Format 3 - IP options
Just like Geneve option format, IP options could be leveraged for
in-band telemetry data.
. Outer MAC Header
. Outer IP Header
oIP protocol - ESP (0x7) or AH (0x8)
oDestination IP, Source IP, other fields
oIP Header length > 5 (indicate presence of IP options)
. Outer IP options Header
oOption-type
. Copied Flag - 1
. Option Class - 2
. Option Number
. 10 - In-band Telemetry (new)
. Option-Length - variable
. Option-Data - in-band telemetry data
. Encrypted or Authenticated payload
3. In-band Telemetry for Service Chaining
Use cases:
Krishnan Expires April 2014 [Page 8]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
. No. 1) Monitoring of the networking interconnect. This would
typically involve monitoring the overlay/underlay across the
individual service chain nodes and the service chaining header
([ietf-nsh] etc.) across the entire service chain at the entry and
exit points.
. No. 2) Monitoring of the individual network functions comprising
a service chain using the service chaining header ([ietf-nsh]
etc.). The network functions could be virtual (VMs, Containers
etc.) or physical.
. No. 1) and 2) Combination of above two use cases involving
simultaneous monitoring of networking interconnect and individual
network functions.
. Value Proposition:
oMonitoring of the networking interconnect provides the usual
benefits of underlay monitoring
oMonitoring of individual network functions through vSwitch/NIC
helps rapid identification of any server side issues
especially for virtual network functions.
. For example, a programmable percentile such as 99.9th
latency/queue depth/queue drops exceeding a high
threshold might mean collision in a scarce shared
resource such as L1/L2/L3 cache in an industry standard
server.
. The above information could be conveyed to a central
management right away, which could remedy the situation
by migrating the virtual network function to another
appropriate industry standard server with no collision
in a scarce shared resource such as L1/L2/L3 cache.
Typical elements involved in service chain monitoring are
vSwitches/NIC/ToR. For each individual network functions comprising
a service chain, vSwitch/NIC/ToR will monitor ingress traffic to the
network function for one or more of the INT [p4-in-band] parameters
such as timestamp, queue depth, bandwidth and egress traffic to the
vSwitch/NIC/ToR for one or more of the aforementioned INT parameters.
Monitoring of the entire service chain at the entry point involves
monitoring traffic sent to the first network function from
vSwitch/NIC/ToR and exit point involves monitoring traffic from the
last network function to the vSwitch/NIC/ToR for one of the
aforementioned INT parameters. For highly accurate monitoring, it is
Krishnan Expires April 2014 [Page 9]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
recommended to use NIC/ToR vs a software based vSwitch. For example,
HW implementations can measure timestamps to a nanosecond accuracy
and can synchronize accurately with the master clock using protocols
like IEEE 1588 PTP. A useful reference is [odl-nsh] which describes
NSH service chaining operations from a ToR perspective.
Typical elements involved in underlay monitoring are ToR,
Aggregation and Core switches/routers.
There are several possible packet formats for achieving the above
the above use cases. Some are described here. More packet formats
are work in progress.
3.1. NSH for service chaining Packet Format
. NSH Header
oNext Protocol - 0x5 - INT
. Needs to sync with next protocol in [ietf-nsh]
. NSH INT Header(s) (processed in vSwitch/NIC/ToR at each
configured service chaining hop besides entry and exit points)
oNext Protocol - 0x3 - Ethernet
. Inner Ethernet payload
3.2. VXLAN-GPE for overlay and NSH for service chaining Packet Format
. Outer MAC Header
. Outer IP Header
oIP protocol - UDP
oDestination IP, Source IP, other fields
. Outer UDP Header
oDestination UDP port - VXLAN GPE (4790)
. Outer VXLAN GPE Header
oNext Protocol - 0x5 - INT
. Outer INT Header(s)
Krishnan Expires April 2014 [Page 10]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
oNext Protocol - 0x4 - NSH
. NSH Header
oNext Protocol - 0x5 - INT
. Needs to sync with next protocol in [ietf-nsh]
. NSH INT Header(s) (processed in vSwitch/NIC/ToR at each
configured service chaining hop besides entry and exit points)
oNext Protocol - 0x3 - Ethernet
. Inner Ethernet payload
3.3. VXLAN-GPE for overlay and NSH for service chaining Packet Format
. Outer MAC Header
. Outer IP Header
oIP protocol - UDP
oDestination IP, Source IP, other fields
. Outer UDP Header
oDestination UDP port - VXLAN GPE (4790)
. Outer VXLAN GPE Header
oNext Protocol - 0x5 - INT
. Outer INT Header(s)
oNext Protocol - 0x4 - NSH
. NSH Header
oNext Protocol - 0x5 - INT
oneeds to sync with next protocol in [ietf-nsh]
. NSH INT Header(s) (processed in vSwitch/NIC/ToR at each
configured service chaining hop besides entry and exit points)
oNext Protocol - 0x3 - Ethernet
Krishnan Expires April 2014 [Page 11]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
. Inner Ethernet payload
4. Pre-construction/Minimizing of In-band Telemetry Header
The following describes methods for pre-constructing and minimizing
the In-band Telemetry header(s) and the corresponding benefits.
4.1. Pre-construction of In-band Telemetry Header
Method:
. Pre-construct timestamp header for all hops; Use timestamp append
model in all switches/routers - virtual switch/router in server,
switch/router in NIC, switch/router in TOR etc.
. Mirror packet with the entire timestamp information in the last
hop; Examine the mirrored packet offline to determine any latency
violations in near-real-time
Benefits:
. Last hop does not have to support latency measurement
. Easy to implement in SW/HW switches/routers with minimal
performance impact with programmable data path
. No MTU change as packet traverses multiple hops leading to
deterministic performance especially for VNF workloads
. Applicable to switches/routers as well as network functions
Besides timestamp/latency measurement, the aforementioned scheme is
applicable to other parameters such as queue depth, bandwidth,
packet drops etc. with the key benefit of no MTU change as the
packet traverses multiple hops.
4.2. Minimizing of In-band Telemetry Header for latency measurement
Method:
. Only 1 additional field in the packet for latency measurement
o Cumulative timestamp - 4 bytes
. Computes hop-by-hop latency information using the last cumulative
timestamp
Krishnan Expires April 2014 [Page 12]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
o Hop-by-hop latency = current timestamp - last cumulative
timestamp
o Update the cumulative timestamp field with the current
timestamp
. Mirror (replicate) the entire (or truncated) packet with the
received cumulative timestamp and append hop-by-hop latency
. Examine the mirrored information from packets of the same flow in
an offline collector to determine any latency violations in near-
real-time
Benefits:
. Only *one* additional fields needed in the packet for cumulative
timestamp
. No MTU change as packet traverses multiple hops leading to
deterministic performance
. Applicable to switches/routers as well as network functions
5. Summarizing information in Telemetry header
Delivering every packet of the flow with detailed per hop telemetry
information, for example through flow mirroring, to the central
management system may not scale for certain use cases. Hence, the
ability to convey summarized monitoring information to the central
management entity, e.g. alert the central management system only
when a programmable percentile (for example 99.9th) queue depth
exceeds a high threshold for a flow is needed [trumpet-dc]
[pingmesh-sla]. A possible element for performing this function
could be vSwitch; this function could be potentially implemented in
the NIC too or this could be a implemented as a separate process in
linux-based server. This needs a common a policy definition across
the monitoring summarizer element and the central management entity.
The following describes fields needed in the packet header which is
conveyed from the summarizing entity to the central management
entity; this example focusses on queue depth, but can be easily
extended to other monitoring parameters such as latency etc.
Telemetry packet structure from monitoring summarizer element to
central management entity for conveying queue depth information
Krishnan Expires April 2014 [Page 13]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
- Monitoring summarizer L2/L3/L4 header (TCP/UDP packet; SIP is
monitoring summarizer node, DIP is central management entity
- L2/L3/L4 Flow Header - 256 bytes - from original packet, covers
IPV6 also
- Monitoring summarizer node id - 6 bytes (unique mac address)
- Monitoring summarizer policy id - 4 bytes - common policy
definition across monitoring summarizer element and the central
management entity
oExample policy: Programmable percentile (for example 99.9th)
queue depth exceeds a pre-programmed high threshold
- No. of nodes where this policy was violated - 3 bits - sufficient
to cover 3 Tier DC topology
- Contiguous List of nodes where this policy was violated - each
entry has this structure
oViolating node id - 6 bytes (unique mac address)
oIngress packet interface id in violating node id - 2 bytes
oEgress packet interface id in violating node id - 2 bytes
6. IANA Considerations
This draft does not have any IANA considerations.
7. Security Considerations
Flexibility must be provided to preserve/strip the in-band telemetry
information across multiple operator domains to address privacy
concerns.
8. Acknowledgements
The authors would like to thank Anoop Ghanwani, Jack Harwood from
Dell EMC and Mukesh Hira, Sumit Verdi from VMware for all the
discussions.
Krishnan Expires April 2014 [Page 14]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
9. References
9.1. Normative References
9.2. Informative References
[RFC 2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels," March 1997.
[RFC 6291] Andersson, L. et al., "Guidelines for the Use of the
"OAM" Acronym in the IETF," June 2011
[p4-in-band] "In-band Network Telemetry (INT)," http://p4.org/wp-
content/uploads/fixed/INT/INT-current-spec.pdf
[ietf-twamp] "A Two-Way Active Measurement Protocol (TWAMP)," RFC
5357
[ietf-in-band-dpp] "Data-plane probe for in-band telemetry
collection," https://tools.ietf.org/html/draft-lapukhov-dataplane-
probe-01
[ietf-sfc-monitor] "Network Service Header KPI Stamping,"
https://datatracker.ietf.org/doc/draft-browne-sfc-nsh-kpi-stamp/
[ietf-nsh] "Network Service Header,"
https://datatracker.ietf.org/doc/draft-ietf-sfc-nsh/?include_text=1
[odl-nsh] "Creating a Service Plane using NSH,"
https://www.opennetworking.org/images/stories/downloads/sdn-
resources/IEEE-papers/service-function-chaining.pdf
[ietf-geneve] "Geneve: Generic Network Virtualization
Encapsulation," https://datatracker.ietf.org/doc/draft-ietf-nvo3-
geneve/
[trumpet-dc] "Trumpet: Timely and Precise Triggers in Data Centers,"
http://www.cs.yale.edu/homes/yu-minlan/writeup/sigcomm16.pdf
[pingmesh-sla] "Pingmesh: A Large-Scale System for Data Center
Network Latency Measurement and Analysis,"
http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p139.pdf
Authors' Addresses
Ram (Ramki) Krishnan
Support Vectors
Krishnan Expires April 2014 [Page 15]
Internet-Draft In-band Telemetry - SLA Monitoring September 2013
Fremont, CA
Email: ramkri123@gmail.com
Krishnan Expires April 2014 [Page 16]
