| TSVWG Working Group | G. Fairhurst |
| Internet-Draft | University of Aberdeen |
| Intended status: Best Current Practice | July 20, 2015 |
| Expires: January 21, 2016 |
Network Transport Circuit Breakers
draft-ietf-tsvwg-circuit-breaker-02
This document explains what is meant by the term "network transport circuit breaker" (CB). It describes the need for circuit breakers when using network tunnels, and other non-congestion controlled applications. It also defines requirements for building a circuit breaker and the expected outcomes of using a circuit breaker within the Internet.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 21, 2016.
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
A network transport Circuit Breaker (CB) is an automatic mechanism that is used to estimate congestion caused by a flow, and to terminate (or significantly reduce the rate of) the flow when persistent congestion is detected. This is a safety measure to prevent congestion collapse (starvation of resources available to other flows), essential for an Internet that is heterogeneous and for traffic that is hard to predict in advance.
The term "Circuit Breaker" originates in electricity supply, and has nothing to do with network circuits or virtual circuits. In electricity supply, a CB is intended as a protection mechanism of last resort. Under normal circumstances, a CB ought not to be triggered; It is designed to protect the supply network and attached equipment when there is overload. Just as people do not expect the electrical circuit-breaker (or fuse) in their home to be triggered, except when there is a wiring fault or a problem with an electrical appliance.
In networking, the CB principle can be used as a protection mechanism of last resort to avoid persistent congestion. Persistent congestion (also known as "congestion collapse") was a feature of the early Internet of the 1980s. This resulted in excess traffic starving other connection from access to the Internet. It was countered by the requirement to use congestion control (CC) by the Transmission Control Protocol (TCP) [Jacobsen88] [RFC1112]. These mechanisms operate in Internet hosts to cause TCP connections to "back off" during congestion. The introduction of CC in TCP (currently documented in [RFC5681] ensured the stability of the Internet, because it was able to detect congestion and promptly react. This worked well while TCP was by far the dominant traffic in the Internet, and most TCP flows were long-lived (ensuring that they could detect and respond to congestion before the flows terminated). This is no longer the case, and non-congestion controlled traffic, including many applications of the User Datagram Protocol (UDP) can form a significant proportion of the total traffic traversing a link. The current Internet therefore requires that non-congestion controlled traffic needs to be considered to avoid congestion collapse.
There are important differences between a transport circuit-breaker and a congestion-control method. Specifically, congestion control (as implemented in TCP, SCTP, and DCCP) needs to operate on the timescale on the order of a packet round-trip-time (RTT), the time from sender to destination and return. Congestion control methods may react to a single packet loss/marking and reduce the transmission rate for each loss or congestion event. The goal is usually to limit the maximum transmission rate that reflects the available capacity of a network path. These methods typically operate on individual traffic flows (e.g., a 5-tuple).
In contrast, CBs are recommended for non-congestion-controlled Internet flows and for traffic aggregates, e.g., traffic sent using a network tunnel. Later sections provide examples of cases where circuit-breakers may or may not be desirable.
A CB needs to measure (meter) the traffic to determine if the network is experiencing congestion and must be designed to trigger robustly when there is persistent congestion. This means the trigger needs to operate on a timescale much longer than the path round trip time (e.g., seconds to possibly many tens of seconds). This longer period is needed to provide sufficient time for transports (or applications) to adjust their rate following congestion, and for the network load to stabilise after any adjustment. A CB trigger will often be based on a series of successive sample measurements taken over a reasonably long period of time. This is to ensure that a CB does not accidentally trigger following a single (or even successive) congestion events (congestion events are what triggers congestion control, and are to be regarded as normal on a network link operating near its capacity). Once triggered, a control function needs to remove traffic from the network, either disabling the flow or significantly reducing the level of traffic. This reaction provides the required protection to prevent persistent congestion being experienced by other flows that share the congested part of the network path.
Section 4 defines requirements for building a circuit breaker.
There are various forms of network transport circuit breaker. These are differentiated mainly on the timescale over which they are triggered, but also in the intended protection they offer:
Examples of each type of circuit breaker are provided in section 4.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
Although circuit breakers have been talked about in the IETF for many years, there has not yet been guidance on the cases where circuit breakers are needed or upon the design of circuit breaker mechanisms. This document seeks to offer advise on these two topics.
Section 3.1 describes the functional components of a circuit breaker and section 3.2 defines requirements for implementing a circuit breaker.
The basic design of a transport circuit breaker involves communication between an ingress point (a sender) and an egress point (a receiver) of a network flow. A simple picture of CB operation is provided in figure 1. This shows a set of routers (each labelled R) connecting a set of endpoints. A CB is used to control traffic passing through a subset of these routers, acting between an ingress and a egress point. In some cases, the ingress and egress may be within one or both network endpoints, in other cases they will be within a network device. For example, one expected use would be at the ingress and egress of a tunnel service.
+--------+ +--------+
|Endpoint| |Endpoint|
+--+-----+ +--+-----+
| |
| +-+ +-+ +---------+ +-+ +-+ +-+ +--------+ +-+ +-+ |
+-+R+--+R+--+ Ingress +--+R+--+R+--+R+--+ Egress |--+R+--+R+-+
+++ +-+ +-------+-+ +-+ +-+ +-+ +-----+--+ +++ +-+
| ^ | | |
+-+ | | +----+----+ | | +-+
+R+--+ | | Measure +<-------------------+ +--+R+
+++ | +----+----+ +++
| | | |
| | +----+----+ |
+--+-----+ | | Trigger + +--+-----+
|Endpoint| | +----+----+ |Endpoint|
+--------+ | | +--------+
+------+
Reaction
Figure 1: A CB controlling the part of the end-to-end path between an ingress point and an egress point.
The set of components needed to implement a circuit breaker are:
The requirements for implementing a CB are:
A CB can be used to control uni-directional UDP traffic, providing that there is a control path to connect the functional components at the Ingress and Egress. This control path can exist in networks for which the traffic flow is purely unidirectional (e.g., a multicast stream that sends packets across an Internet path and can use multicast routing to prune flows to shed network load).
Some paths are provisioned using a control protocol, e.g., flows provisioned using the Multi-Protocol Label Switching (MPLS) services, path provisioned using the Resource reservation protocol (RSVP), or admission-controlled Differentiated Services. For these paths the control protocol may be invoked to shed the network load when the circuit breaker is triggered.
There are multiple types of CB that may be defined for use in different deployment cases. This section provides examples of different types of circuit breaker:
A fast-trip circuit breaker is the most responsive form of CB. It has a response time that is only slightly larger than that of the traffic it controls. It is suited to traffic with well-understood characteristics. It is not be suited to arbitrary network traffic, since it may prematurely trigger (e.g., when multiple congestion-controlled flows lead to short-term overload).
A set of fast-trip CB methods have been specified for use together by a Real-time Transport Protocol (RTP) flow using the RTP/AVP Profile [RTP-CB]. It is expected that, in the absence of severe congestion, all RTP applications running on best-effort IP networks will be able to run without triggering these circuit breakers. A fast-trip RTP CB is therefore implemented as a fail-safe.
The sender monitors reception of RTCP Reception Report (RR or XRR) packets that convey reception quality feedback information. This is used to measure (congestion) loss, possibly in combination with ECN [RFC6679].
The CB action (shutdown of the flow) is triggered when any of the following trigger conditions are true:
A slow-trip CB may be implemented in an endpoint or network device. This type of CB is much slower at responding to congestion than a fast-trip CB and is expected to be more common.
One example where a slow-trip CB is needed is where flows or traffic-aggregates use a tunnel or encapsulation and the flows within the tunnel do not all support TCP-style congestion control (e.g., TCP, SCTP, TFRC), see [RFC5405] section 3.1.3. A use case is where tunnels are deployed in the general Internet (rather than "controlled environments" within an ISP or Enterprise), especially when the tunnel may need to cross a customer access router.
A managed CB is implemented in the signalling protocol or management plane that relates to the traffic aggregate being controlled. This type of circuit breaker is typically applicable when the deployment is within a "controlled environment".
A Circuit Breaker requires more than the ability to determine that a network path is forwarding data, or to measure the rate of a path - which are often normal network operational functions. There is an additional need to determine a metric for congestion on the path and to trigger a reaction when a threshold is crossed that indicates persistent congestion.
[RFC4553], SAToP Pseudo-Wires (PWE3), section 8 describes an example of a managed circuit breaker for isochronous flows.
If such flows were to run over a pre-provisioned (e.g., MPLS) infrastructure, then it may be expected that the Pseudo-Wire (PW) would not experience congestion, because a flow is not expected to either increase (or decrease) their rate. If instead Pseudo-Wire traffic is multiplexed with other traffic over the general Internet, it could experience congestion. [RFC4553] states: "If SAToP PWs run over a PSN providing best-effort service, they SHOULD monitor packet loss in order to detect "severe congestion". The currently recommended measurement period is 1 second, and the trigger operates when there are more than three measured Severely Errored Seconds (SES) within a period.
If such a condition is detected, a SAToP PW should shut down bidirectionally for some period of time...". The concept was that when the packet loss ratio (congestion) level increased above a threshold, the PW was by default disabled. This use case considered fixed-rate transmission, where the PW had no reasonable way to shed load.
The trigger needs to be set at the rate that the PW was likely to experience a serious problem, possibly making the service non-compliant. At this point, triggering the CB would remove the traffic preventing undue impact on congestion-responsive traffic (e.g., TCP). Part of the rationale, was that high loss ratios typically indicated that something was "broken" and ought to have already resulted in operator intervention, and therefore need to trigger this intervention.
An operator-based response provides opportunity for other action to restore the service quality, e.g., by shedding other loads or assigning additional capacity, or to consciously avoid reacting to the trigger while engineering a solution to the problem. This may require the trigger to be sent to a third location (e.g., a network operations centre, NOC) responsible for operation of the tunnel ingress, rather than the tunnel ingress itself.
A CB is not required for a single CC-controlled flow using TCP, SCTP, TFRC, etc. In these cases, the CC methods are already designed to prevent congestion collapse.
One common question is whether a CB is needed when a tunnel is deployed in a private network with pre-provisioned capacity?
In this case, compliant traffic that does not exceed the provisioned capacity ought not to result in congestion collapse. A CB will hence only be triggered when there is non-compliant traffic. It could be argued that this event ought never to happen - but it may also be argued that the CB equally ought never to be triggered. If a CB were to be implemented, it will provide an appropriate response if persistent congestion occurs in an operational network. Implementing a CB will not reduce the performance of the flows, but offers protection in the event that persistent congestion occurs.
IP-based traffic is generally assumed to be congestion-controlled, i.e., it is assumed that the transport protocols generating IP-based traffic at the sender already employ mechanisms that are sufficient to address congestion on the path [RFC5405]. A question therefore arises when people deploy a tunnel that is thought to only carry an aggregate of TCP (or some other CC-controlled) traffic: Is there advantage in this case in using a CB?
For sure, traffic in a such a tunnel will respond to congestion. However, the answer to the question may not be obvious, because the overall traffic formed by an aggregate of flows that implement a CC mechanism does not necessarily prevent congestion collapse. For instance, most CC mechanisms require long-lived flows to react to reduce the rate of a flow, an aggregate of many short flows may result in many terminating before they experience congestion. It is also often impossible for a tunnel service provider to know that the tunnel only contains CC-controlled traffic (e.g., Inspecting packet headers may not be possible). The important thing to note is that if the aggregate of the traffic does not result in persistent congestion (impacting other flows), then the CB will not trigger. This is the expected case in this context - so implementing a CB will not reduce performance of the tunnel, but offers protection in the event that persistent congestion occur.
A one-way forwarding path could have no associated control path, and therefore cannot be controlled using an automated process. This service could be provided using a path that has dedicated capacity and does not share this capacity with other elastic Internet flows (i.e., flows that vary their rate).
When capacity is shared, one way to mitigate the impact on other flows is to manage the traffic envelope by using ingress policing.
Supporting this type of traffic in the general Internet requires operator monitoring to detect and respond to persistent congestion.
All circuit breaker mechanisms rely upon coordination between the ingress and egress meters and communication with the trigger function. This is usually achieved by passing network control information across the network. Timely operation of a circuit breaker depends on the choice of measurement period. If the receiver has an interval that is overly long, then the responsiveness of the circuit breaker decreases. This impacts the ability of the circuit breaker to detect and react to congestion.
Mechanisms need to be implemented to prevent attacks on the network control information that would result in Denial of Service (DoS). The source and integrity of control information (measurements and triggers) MUST be protected from off-path attacks. Without protection, it may be trivial for an attacker to inject packets with values that could prematurely trigger a circuit breaker resulting in DoS. Simple protection can be provided by using a randomised source port, or equivalent field in the packet header (such as the RTP SSRC value and the RTP sequence number) expected not to be known to an off-path attacker. Stronger protection can be achieved using a secure authentication protocol.
Transmission of network control information consumes network capacity. This control traffic needs to be considered in the design of a circuit breaker and could potentially add to network congestion. If this traffic is sent over a shared path, it is RECOMMENDED that this control traffic is prioritized to reduce the probability of loss under congestion. Control traffic also needs to be considered when provisioning a network that uses a circuit breaker.
The circuit breaker MUST be designed to be robust to packet loss that can also be experienced during congestion/overload. Loss of control traffic may be a side-effect of a congested network, but also may arise from other causes.
Each design of a circuit breaker must evaluate whether the particular circuit breaker mechanism has new security implications.
This document makes no request from IANA.
There are many people who have discussed and described the issues that have motivated this draft. Contributions and comments included: Lars Eggert, Colin Perkins, David Black, Matt Mathis and Andrew McGregor. This work was part-funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700).
XXX RFC-Editor: Please remove this section prior to publication XXX
Draft 00
This was the first revision. Help and comments are greatly appreciated.
Draft 01
Contained clarifications and changes in response to received comments, plus addition of diagram and definitions. Comments are welcome.
WG Draft 00
Approved as a WG work item on 28th Aug 2014.
WG Draft 01
Incorporates feedback after Dallas IETF TSVWG meeting. This version is thought ready for WGLC comments.
WG Draft 02
Minor fixes for typos. Rewritten security considerations section.
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997. |
| [RFC3168] | Ramakrishnan, K., Floyd, S. and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001. |
| [RFC5405] | Eggert, L. and G. Fairhurst, "Unicast UDP Usage Guidelines for Application Designers", BCP 145, RFC 5405, November 2008. |
| [Jacobsen88] | European Telecommunication Standards, Institute (ETSI), "Congestion Avoidance and Control", SIGCOMM Symposium proceedings on Communications architectures and protocols", August 1998. |
| [RFC1112] | Deering, S., "Host extensions for IP multicasting", STD 5, RFC 1112, August 1989. |
| [RFC4553] | Vainshtein, A. and YJ. Stein, "Structure-Agnostic Time Division Multiplexing (TDM) over Packet (SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006. |
| [RFC5681] | Allman, M., Paxson, V. and E. Blanton, "TCP Congestion Control", RFC 5681, DOI 10.17487/RFC5681, September 2009. |
| [RFC6679] | Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P. and K. Carlberg, "Explicit Congestion Notification (ECN) for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 2012. |
| [RTP-CB] | Perkins, and Singh, "Multimedia Congestion Control: Circuit Breakers for Unicast RTP Sessions", February 2014. |