Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Problem Statement
draft-briscoe-tsvwg-aqm-tcpm-rmcat-l4s-problem-00

Abstract

This document motivates a new service that the Internet could provide to eventually replace best efforts for all traffic: Low Latency, Low Loss, Scalable throughput (L4S). It is becoming common for all (or most) applications being run by a user at any one time to require low latency, but the only solution the IETF can offer for ultra-low queuing latency is Diffserv, which only offers low latency for some packets at the expense of others. Diffserv has also proved hard to deploy widely end-to-end.

In contrast, a zero-config incrementally deployable solution has been demonstrated that keeps average queuing delay under a millisecond for all applications even under very heavy load; and it keeps congestion loss to zero. At the same time it solves the long-running problem with the scalability of TCP throughput. Even with a high capacity broadband access, the resulting performance under load is remarkably and consistently improved for applications such as interactive video, conversational video, voice, Web, gaming, instant messaging, remote desktop and cloud-based apps. This document explains the underlying problems that have been preventing the Internet from enjoying such performance improvements. It then outlines the parts necessary for a solution and the steps that will be needed to standardize them.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on December 5, 2016.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 1.1. The Application Performance Problem
• 1.2. The Technology Problem
• 1.3. Terminology
• 1.4. The Standardization Problem
• 2. Rationale
• 2.1. Why These Primary Components?
• 2.2. Why Not Alternative Approaches?
• 3. Opportunities
• 3.1. Use Cases
• 4. IANA Considerations
• 5. Security Considerations
• 5.1. Traffic (Non-)Policing
• 5.2. 'Latency Friendliness'
• 5.3. ECN Integrity
• 6. Acknowledgements
• 7. References
• 7.1. Normative References
• 7.2. Informative References
• Appendix A. The "TCP Prague Requirements"
• Authors' Addresses

1. Introduction

1.1. The Application Performance Problem

It is increasingly common for all of a user's applications at any one time to require low delay: interactive Web, Web services, voice, conversational video, interactive video, instant messaging, online gaming, remote desktop and cloud-based applications. In the last decade or so, much has been done to reduce propagation delay by placing caches or servers closer to users. However, queuing remains a major, albeit intermittent, component of latency. Low loss is also important because, for interactive applications, losses translate into delays.

It has been demonstrated that, once access network bit rate reaches levels now common in the developed world, increasing capacity offers diminishing returns if latency (delay) is not addressed. Differentiated services (Diffserv) offers Expedited Forwarding [RFC3246] for some packets at the expense of others, but this is not applicable when all (or most) of a user's applications require low latency.

Therefore, the goal is an Internet service with ultra-Low queueing Latency, ultra-Low Loss and Scalable throughput (L4S) - for all traffic. Having motivated the goal of 'L4S for all', this document enumerates the problems that have to be overcome to reach it.

It must be said that queuing delay only degrades performance infrequently [Hohlfeld14]. It only occurs when a large enough capacity-seeking (e.g. TCP) flow is running alongside the user's traffic in the bottleneck link, which is typically in the access network. Or when the low latency application is itself a large capacity-seeking flow (e.g. interactive video). At these times, the performance improvement must be so remarkable that network operators will be motivated to deploy it.

1.2. The Technology Problem

Active Queue Management (AQM) is part of the solution to queuing under load. AQM improves performance for all traffic, but there is a limit to how much queuing delay can be reduced by solely changing the network; without addressing the root of the problem.

The root of the problem is the presence of standard TCP congestion control (Reno [RFC5681]) or compatible variants (e.g. TCP Cubic [I-D.ietf-tcpm-cubic]). We shall call this family of congestion controls 'Classic' TCP. It has been demonstrated that if the sending host replaces Classic TCP with a 'Scalable' alternative, when a suitable AQM is deployed in the network the performance under load of all the above interactive applications can be stunningly improved - even in comparison to a state-of-the-art AQM such as fq_CoDel [I-D.ietf-aqm-fq-codel] or PIE [I-D.ietf-aqm-pie].

It has been convincingly demonstrated [DCttH15] that it is possible to deploy such an L4S service alongside the existing best efforts service so that all of a user's applications can shift to it when their stack is updated. Access networks are typically designed with one link as the bottleneck for each site (which might be a home, small enterprise or mobile device), so deployment at a single node should give nearly all the benefit. Although the main incremental deployment problem has been solved, and the remaining work seems straightforward, there may need to be changes in approach during the process of engineering a complete solution.

There are three main parts to the L4S approach (illustrated in Fig {ToDo: ASCII art of slide 9 from https://riteproject.files.wordpress.com/2015/10/1604-l4s-bar-bof.pdf}):

1. The L4S service needs to be isolated from the queuing latency of the Classic service. However, the two must be able to freely share a common pool of capacity. There is no way to predict how many flows at any one time might use each service and capacity in access networks is too scarce to partition into two. The Dual Queue Coupled AQM is an example of such a 'semi-permeable' membrane [I-D.briscoe-aqm-dualq-coupled]. Per-flow queuing such as in [I-D.ietf-aqm-fq-codel] could be used, but it is rather overkill, which brings disadvantages (see Section 2.2).
2. An identifier is needed to so that L4S and Classic packets can be classified into their separate treatments. [I-D.briscoe-tsvwg-ecn-l4s-id] considers various alternative identifiers, and concludes that all alternatives involve compromises, but the ECT(1) codepoint of the ECN field is a workable solution.
3. Scalable congestion controls already exist. They solve the scaling problem with TCP first pointed out in [RFC3649]. The one used most widely (in controlled environments) is Data Centre TCP (DCTCP [I-D.ietf-tcpm-dctcp]), which has been implemented and deployed in Windows Server Editions (since 2012), in Linux and in FreeBSD. Although DCTCP as-is 'works' well over the public Internet, most implementations lack certain safety features that will be necessary once it is used outside controlled environments like data centres (see later). A similar scalable congestion control will also need to be transplanted into protocols other than TCP (SCTP, RTP/RTCP, RMCAT, etc.)

1.3. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. In this document, these words will appear with that interpretation only when in ALL CAPS. Lower case uses of these words are not to be interpreted as carrying RFC-2119 significance.

Classic service:

The 'Classic' service is intended for all the behaviours that currently co-exist with TCP Reno (e.g. TCP Cubic, Compound, SCTP, etc).

Low-Latency, Low-Loss and Scalable (L4S) service:

The 'L4S' service is intended for traffic from scalable TCP algorithms such as Data Centre TCP. But it is also more general—it will allow a set of congestion controls with similar scaling properties to DCTCP (e.g. Relentless [Mathis09]) to evolve.

Both Classic and L4S services can cope with a proportion of unresponsive or less-responsive traffic as well (e.g. DNS, VoIP, etc).

Scalable Congestion Control:

A congestion control where flow rate is inversely proportional to the level of congestion signals. Then, as flow rate scales, the number of congestion signals per round trip remains invariant, maintaining the same degree of control. With Classic congestion controls such as TCP Reno and Cubic, as capacity increases enable higher flow rates, the number of round trips between signals becomes very large, so control of queuing and/or utilization becomes very slack.

Classic ECN:

The original Explicit Congestion Notification (ECN) protocol [RFC3168].

1.4. The Standardization Problem

1. The first step will be to articulate the structure and interworking requirements of the set of parts that would satisfy the overall application performance requirements.

Then specific interworking aspects of the following three components parts will need to be defined:

1. The L4S service needs to be isolated from the queuing latency of the Classic service. However, the two must be able to freely share a common pool of capacity. There is no way to predict how many flows at any one time might use each service and capacity in access networks is too scarce to partition into two. The Dual Queue Coupled AQM is an example of such a 'semi-permeable' membrane [I-D.briscoe-aqm-dualq-coupled]. Per-flow queuing such as in [I-D.ietf-aqm-fq-codel] could be used, but it has disadvantages, not least that thousands of queues are not needed if two are sufficient.
2. Identifier
   1. [I-D.briscoe-tsvwg-ecn-l4s-id] recommends ECT(1) is used as the identifier to classify L4S and Classic packets into their separate treatments, as required by [RFC4774]. The draft also points out that the experimental assignment of this codepoint as an ECN nonce [RFC3540] will need to be made obsolete (it was never deployed, and it offers no security benefit now that deployment is optional).
   2. An essential aspect of a scalable congestion control is the use of Explicit Congestion Notification (ECN [RFC3168]). 'Classic' ECN requires an ECN signal to be treated the same as a drop, both when it is generated in the network and when it is responded to by hosts. A separate queue for L4S allows the network to support two separate meanings for ECN. And break from this 'same as drop' constraint is an essential feature of a scalable congestion control as well.
3. Scalable congestion controls
   1. Data Centre TCP is being documented in the TCPM WG as an informational record of the protocol currently in use [I-D.ietf-tcpm-dctcp]. It will be necessary to define a number of safety features for a variant usable on the public Internet. A draft list of these, known as the TCP Prague requirements, has been drawn up (see Appendix A).
   2. Transport protocols other than TCP use various congestion controls designed to be friendly with Classic TCP. It will be necessary to implement scalable variants of each of these transport behaviours before they can use the L4S service, by sending packets with the ECT(1) identifier. The following standards track RFCs currently define these protocols: ECN in TCP [RFC3168], in SCTP [RFC4960], in RTP [RFC6679], and in DCCP [RFC4340].
   3. For the case of TCP, the feedback protocol for ECN is too tightly coupled to Classic ECN to be usable for a scalable TCP. Therefore, the implementation of TCP receivers will have to be upgraded [RFC7560]. Work to standardize more accurate ECN feedback for TCP (AccECN [I-D.ietf-tcpm-accurate-ecn]) is already in progress.

2. Rationale

2.1. Why These Primary Components?

{ToDo: /Why/ the various elements are necessary:}

ECN rather than drop

Packet identifier (pretty obvious why)

Scalable congestion notification (host behaviour)

Semi-permeable membrane (network behaviour)

{We will probably move some of the text in the bullets under "The Technology Problem" to here, e.g. why you need capacity shared across the semi-permeable membrane.}

2.2. Why Not Alternative Approaches?

All the following approaches address some part of the same problem space as L4S. In each case, it is shown that L4S complements them or improves on them, rather than being a mutually exclusive alternative:

Diffserv:

Diffserv addresses the problem of bandwidth apportionment for important traffic as well as queuing latency for delay-sensitive traffic. L4S solely addresses the problem of queuing latency. Diffserv will still be necessary where important traffic requires priority (e.g. for commercial reasons, or for protection of critical infrastructure traffic). Nonetheless, if there are Diffserv classes for important traffic, the L4S approach can provide low latency for all traffic within each Diffserv class (including the case where there is only one Diffserv class).

Also, as already explained, Diffserv only works for a small subset of the traffic on a link. It is not applicable when all the applications in use at one time at a single site (home, small business or mobile device) require low latency. Also, because L4S is for all traffic, it needs none of the management baggage (traffic policing, traffic contracts) associated with favouring some packets over others. This baggage has held Diffserv back from widespread end-to-end deployment.

State-of-the-art AQMs:

AQMs such as PIE and fq_CoDel give a significant reduction in queuing delay relative to no AQM at all. The L4S work is intended to complement these AQMs, and we definitely do not want to distract from the need to deploy them as widely as possible. Nonetheless, without addressing the large saw-toothing rate variations of Classic congestion controls, they cannot reduce queuing delay too far without significantly reducing link utilization. The L4S approach resolves this tension by ensuring hosts can minimize the sawtoothing.

Per-flow queuing:

Similarly per-flow queuing is not incompatible with the L4S approach. However, one queue for every flow can be thought of as overkill compared to the minimum of two queues for all traffic needed for the L4S approach. The overkill of per-flow queuing has side-effects:

1. fq makes high performance networking equipment costly (processing and memory) - in contrast dual queue code can be very simple;
2. fq requires packet inspection into the end-to-end transport layer, which doesn't sit well alongside encryption for privacy - in contrast a dual queue, which only operates at the IP layer;
3. fq has to take control of the decisions over which flows are scheduled when - in contrast, in the L4S approach the sender still controls the relative rate of each flow dependent on the needs of each application.

Alternative Back-off ECN (ABE):

Yet again, L4S is not an alternative to ABE but a complement. ABE alters the host behaviour in response to ECN marking to utilize a link better and give ECN flows a faster throughput, but it assumes the network still treats ECN and drop the same. Therefore ABE exploits any lower queuing delay that AQMs can provide. But as explained above, AQMs still cannot reduce queuing delay too far without losing link utilization (for other non-ABE flows).

3. Opportunities

A transport layer that solves the current latency issues will provide new service, product and application opportunities.

If applications can rely on minimal queues in the network, they can focus on reducing their own latency by only minimizing the application send queue. Following existing applications will immediately experience a better quality of experience in the best effort class:

• Gaming
• VoIP
• Video conferencing
• Web browsing
• (Adaptive) Video Streaming

The lower transport layer latency will also allow more interactive application functions offloading to the cloud. If last-minute interactions need to be done locally, more data must be send over the link. When all interactive processing can be done in the cloud, only the info to be rendered to the end user can be sent. It will allow applications such as:

• Cloud based interactive video
• Cloud based virtual and augmented reality

Also lower network layers can finally be further optimized for low latency and stable throughput. Today it is not cost efficient, as the largest part of the traffic (classic best effort) needs to allow "big" queues anyway (up to several 100s of milliseconds) to make classic congestion control work correctly. While technology is known and feasible to support low latency with reliable throughput (even mobile), it is today not considered as economically relevant, as best effort can absorb any burst, delay or throughput variations without end-users experiencing any difference from the normal tay-to-day operation due to congestion control limitations.

3.1. Use Cases

{ToDo: Just bullets below - text to be added by those interested in various use-cases}

Different types of access network: DSL, cable, mobile

The challenges and opportunities with radio links: cellular, Wifi

Private networks of heterogeneous data centres (DC interconnect, multi-tenant cloud, etc)

Different types of transport/app: elastic (TCP/SCTP); real-time (RTP, RMCAT); query (DNS/LDAP).

Avoiding reliance on middleboxes to enable encryption/privacy (because the L4S approach does not look deeper than IP in the network).

4. IANA Considerations

This specification contains no IANA considerations.

5. Security Considerations

5.1. Traffic (Non-)Policing

Because the L4S service can serve all traffic that is using the capacity of a link, it should not be necessary to police access to the L4S service. In contrast, Diffserv has to use traffic policers to limit how much traffic can access each service, otherwise it doesn't work, In turn, traffic policers require traffic contracts between users and networks and between networks. Because L4S will lack all this management complexity, it is more likely to work end-to-end.

During early deployment (and perhaps always), some networks will not offer the L4S service. These networks do not need to police or re-mark L4S traffic - they just forward it unchanged as best efforts traffic, as they would already forward traffic with ECT(1) today. At a bottleneck, such networks will introduce some queuing and dropping. When the scalable congestion controll detects a drop it has to respond as if it is a Classic congestion control, and there will then be no interworking problems.

Certain network operators might choose to restict access to the L4S class, perhaps only to customers who have paid a premium. In the packet classifer, they could identify such customers using some other field (e.g. source address range), and just ignoring the L4S identifier for non-paying customers. This will ensure that the L4S identifier survives end-to-end even though the service does not have to be supported at every hop. Such arrangements would only require simple registered/not-registered packet classification, rather than the complex application-specific traffic contracts of Diffserv.

5.2. 'Latency Friendliness'

The L4S service does rely on self-constraint - not in terms of limiting capacity usage, but in terms of limiting burstiness. It is believed that standardisation of dynamic behaviour (cf. TCP slow-start) and self-interest will be sufficient to prevent transports from sending excessive bursts of L4S traffic, given the application's own latency will suffer most from such behaviour.

Whether burst policing becomes necessary remains to be seen. Without it, there will be potential for attacks on the low latency of the L4S service. However it may only be necessary to apply such policing reactively, e.g. punitively targeted at any deployments of new bursty malware.

5.3. ECN Integrity

{ToDo: Paraphrase discussion from ecn-l4s-id}

6. Acknowledgements

7. References

7.1. Normative References

7.2. Informative References

Appendix A. The "TCP Prague Requirements"

This list of requirements was produced at an ad hoc meeting during IETF-94 in Prague. The list prioritised features that would need to be added to DCTCP to make it safe for use on the public Internet alongside existing non-DCTCP traffic. It also includes features to improve the performance of DCTCP in the wider range of conditions found on the public Internet.

The table is too wide for the ASCII draft format, so it been split into two, with a common column of row index numbers on the left.

Authors' Addresses