| Internet-Draft | QUIC Multipath Questions | January 2021 |
| Dawkins | Expires 21 July 2021 | [Page] |
The IETF QUIC working group has been chartered to produce extensions that would "enable ... multipath capabilities" since the working group was formed in 2016, but because multipath was an extension, work on multipath, and the other extensions named in the charter, waited while work proceeded on the core QUIC protocol specifications.¶
After the QUIC working group chairs requested publication for the core QUIC protocol specifications, they scheduled a virtual interim meeting to understand the use cases that various groups inside and outside the IETF were envisioning for multipath with QUIC.¶
As part of that discussion, it became obvious that people had a variety of ideas about how multiple paths would be used, because they weren't looking at the same use cases, and so had different assumptions about how applications might use QUIC over multiple paths.¶
This document is intended to capture questions that have come up in discussions, with some suggested answers, to inform further discussion in the working group.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 21 July 2021.¶
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
The IETF QUIC working group has been chartered to produce extensions that would "enable ... multipath capabilities" ([QUIC-charter]) since the working group was formed in 2016, but because multipath was an extension, work on multipath, and the other extensions named in the charter, waited while work proceeded on the core QUIC protocol specifications ([I-D.ietf-quic-transport] and related specifications).¶
After the QUIC working group chairs requested publication for the core QUIC protocol specifications, they scheduled a virtual interim meeting ([QUIC-interim-20-10]) to understand the use cases that various groups inside and outside the IETF were envisioning for multipath with QUIC.¶
As part of that discussion, which continued on the QUIC working group mailing list and at IETF 109 [QUIC-IETF-109-minutes], it became obvious that people had a variety of ideas about how multiple paths would be used, because they weren't looking at the same use cases, and so had different assumptions about how QUIC applications might use multiple paths.¶
This document is intended to capture questions that have come up in discussions, with some suggested answers, to inform further discussion in the working group.¶
The story that might have been titled "The People Unable to See and the Elephant" [Elephant-By-Touch] in 2021, is from the Indian subcontinent, and is at least 2500 years old (it is recorded in the Buddhist text Udana 6.4 [Udana-6-4]). The story, which has spread widely throughout the world, is about people unable to see, who are brought before the king, and shown an elephant - but they only touched the part of the elephant they were each standing in front of, and then began to argue about what an elephant was like (in [Udana-6-4]. Their descriptions were that "an elephant is like" a jar, a basket, a plowshare, the pole of a plow, a granary, a post, a mortar, a pestle, or a broom.¶
None were completely wrong (they had their hands on the part of the elephant they were right about), but none were completely right, either (an elephant was more than the parts they hand their hands on). And the king pointed to the monks of different sects who had been arguing about the teachings of the Buddha, and said, "they keep on arguing, quarreling, and disputing, wounding one another with weapons of the mouth, saying, 'The teaching is like this, it's not like that. The teaching is not like that, it's like this.'"¶
Let us turn our attention from "describing an elephant", to "describing what multiple paths in QUIC might mean",¶
We are not unable to see the parts of QUIC over multiple paths that we have touched, but we can't agree on the shape of multipath for QUIC without understanding what other people are touching.¶
It cannot be emphasized enough that multiple proposals for "QUIC multipath" have been submitted to the IETF (at a minimum, [I-D.deconinck-quic-multipath], [I-D.an-multipath-quic] and [I-D.an-multipath-quic-application-policy], [I-D.liu-multipath-quic], and [I-D.huitema-quic-mpath-option]). In this document, "QUIC multipath" means only "QUIC using multiple paths", and draws from the discussion in the QUIC working group mailing list, some of which has been guided by one or more of the current proposals.¶
This document does reuse some terminology from [I-D.dawkins-quic-what-to-do-with-multipath}}, especially "Traffic Switching" and "Traffic Splitting", which are summarized in Section 3.2.¶
This document is an informational Internet-Draft, not adopted by any IETF working group, and does not carry any special status within the IETF.¶
Please note well that this document reflects the author's current understanding of working group discussions. It is likely that there are more questions than currently included in the document, and it is even more likely that some of the suggested answers are incomplete or (unlike the people in Section 1.1) completely wrong. Contributions that add or improve questions and answers, or suggest improvements and clarifications, are welcomed, as described in Section 1.4.¶
(Note to RFC Editor - if this document ever reaches you, please remove this section)¶
This document is under development in the Github repository at https://github.com/SpencerDawkins/draft-dawkins-quic-multipath-questions. Readers are invited to open issues and send pull requests with contributed text for this document, but since the document is intended to guide discussion for the QUIC working group, substantial discussion of this document should take place on the QUIC working group mailing list (quic@ietf.org). Subscription and archive details are at https://www.ietf.org/mailman/listinfo/quic.¶
We've had considerable discussion about the details of multipath implementation in QUIC, but it's worth starting out with a couple of metaquestions:¶
The most compelling explanation of the point of view that we don't need multipath is likely the [Chromium-Multipath] presentation given at [QUIC-interim-20-10]. The short summary of this presentation was that Google had included multipath in planning for their gQUIC implementation, but subsequently stopped working on this, choosing instead to focus on making connection migration work, given that this was what customers were asking for.¶
As of the date of [QUIC-interim-20-10], few if any implementations were using connection migration, which was defined in [I-D.ietf-quic-transport]. So there was a well-supported point of view that the right thing to do was to get deployment experience with connection migration, and then see what use cases remain that could not be supported using connection migration.¶
Suggested answer to MQ-01: "Maybe".¶
We noted in discussions that [I-D.ietf-quic-transport] allows an implementation to open two or more connections on different paths, verify the paths, and even probe to ensure that a path is still valid before migrating from one connection to another. When an implementation does this proactively, there is no connection establishment delay when the implementation migrates from one path to another. So it's a reasonable question to ask, "do we already have multipath support in QUIC?"¶
For some applications, yes, but this is highly application-dependent.¶
The key point in discussion was that this level of support allows rapid migration from one path to another (what [I-D.dawkins-quic-what-to-do-with-multipath] calls "traffic switching"}, but does not allow sustained simultaneous use of multiple paths (what [I-D.dawkins-quic-what-to-do-with-multipath] calls "traffic splitting").¶
Even for traffic switching, the QUIC implementation does not know the capacity of the path that the connection migrates to, and must learn it, and this means that a successful connection migration can lead to a detectable difference in performance if the migrated-from connection was carrying a significant amount of traffic.¶
Suggested answer to MQ-02: "For applications that perform traffic switching, possibly so. For applications that perform traffic splitting, no".¶
Once discussion moves beyond the metaquestions in Section 2, questions remain.¶
Please note that this document only includes high-level questions. There have also been discussions about topics like whether a single sequence number space is shared across paths, etc. These questions aren't included in this document, even though they're important.¶
Based on mailing list discussion, the answer is almost certainly "No". "Multipath" is at best an umbrella term covering a variety of use cases that can make use of multiple paths in a variety of ways, to accomplish a variety of goals. Some examples are listed in Section 3.7.¶
If an application can use connections over multiple paths independently from each other, the application could possibly make use of multiple paths, without any "multipath" support in QUIC at all. But this is highly application-dependent.¶
We noted that there are many strategies for using multiple paths (some, but certainly not all, described in [I-D.dawkins-quic-what-to-do-with-multipath]). It's worth also noting that these strategies may not have much in common with each other. There are use cases that expect to move from path to path, depending on the relative financial costs of those paths. Other use cases expect to move from path to path depending on the measured RTT of each path, picking a path with the lowest measured RTT. Still others may expect to use multiple paths for a single connection, if the measured path characteristics are "sufficiently similar" to each other.¶
Suggested answer to Q-HowMany: "It's not clear that one multipath scheduler can support all of these use cases".¶
This question seems to revolve around two sub-questions:¶
From a code reuse perspective, we note that multipath is complicated enough to get right that depending on applications to get multipath right isn't desirable, although it's unavoidable. One reason QUIC was originally chartered to do multipath was because we noticed that both TCP [RFC0793] and SCTP [RFC4960] were retrofitted with support for some aspects of multipath operation ([RFC8684] added multipath support for TCP, and [RFC5061] added fast failover to another path for SCTP).¶
We have been successful when we've included Traffic Switching in transport protocols, but we've only been successful including Traffic Splitting in transport protocols when relevant path characteristics have been similar between various paths, and the use case involved traffic that didn't require the transport to distinguish between different types of application-level information when selecting an appropriate path. If an application wants to send some types of video frames over one path and other types over a different path, that's not going to be easy to do using a general-purpose transport mechanism. And while the Internet is more than the World Wide Web, we noted that the World Wide Web has been moving away from bulk transfers for some time, so that user-perceived latency matters much more than bandwidth utilization.¶
Suggested answer to Q-Transport: "Maybe. But it's important to have realistic expectations. We will likely end up with multiple schedulers, and we may very well end up with applications that can't use any of the schedulers we describe, and have to handle multiple paths on their own".¶
[QUIC-charter] has focused on supporting HTTP since the working group was chartered in 2016. Although it is certainly possible for applications to use [I-D.ietf-quic-transport] as their interface to the QUIC protocol, that's not what most applications we have implementation and deployment experience with have used, and the recent discussions in the MASQUE working group [MASQUE-charter] has pointed towards reusing HTTP/3, even for a tunneling and proxying application that may not make much use of HTTP beyond connection set-up. Some use cases include tunneling, so are reasonable candidates to use HTTP/3 to set up MASQUE tunnels anyway.¶
Suggested answer to Q-HTTP3: "Probably so, especially if the goal is to provide a multipath capability for QUIC without duplicating testing that has already been carried out at scale for HTTP/3".¶
The QUIC transport protocol is so closely tied to HTTP/3 that both [I-D.ietf-quic-transport] and [I-D.ietf-quic-http] were specified in the same working group ([QUIC-charter]), and one result of this close relationship is that QUIC is an asymmetric client-server transport protocol, not a symmetric "host to host" transport protocol. Each connection is initiated by a client endpoint, sending to a server endpoint.¶
This is a very reasonable architecture, for a transport protocol targeting use by HTTP, a client-server application protocol, and this fits well into the current Internet architecture, which (for better or worse) includes middleboxes that enforce this directionality. These middleboxes include firewalls that force communications to be initiated from "inside" a network perimeter, which works fine when the goal is to provide clients "inside" access to servers "outside", and Network Address Translators often enforce similar restrictions, so that a packet arriving at the NAT from an unassigned address "inside" the network will be bound to an address "outside" the network, but a packet arriving from an unassigned address "outside" the network will simply be dropped.¶
If HTTP/3 is the driving use case for QUIC multipath, the client-server nature of the QUIC transport protocol is fine. If the goal is to support other applications, especially bidirectional application protocols, we need to think about this in more detail.¶
We noted that QUIC connection migration isn't entirely client-server - a server can send the client a preferred_address transport parameter during the initial handshake. This capability is limited, in that it can only happen once, early in the life of a connection, and the client might not actually migrate to the preferred address. Even if the server does want to migrate the connection, often the client must be the one to initiate that communication because of the previously noted middlebox constraints.¶
Suggested answer for Q-Symmetric: "Need a clearer understanding of the applications that will make use of QUIC multipath, in order to know this".¶
We've had at least two starting points for people in these discussions - one that expects traffic to be delivered in-order to applications, and one that expects applications to handle their own out-of-order delivery, since the sending application needs to track what's actually being delivered, including being delivered significantly out of order, in order to select a path more effectively.¶
For latency-sensitive applications, it's likely that out-of-order delivery across paths is something the application would want to be aware of. And it's worth noting that we're chatting about the desirability of reliable streams, versus partially reliable streams, versus datagrams, in at least a couple of use cases, and partial reliability isn't part of streams in [I-D.ietf-quic-transport], so if we think partially reliable streams are the right answer, there's still some specification work to do.¶
It's also worth mentioning that repair strategies for reordering across multiple paths may be closer to research than to engineering. Some helpful perspectives are provided in [I-D.amend-iccrg-multipath-reordering].¶
Suggested answer for Q-Reorder: "Answer unclear".¶
This has come up most often in private conversations, but I've heard concerns about our ability to measure whether we are making the best use of multiple paths at scale, in a reproducible way.¶
It seems obvious that this question is important, and while it may be early in the process to expect a detailed answer, this question is recorded in this document so that we don't lose track of it.¶
Suggested answer for Q-CB: "Please check back later".¶
At least one point of view is that different use cases have different goals. Some goals that were shared in the QUIC multipath virtual interim ((QUIC-interim-20-10}} include¶
Other goals certainly exist, including goals described in [I-D.an-multipath-quic-application-policy] and [I-D.bonaventure-iccrg-schedulers], and it's likely that even this set doesn't include all possible strategies. It's also worth noting that few of the presentations given at [QUIC-interim-20-10] shared a common set of even two or three strategies.¶
Note that actual use cases may be more nuanced about their goals - for instance, sending low-latency traffic over the lowest-latency path, and then using all remaining available bandwidth across all paths for other traffic.¶
It will be very helpful if we can look for a small number of simple concepts that would allow a small number of schedulers to meet most application requirements. The alternative - adding schedulers every time someone thinks of a new strategy - is too painful to contemplate.¶
Suggested answer for Q-Goals: "We sure hope we can narrow the set of goals down to something manageable".¶
We've noted a number of times that the most useful features of protocols won't be used if there is no way for an application to use them.¶
HTTP/2 stream priorities [RFC7540] was mentioned frequently, but other features were mentioned.¶
We need a better understanding of the goals (see Section 3.7 to answer this question, but it's included in this document so we don't forget it.¶
Suggested answer for Q-API: "Please check back later".¶
This document does not make any request to IANA.¶
QUIC-specific security considerations are discussed in Section 21 of [I-D.ietf-quic-transport].¶
Section 6 of [I-D.ietf-quic-datagram] discusses security considerations specific to the use of the Unreliable Datagram Extension to QUIC.¶
Some multipath QUIC-specific security considerations can be found in Section 8 of the individual draft [I-D.deconinck-quic-multipath]. The other QUIC multipath proposals named in Section 1.3 have "to be determined" security considerations sections, which is not unusual for individual Internet-Drafts.¶
I'd like to thank Lars Eggert and Lucas Pardue, the QUIC working group chairs, who called the QUIC virtual interim meeting on multipath.¶
I'd also like to thank the presenters at the QUIC virtual interim, who put together valuable presentations on short notice.¶
Mikkel Fahnoee Joergensen, Lars Eggert, and Mike Bishop provided thoughts about symmetrical multipath QUIC that informed Section 3.4.¶
Christian Huitema made a plea for sanity about the number of goals for multipath QUIC, that is now included in Section 3.7.¶
Many thanks to (your name could easily appear here) for reviews and comments.¶
I've been through the QUIC working group archives on multipath discussions, along with the minutes from [QUIC-interim-20-10] and IETF 109 [QUIC-IETF-109-minutes], and too many people have commented in those venues for me to list them all. My apologies for that, but thank you all for contributing.¶
And it's worth noting that the story described in Section 1.1 included the people who were trying to describe the elephant beating other people who disagreed with their description. Thanks for not going there in the QUIC working group.¶