0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Kind          |    Length     |     ExID                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Connection ID |
+-+-+-+-+-+-+-+-+
                    

Figure 3: TiU Setup TCP Option

Every TiU SYN or TiU SYN-ACK packet also carries at least the TiU-Setup TCP option. This option contains a Connection ID number. On a SYN packet, it is the Connection ID that the sender intends to use in future packets to represent the Encapsulated Source Port and Encapsulated Destination Port. On a SYN/ACK packet, it confirms that such usage is accepted by the recipient of the SYN. A special value of 255 is used to signify an error, upon which TiU will no longer be used (i.e., the next packet is expected to be a non-encapsulated TCP packet). The TiU-Setup TCP option is defined as follows: [RFC6994]. It has Kind=253, Length=5, an ExID that is with value TBD (see Section 8) and the Connection ID. The Connection ID is an 8-bit field for easier parsing, but only values 0-31 are valid Connection IDs (because the Connection ID in non - SYN or SYN/ACK TiU packets is only 5 bit long).

4. Protocol operation and implementation notes

There can be several ways to implement TCP-in-UDP. The following gives an overview of how a TiU implementation can operate. This description matches the implementation described in Section 7.

A goal of TiU is to achieve congestion control coupling with a simple implementation that minimizes changes to existing code. It is thus recommendable to implement TiU in the kernel, as a change to the existing kernel TCP code. The changes fall in two basic categories:

• Encapsulation and decapsulation: this is code that should, in the simplest case, operate just before a TCP segment is transmitted. Based on e.g. a socket option that enables/disables TiU, the TCP segment is changed into the TiU header format (Figure 1). In case it is a TCP SYN or TCP SYN/ACK packet, the header format is defined as in Figure 2, and the TiU-Setup TCP option is appended. This packet is then transmitted. For decapsulation, the reverse mechanism applies, upon reception of a UDP packet that uses destination port XXX (TBD, see Section 8). Both hosts keep a list of encapsulated TCP port numbers and their corresponding Connection IDs. In case a SYN packet requests using a Connection ID that is already reserved, an error (Connection ID value 255 in the TiU Setup TCP option) must be signified to the other end in a TiU-encapsulated TCP SYN/ACK, and encapsulation must be disabled on all further TCP packets. Similarly, when receiving a TiU SYN/ACK with an error, a TCP sender must stop encapsulating TCP packets.
• Coupled congestion control: this is code that influences the congestion control of TCP. Section 5 describes a simple coupled congestion control algorithm that can be applied to couple TCP connections and assign them a share of the total congestion window that is based on a priority.

The TCP port number space usage on the host is left unchanged: the original code can reserve TCP ports as it always did. Except for the TiU encapsulation compressing the port numbers into a Connection ID field, TCP ports should be used similar to normal TCP operation. A TCP port that is in use by a TiU-encapsulated TCP connection must therefore not be made available to non-encapsulated TCP connections, and vice versa.

For each TCP connection, two variables must be configured: 1) TiU-ENABLE, which is a boolean, deciding whether to use TiU or not, and 2) Priority, which is a value, e.g. from 1 to 10, that is used by the coupled congestion control algorithm to assign an appropriate share of the total cwnd to the connection. Priority values are local and their range does not matter for this algorithm: the algorithm works with a flow's priority portion of the sum of all priority values. The configuration of the two per-connection variables can be implemented in various ways, e.g. through an API option.

With these code changes in place, TiU can operate as follows, assuming no previous TiU connections have been made between a specific host pair and a client tries to connect to a server:

• An application uses an API option to request TiU operation. The kernel then sends out a TiU TCP SYN that contains a TiU-Setup TCP option. This packet header contains the encapsulated TCP port numbers (source port A and destination port B) and the Connection ID X.
• The server listens on UDP port XXX (TBD, see Section 8). Upon receiving a packet on this port, it knows that it is a TiU packet and decodes it, handing the resulting TCP packet over to "normal" TCP processing. The TiU-Setup TCP option allows the server to associate future TiU packets containing Connection ID X with ports A and B. The server sends its response as a TiU SYN-ACK.
• TCP operates as normal from here on, but packets are TiU-encapsulated before sending them out and decapsulated upon reception, using Connection ID X. Both hosts associate TiU packets carrying Connection ID X with a local identifier that matches ports A and B, just like they would associate non-encapsulated TCP packets with the same local identifier when seeing ports A and B in the TCP header.
• If an application on either side of the TiU connection wants to connect to a destination host on the other side and requests TiU operation, the kernel sends out another TiU TCP SYN, this time containing a different TCP source port number and either the same or a different destination port number (C and D), and a TiU-Setup TCP option with Connection ID Y. From now on, packets carrying Connection ID Y will be associated with ports C and D on both hosts. Otherwise, TiU operation continues as described above.
• Now, because there are two or more connections available between the same host pair, coupled congestion control begins to operate for all outgoing TiU packets (see Section 5 for details). This is a local operation, applying the priority values that were configured to use for the TiU-encapsulated TCP connections.

Unless it is known that UDP packets with destination port number XXX (TBD, see Section 8) can be used without problems on the path between two communicating hosts, it is advisable for TiU implementations to contain methods to fall back to non-encapsulated ("raw") TCP communication. Such fall-back must be supported for the case of Connection ID collisions anyway. Middleboxes have been known to track TCP connections [Honda11], and falling back to communication with raw TCP packets without ever using a raw TCP SYN - SYN/ACK handshake may lead to problems with such devices. The following method is recommended to efficiently fall back to raw TCP communication: [RFC6555]), but it can be eliminated by prescribing the processing order as above.

• After sending out a TiU SYN packet, additionally send a raw TCP SYN packet.
• After sending out a TiU SYN/ACK packet, additionally send a raw TCP SYN/ACK packet.
• Upon receiving a TiU SYN packet, after responding with a TiU SYN/ACK packet and raw TCP SYN/ACK packet, immediately store the encapsulated port numbers and Connection ID. As long as a TiU connection is ongoing, ignore any additional incoming TCP SYN or TCP SYN/ACK packets from the same host that carry port numbers matching the stored encapsulated port numbers. Otherwise, process TCP SYN or TCP SYN/ACK packets as normal.

This method ensures that the TCP SYN / SYN/ACK handshake is visible to middleboxes and allows to immediately switch back to raw TCP communication in case of failures. If implemented on both sides as described above and no TiU SYN or TiU SYN/ACK packet arrives, yet a TCP SYN or TCP SYN/ACK packet does, this can only mean that the other host does not support TiU, a UDP packet was dropped, or the UDP and TCP packets were reordered in transit. Reordering in the host (e.g., a server responding to a TCP SYN before it responds to a TiU SYN) can be a problem for similar methods (e.g.

Because TCP does not preserve message boundaries and the size of the TCP header can vary depending on the options that are used, it is also no problem to precede the TCP header in the UDP packet with a different header (e.g. SPUD [I-D.hildebrand-spud-prototype]) without exceeding the known MTU limit. When creating a TCP segment, a TCP sender needs to consider the length of this header when calculating the segment size, just like it would consider the length of a TCP option. For this to work, the usage of other headers such as SPUD in-between the UDP header and the TiU header must therefore be known to both the sender-side and receiver-side code that processes TiU.

5. Coupled congestion control

For each TCP connection c, the algorithm described below receives cwnd and ssthresh as input and stores the following information:

• the Connection ID.
• a priority P(c) -- e.g., an integer value in the range from 1 (unimportant) to 10 (very important).
• The previously used cwnd used by the connection c, ccc_cwnd(c).
• The previously used ssthresh used by the connection c, ccc_ssthresh(c).

Three global variables S_CWND, S_SSTHRESH and S_P are used to represent the sum of all the ccc_cwnd values, ccc_sshtresh values and priorities of all TCP connections, respectively. S_CWND and S_SSTHRESH are used to update the cwnd and ssthresh values for all connections.

5.1. Example algorithm

This algorithm emulates the behavior of a single TCP connection by choosing one connection as the connection that dictates the increase / decrease behavior for the aggregate. It was designed to be as simple as possible. In the algorithm description below, abbreviations are used to refer to the phases of TCP congestion control as defined in [RFC5681]: SS refers to Slow Start, CA refers to Congestion Avoidance and FR refers to Fast Recovery.

For simplicity, this algorithm refrains from changing cwnd when a connection is in FR. SS should not happen as long as ACKs arrive. Hence, the algorithm ensures that the aggregate's behavior is only dictated by SS when all connections are in the SS phase.

             
  S_P = S_P + P(c)
  ccc_cwnd(c) = P(c) * S_CWND / S_P
  ccc_ssthresh(c) = ssthresh
  if (S_SSTHRESH > 0)
    ccc_ssthresh(c) = P(c) * S_SSTHRESH / S_P
  end if
  // Update c's own cwnd and ssthresh for immediate use:
  send ccc_cwnd(c) and ccc_ssthresh(c) to the connection c
             
        

      
  if(all of the connections including CoCo are in CA but c is in FR)
     c becomes the new CoCo.
  else
     if(c is in CA or SS)
        c's cwnd is assigned its previously stored ccc_cwnd value. 
      

              
  if (c is in CA)
      if(cwnd >= ccc_cwnd(c)) // cwnd has increased
         S_CWND = S_CWND + cwnd - ccc_cwnd(c)
      else
         S_CWND = S_CWND * cwnd / ccc_cwnd(c)
      end if
      ccc_cwnd(c) = P(c) * S_CWND / S_P
      ccc_ssthresh(c) = ssthresh
      if (S_SSTHRESH > 0)
         ccc_ssthresh(c) = P(c) * S_SSTHRESH / S_P
      end if
      // Update c's own cwnd and ssthresh for immediate use:
      send ccc_cwnd(c) and ccc_ssthresh(c) to the connection c
  end if    

  else if (c is in FR)
      S_SSTHRESH = S_CWND/2

  else if (c is in SS)
      if (all other connections are in SS)
         S_SSTHRESH = S_CWND/2
         S_CWND = S_CWND * cwnd / ccc_cwnd(c)
         ccc_cwnd(c) = P(c) * S_CWND / S_P
         // Update c's own cwnd for immediate use:
         send ccc_cwnd(c) to the connection c
      else                      
         make any other connection which is not in SS the CoCo
      end if
  end if
            

(1)

When a connection c starts, it adds its priority P(c) to S_P. If it is the very first connection that uses the outer UDP port number pair, it also sets S_CWND to its own cwnd. After that, the connection's globally known cwnd and ssthresh values (ccc_cwnd(c) and ccc_ssthresh(c)) are updated, and the connection updates its own cwnd and ssthresh values to be equal to ccc_cwnd(c) and ccc_ssthresh(c).

(2)

When a connection c stops, its entry is removed. S_P is recalculated.

(3)

Every time the congestion controller of a connection c calculates a new cwnd, the connection calls UPDATE, which carries out the tasks listed below to derive the new cwnd and ssthresh values for all the connections. Since we intend to emulate the behavior of one connection, we designate one of the connections as the "Coordinating Connection" (CoCo). Whenever the coordinating connection calls UPDATE, S_CWND and S_SSTHRESH are additionally updated to reflect the current sum of all stored ccc_cwnd and ccc_ssthresh values. Initially, there is only one connection and this connection automatically becomes the CoCo. It updates S_CWND to its own cwnd and sets S_SSTHRESH to 0.

(4)

WHEN a non-CoCo connection c CALLS UPDATE......

(5)

WHEN c(CoCo) CALLS UPDATE......

6. Usage considerations

TiU cannot work with applications that require the Urgent pointer (which is not recommended for use by new applications anyway [RFC6093], but should be consider if TiU is implemented in a way that allows it to be applied onto existing applications; telnet is a well-known example of an application that uses this functionality). It enables use of TCP with methods such as SPUD [I-D.hildebrand-spud-prototype]. It can also be used as a method to experimentally test new TCP functionality in the presence of middleboxes that would otherwise create problems (as some have been known to do [Honda11]). TCP option space is getting scarce, in particular on TCP SYN and TCP SYN/ACK packets. Rather than stretching the Data Offset field on TCP SYN / TCP SYN/ACK packets (which was considered for TiU design), it is recommended to use one of the other proposed mechanisms to stretch option space, e.g. "Inner Space" [I-D.briscoe-tcpm-inner-space].

Reasons to use TiU include the benefits of [Che13] and [Den08] that were discussed in Section 1. TiU has the disadvantage of disabling ECMP for the TCP connections that it encapsulates. This can reduce the capacity usage of these TCP connections. It has the advantage of being able to apply coupled congestion control, which can provide precise congestion window assignment based on a priority. Other benefits of TiU's coupled congestion control are:

• Reduced average loss and queuing delay (because the competition between the encapsulated TCP connections is avoided)
• Even in the absence of prioritization, better fairness between the TiU-encapsulated TCP connections
• No need for new TiU connections to slow start up to a reasonable cwnd value that ongoing TiU connections already have: a connection can immediately be assigned its share of the aggregate's total cwnd. This can significantly reduce the completion time of short connections.

All of these benefits only play out when there are more than one TCP connections. Some of the benefits in the list above are more significant when some transfers are short. Moreover, short transfers are less likely than long ones to saturate the capacity of a path, reducing the chance to benefit from ECMP (which TiU eliminates). This makes the usage of TiU especially attractive in situations where some transfers are short.

7. Implementation status

The University of Oslo is currently working on a FreeBSD kernel implementation of TCP-in-UDP.

8. IANA Considerations

This document specifies a new TCP option that uses the shared experimental options format [RFC6994]. No value has yet been assigned for ExID.

This document requires a well-known UDP port (referred to as port XXX in this document). Due to the highly experimental nature of TiU, this document is being shared with the community to solicit comments before requesting such a port number.

9. Security Considerations

We have not thought about security yet. This will surely be fun!

10. Acknowledgement

This work has received funding from Huawei Technologies Co., Ltd., and the European Union's Horizon 2020 research and innovation programme under grant agreement No. 644334 (NEAT). The views expressed are solely those of the author(s).

11. References

11.1. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.

11.2. Informative References

[Che13]	Cheshire, S., Graessley, J. and R. McGuire, "Encapsulation of TCP and other Transport Protocols over UDP", Internet-draft draft-cheshire-tcp-over-udp-00, June 2013.
[Den08]	Denis-Courmont, R., "UDP-Encapsulated Transport Protocols", Internet-draft draft-denis-udp-transport-00, July 2008.
[EFCM]	Savoric, M., Karl, H., Schlager, M., Poschwatta, T. and A. Wolisz, "Analysis and performance evaluation of the EFCM common congestion controller for TCP connections", Computer Networks (2005) , 2005.
[ETCP]	Eggert, L., Heidemann, J. and J. Joe, "Effects of ensemble-TCP", ACM SIGCOMM Computer Communication Review (2000) , 2000.
[fse]	Islam, S., Welzl, M., Gjessing, S. and N. Khademi, "Coupled Congestion Control for RTP Media", ACM SIGCOMM Capacity Sharing Workshop (CSWS 2014) and ACM SIGCOMM CCR 44(4) 2014; extended version available as a technical report from http://safiquli.at.ifi.uio.no/paper/fse-tech-report.pdf , 2014.
[Honda11]	Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., Handley, M. and H. Tokuda, "Is it still possible to extend TCP?", Proc. of ACM Internet Measurement Conference (IMC) '11, November 2011.
[I-D.briscoe-tcpm-inner-space]	Briscoe, B., "Inner Space for TCP Options", Internet-Draft draft-briscoe-tcpm-inner-space-01, October 2014.
[I-D.hildebrand-spud-prototype]	Hildebrand, J. and B. Trammell, "Substrate Protocol for User Datagrams (SPUD) Prototype", Internet-Draft draft-hildebrand-spud-prototype-03, March 2015.
[I-D.ietf-rmcat-coupled-cc]	Islam, S., Welzl, M. and S. Gjessing, "Coupled congestion control for RTP media", Internet-Draft draft-ietf-rmcat-coupled-cc-00, September 2015.
[RFC1078]	Lottor, M., "TCP port service Multiplexer (TCPMUX)", RFC 1078, DOI 10.17487/RFC1078, November 1988.
[RFC2140]	Touch, J., "TCP Control Block Interdependence", RFC 2140, DOI 10.17487/RFC2140, April 1997.
[RFC3124]	Balakrishnan, H. and S. Seshan, "The Congestion Manager", RFC 3124, DOI 10.17487/RFC3124, June 2001.
[RFC5681]	Allman, M., Paxson, V. and E. Blanton, "TCP Congestion Control", RFC 5681, DOI 10.17487/RFC5681, September 2009.
[RFC6093]	Gont, F. and A. Yourtchenko, "On the Implementation of the TCP Urgent Mechanism", RFC 6093, DOI 10.17487/RFC6093, January 2011.
[RFC6356]	Raiciu, C., Handley, M. and D. Wischik, "Coupled Congestion Control for Multipath Transport Protocols", RFC 6356, DOI 10.17487/RFC6356, October 2011.
[RFC6555]	Wing, D. and A. Yourtchenko, "Happy Eyeballs: Success with Dual-Stack Hosts", RFC 6555, DOI 10.17487/RFC6555, April 2012.
[RFC6824]	Ford, A., Raiciu, C., Handley, M. and O. Bonaventure, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013.
[RFC6994]	Touch, J., "Shared Use of Experimental TCP Options", RFC 6994, DOI 10.17487/RFC6994, August 2013.

Authors' Addresses