a)                 __ ____ _______ _                    __
                  |__|____|_______|_|                  |__|   control
                  :E : D  :  C    :B:                  :A :
  ________________:  :    :       : :__________________:  :
 |________________|                 |__________________|      data

b)               __
                |__|  E
                |_|__ B               __
                |____|D              |__|A   control
                 \  /                \  /
  ________________\/__________________\/
 |________________||__________________|      data
                   !
                   !drop
               ____!__
              |_______|C

Figure 4: Control options a) inside vs. b) outside TCP sequence space`

By including Inner Options within the sequence space, each control option is automatically bound to the start of a particular byte in the data stream, which makes it easy to switch behaviour at a specific point mid-stream (e.g. re-keying or switching to a different control mode). With traditional TCP options, a bespoke reliable and ordered binding to the data stream would have to be developed for each TCP option that needs this capability (e.g. co-ordinating use of new keys in TCP-AO [RFC5925] or tcpcrypt [I-D.bittau-tcpinc-tcpcrypt]).

Including Inner Options in sequence also allows the receiver to tell the sender the exact point at which it encountered an unrecognised TCP option using only TCP's pre-existing byte-granularity acknowledgement scheme.

Middleboxes exist that rewrite TCP sequence and acknowledgement numbers, and they also rewrite options that refer to sequence numbers (at least those known when the middlebox was produced, such as SACK, but not any introduced afterwards). If Inner Options were not included in sequence, the number of bytes beyond the TCP Data Offset in each segment would not match the sequence number increment between segments. Then, such middleboxes could unintentionally corrupt the user-data and options by 'normalising' sequence or acknowledgement numbering. Fortunately, including Inner Options in sequence improves robustness against such middleboxes.

3.2.6.2. Fire-and-Forget Options

The Inner Space protocol allows Fire-and-Forget Options to be tunnelled within the TCP Data so that they can traverse middleboxes that would otherwise strip them or somehow normalise their contents. Two question then arise: i) should Fire-and-Forget Objects (padding, the InSpace Option and Prefix Options) consume sequence space and ii) should they be covered by flow control? The answers to these questions will also be re-usable to multiplex streams within one TCP connection:

Sequence Space:

Ideally, fire-and-forget objects would not consume sequence space, because they do not need to be retransmitted. However, many middleboxes expect the TCP sequence number to increment consistently with the amount of TCP Data. For instance, a split connection would be likely to 'normalise' sequence numbers, being unaware that certain items in the datastream might be exempt from sequence space consumption.

Therefore, although it is not elegant, the sender has to consume sequence space for fire-and-forget objects, but it implicitly considers these octets to be immediately acknowledged. And the receiver does not have to immediately acknowledge sequence space consumed solely by fire-and-forget objects; it can defer until it acknowledges reliably delivered flow-controlled objects—when it does no harm to cumulatively acknowledge intervening fire-and-forget objects as well. This is the underlying principle behind the normative rules given on sequence space consumption and ACK withholding in Section 2.3.1.3 and Section 2.3.1.5.

Flow Control:

The sender does not need to count Fire-and-Forget Objects against the receive window (Rcv.Wind), just as it does not count Outer TCP Options against Rcv.Wind.This should work because It is impossible for middleboxes to 'normalise' the receive window and flow control, because they cannot know when the application is releasing data from the receive buffer. Also the receiver always processes Fire-and-Forget Objects immediately without buffering them; it could be considered that the receiver effectively subtracts their size from Rcv.Wind then immediately restores Rcv.Wind to its former value.

In fact, as shall now be explained, it has to be mandatory for the sender not to count fire-and-forget objects against Rcv.Wind. It is important for deadlock avoidance that certain TCP options never consume Rcv.Wind. Some TCP options acknowledge data, e.g. SACK or the Data ACK within the Data Sequence Signal (DSS) sub-option of MPTCP. Other TCP options need to be applied to all ACKs, e.g. the MAC of tcpcrypt. If an acknowledgement were to need sufficient advertised receive window before it could be sent, there would always be a risk of deadlock if the receiver ever needed the acknowledgement before it could release more receiver buffer [Raiciu12].

The rule above concerning sequence space is a compromise needed to traverse middleboxes. So, perhaps predictably, this begets further compromises. The rule concerning flow-control is principled. So perhaps predictably, it has to be compromised to traverse certain middleboxes. The rationale for these compromises is explained below, referring to the normative rules in the protocol specification where appropriate:

Sequence Space:

If the sender does not retransmit unacknowledged data after a RTO, some middleboxes will mimic TCP's retransmission timeout (RTO) and resend the fire-and-forget data themselves, which could lead to an ACK storm. Therefore, Section 2.3.1.5 allows a receiver to emit a pure ACK every round trip, just to keep such middleboxes quiet. In general, allowing TCP to ACK an ACK can lead to an ACK storm. However, in this case, all that is allowed is a Pure ACK in response to an Impure ACK, which immediately terminates any potential for a vicious circle. This solution even works in the case where both TCP hosts ignore ACKs unless they are authenticated (which the pure ACK will not be). No harm will arise if the remote host ignores the pure ACK, because it is only for the benefit of a middlebox anyway.

If a sequence of one or more Impure ACKs is lost the receiver cannot suppress retransmission, because it can only decide whether it needed in-sequence data once it arrives. Therefore, loss of fire-and-forget data causes a retransmission that may prove to be unnecessary. By the rules in Section 2.3.1.5, an ACK would only include fire-and-forget data in the first place if it was actually necessary. Therefore, normally retransmission of Impure ACKs will be required and useful. However, sometimes, the Prefix Option(s) within the Impure ACK(s) might have become unnecessary. This inefficiency could just be ignored, or partial reliability could be added to TCP to address it. The Inner Space protocol does not prevent partial reliability being added, but it does not require it either.

Flow Control:

Some middleboxes attempt to mitigate scanning or DoS attacks by reading the window field in the main TCP header (and the Window Scale outer TCP option if present) and discarding segments that they calculate contain data that is out-of-window.

Section Section 2.3.1.4 requires the two endpoints to tacitly agree that the fire-and-forget portion of the TCP Data is exempt from flow control. A legacy middlebox will not know this, so it might think data is out-of-window when the endpoints have agreed it is in-window. Section Section 2.3.1.4 provides a solution to this problem, which is only necessary if a TCP implementation is deployed where there is a risk of encountering such middleboxes. The solution involves the TCP sender denying itself the use of the bottom of the buffer advertised by the receiver. Normally the sender stops sending when it calculates the remaining receive window is zero. Instead, the modified sender sets itself a threshold (Snd.Wind.Min) to allow for the Fire-and-Forget Objects it might need in flight, and it stops sending before the receive window drops below this threshold.

Snd.Wind.Min bytes at the 'left-hand' end of the receive buffer are wasted by this solution (to be fair, the middlebox behaviour is really to blame). An alternative was considered where the sender and receiver use a new Inner TCP Option to agree a window offset between themselves, so that middleboxes are not party to their agreement. Although, this would not waste any of the left-hand end of the receive buffer, it would reduce the maximum advertised buffer at the right-hand end by the same amount. Therefore the sender-only solution was chosen, given it is much simpler, and the sender can continuously adapt how much allowance it sets aside throughout the connection, rather than having to commit to a necessarily conservative estimate at the start.

3.3. Deployment Approach

3.3.1. Substrate Protocol: TCP vs. UDP

Inner Space uses TCP as a substrate protocol, i.e. on the wire, the headers look like an RFC793-compliant TCP, and there is only a difference if one looks inside the TCP Data. Other transport extensibility approaches have used UDP as a substrate protocol, for instance, to carry SCTP through middleboxes.

In design and implementation terms, it is much easier to turn UDP into a reliable protocol, than it is to selectively turn TCP into an unreliable protocol. However, UDP is already blocked on about 15% of Internet paths {ToDo: ref}, whereas vanilla TCP is still universally permitted. Therefore, because the goal is middlebox traversal, not just ease of implementation, Inner Space uses TCP as a substrate.

It may well turn out that Inner Space cannot reach some places that UDP can. It is expected that applications (or even the TCP stack) might sometimes have to resort to tryinging UDP as a substrate in such cases.

3.3.2. Kernel-Space vs. User-Space

At an earlier stage in the specification of the Inner Space protocol [I-D.briscoe-tcpm-inner-space] before unordered delivery of Inner Options was introduced, Inner Options could all be processed in either user-space or kernel-space. The only exception was the interactions controlling the handshake on the first segment in each direction. However, with the addition of unordered delivery of Prefix Options, the protocol has to be implemented in the kernel, because the protocol modifies the behaviour of TCP, not just its payload.

3.4. Rationale for the InSpace Option Format

The format of the InSpace Option (Figure 3) does not necessarily have to comply with the RFC 793 format for TCP options, because it is not intended to ever appear in a sequence of TCP options. In particular, it does not need an Option Kind, because the option is always in a known location. In effect the magic number serves as a multi-octet Option Kind for the first InSpace Option, and the location of each subsequent option is always known by the marker in the InSpace option as well as by the offset from the previous one, using the Sent Data Size field.

Other aspects of the layout are justified as follows:

Length:

Whatever the size of the InSpace Option, the right-hand edge of the Length field is always located 8 octets from the left-hand edge of the marker that starts the InSpace Option. From the Length, the receiver can always determine the layout of the rest of the option. The length is in 4-octet words because the InSpace option is always a multiple of 4 octets long, so that any subsequent Inner TCP Options comply with TCP's option alignment requirements.

Sent Data Size:

This field is 16 bits wide, which is reasonable given segment size cannot exceed the limits set by the Total Length field in the IPv4 header and the Payload Length field in the IPv6 header, both of which are 16 bits wide.

If the sender were to use a jumbogram [RFC2675], it could use the Jumbo InSpace Option defined in Appendix C.2, which offers a 32-bit Sent Data Size field. The Jumbo InSpace Option is not mandatory to implement for the present experimental specification. Even if it is implemented, it is only defined when SYN=0, given use of a jumbogram for a SYN or SYN/ACK would significantly exceed other limits that TCP sets for these segments.

Inner Options Offset:

This field is in units of 4-octet words, so its width is 14-bits. Then, if necessary Suffix Options can be as large as a maximum sized segment (given 4 * 2^14 = 2^16 octets).

Suffix Options Offset:

The InOO field is the same 14-bit width as the SOO field, and for the same reason. Both the SOO and InOO fields are aligned 2 bits to the left of a word boundary so that they can be used directly in units of octets by masking out the 2-bit field to the right.

When SYN=1 the layout of the InSpace Option includes:

Magic Numbers:

The 32-bit size of Magic Number A is not enough to reduce the probability of mistaking the start of an Ordinary SYN Payload for the start of the Inner Space protocol. A 64-bit magic number could have been provided by using the next 4-octet word, but this would be unnecessarily large. Therefore, when SYN=1, Magic Number B provides 16 more bits of magic number. Otherwise, these 16-bits would only have to be used for padding to align with the next 4-octet word boundary anyway.

When SYN=0, the following further considerations determined the layout of the InSpace Option:

ZOMBI:

The ZOMBI field holds an offset that has to be sufficiently wide to span the extent of a maximum-sized segment of 2^16 bits. Given the offset is measured in 2-octet units, this means the ZOMBI field has to be at least 15 bits wide (see Appendix C.2 for the size of the ZOMBI field for a jumbogram).

Marker:

Given occurrences of the marker are replaced by offsets of the size of the ZOMBI field, the marker has to be at least as wide as the ZOMBI field. However, a 16-bit marker is used, because it is more efficient than having to replace 15-bit markers.

Currently Unused (CU):

There are three CU fields in the InSpace option when SYN=0 that fill odd corners of space. Unfortunately, this is necessary to ensure 4-octet alignment of the first Inner Options.

Prefix (P) flag:

When there are solely Prefix Options, or solely Suffix Options, a short-form InSpace Option can be used (Len = 1) by omitting the last 4-octet word. Then the P flag determines whether there are solely Prefix Options or solely Suffix Options in the Inner Options field. Whenever both Prefix and a Suffix Option are needed on the same segment, even though only 14 more bits of framing information are needed, the InSpace option has to grow in steps of 32 bits to maintain 4-octet alignment. Therefore 18 bits have to be assigned as Currently Unused (CU).

4. Protocol Overhead

The overhead of the Inner Space protocol is quantified as follows:

Dual Handshake:

Latency:

Upgraded Server :

zero;

Legacy Server:

worst latency of the two, if dual handshakes are used.

Connection Rate:

The typical connection rate will inflate by P*D, where:

P

[0-100%] is the proportion of connections that use extra option space;

D

[0-100%] is the proportion of these that use a dual handshake (the remainder use a single handshake, e.g. by caching knowledge of upgraded servers).

For example, if P=80% and D=10%, the connection rate will inflate by 8%. P is difficult to predict. D is likely to be small, and in the longer term it should reduce to the proportion of connections to remaining legacy servers, which are likely to be the less frequently accessed ones. In the worst case if both P & D are 100%, the maximum that the connection rate can inflate by is 100% (i.e. to twice present levels).

Connection State:

Connection state on servers and middleboxes will inflate by P*D/R, where

R

is the average hold time of connection state measured in round trip times

This is because a server or middlebox only holds dual connection state for one round trip, until the RST on one of the two connections. For example, keeping P & D as they were in the above example, if R = 3 round trips {ToDo: TBA}, connection state would inflate by 2.7%. In the longer term, any extra connection state would be focused on legacy servers, with none on upgraded servers. Therefore, if memory for dual handshake flow state was a problem, upgrading the server to support the Inner Space protocol would solve the problem.

Network Traffic:

The network traffic overhead is 2*H*P*D/J counting in bytes or 2*P*D/K counting in packets, where

H

is (h+60B+12B) where h is the IP header size (assuming the Ordinary SYN and SYN/ACK have a TCP header packed to the maximum of 60B with TCP options, they have no TCP Payload, their IP headers have no extensions and the InSpace Option in the SYN-U and SYN/ACK-U is 12B). That is H will be 92B for IPv4 or 112B for IPv6;

J

is the average number of bytes per TCP connection (in both directions)

K

is the average number of packets per TCP connection (in both directions);

For example, keeping and P & D as they were in the above example, if J = 50KiB for IPv4 and K = 70 packets (ToDo: TBA), traffic overhead would be 0.03% counting in bytes or 0.2% counting in packets.

Processing:

{ToDo: Implementation tests}

InSpace Option on every non-empty SYN=0 segment:

Network Traffic:

The traffic overhead is P*Q*8/F, where

Q

is the proportion of Inner Space connections that leave the protocol enabled after the initial handshake;

F

is the average frame size in bytes (assuming one segment per frame).

This assumes an InSpace option adds 8B per segment (i.e. both Prefix and Suffix Options together on every segment will be rare). For example, keeping P as it was in the above example and taking Q=10% and F=750B, the traffic overhead is 0.09%. It is as difficult to predict Q as it is to predict P.

Processing:

{ToDo: Implementation tests}

5. Interaction with Pre-Existing TCP Implementations

5.1. Compatibility with Pre-Existing TCP Variants

It is believed that all TCP options that were designed as Outer Options can be relocated without alteration as Prefix Options, because the unreliable unordered semantics are the same as TCP Outer Options. However, some yet-to-be-defined TCP options might be better suited to the reliable ordered semantics of Suffix Options. Specifically, existing or proposed TCP options fall into the following categories:

Segment-Related:

Concerned with the delivery of individual segments as they arrive at the receiver. Therefore these options MUST NOT be located as Suffix Options:

• Timestamp [RFC7323] on SYN=0 segments;
• SACK [RFC2018];
• The Data ACK part of the DSS option of Multipath TCP [RFC6824];
• TCP-AO [RFC5925] if covering TCP Options;

Stream-Related:

Controlling delivery of an ordered stream. Therefore these options SHOULD be located as Suffix Options:

• The tcpcrypt CRYPT sub-options [I-D.bittau-tcpinc-tcpcrypt].

Connection-Related:

Controlling the parameters of a connection. These options can be located either as Suffix, Prefix or Outer Options:

• No-op and end of option list [RFC0793];
• Maximum Segment Size (MSS) [RFC0793];
• SACK-ok [RFC2018];
• The timestamp when used on SYN=1 segments to indicate support for timestamps [RFC7323];
• Window Scale [RFC7323];
• Multipath TCP [RFC6824], except the Data ACK part of the Data Sequence Signal (DSS) option;
• TCP Fast Open [I-D.ietf-tcpm-fastopen];

{ToDo: The above list is not authoritative. Some TCP options include suboptions, some of which are discussed below, but others remain to be fully assessed.}

The specification of any future TCP option MUST state whether it is designed as a Suffix Option (reliable ordered) or as a Prefix / Outer Option (unreliable unordered) or "Don't Care". A TCP option MUST by default only be used as an Outer or Prefix Option, unless it is explicitly specified that it can (or must) be used as a Suffix Option.

The Inner Space protocol supports TCP Fast Open, by constraining the client to obey the rules in Section 2.3.1.1).

All the sub-types of the MPTCP option [RFC6824] except one could be located as Suffix or Prefix Options. That is, MP_CAPABLE, MP_JOIN, ADD_ADDR(2), REMOVE_ADDR, MP_PRIO, MP_FAIL, MP_FASTCLOSE. The Data Sequence Signal (DSS) of MPTCP consists of four separable parts: i) the Data ACK; ii) the mapping between the Data Sequence Number and the Subflow Sequence Number over a Data-Level Length; iii) the Checksum; and iv) the DATA_FIN flag. If MPTCP were re-factored to take advantage of the Inner Space protocol, all these parts except the Data ACK could be located as Suffix Options (the Checksum would not be necessary).

The MPTCP Data ACK has to remain as a Prefix or Outer Option otherwise there would be a risk of flow control deadlock, as pointed out in [Raiciu12]. For instance, a Web client might pipeline multiple requests that fill a Web server's receive buffer, while the Web server might be busy sending a large response to the first request before it reads the second request. If the Data ACK were a Suffix Option, the Web client would have to stop acknowledging the first response from the server (due to lack of receive window). Then the server would not be able to move on to the next request—a classic deadlock.

The TCP authentication option can be configured either to cover TCP Options or not (when it was defined only Outer Options existed). If it covers any TCP Options it has to be located as an Outer or Prefix Option to prevent the possibility of flow-control deadlock (because it would consume receive window on pure ACKs if it were located as a Suffix Option).

All sub-options of the tcpcrypt CRYPT option could be located as Suffix Options. However, as long as the tcpcrypt MAC option covers the TCP header and Outer Options, it has to be located as an Outer Option for the same deadlock reason as TCP-AO.

An Upgraded Server can support SYN Cookies [RFC4987] for Ordinary Connections. For Upgraded Connections Section 2.5 defines a new EchoCookie TCP option that is a prerequisite for InSpace implementations, and provides sufficient space for the more extensive connection state requirements of an InSpace server.

{ToDo: TCP States and Transitions, Connectionless Resets, ICMP Handling, Forward-Compatibility.}

5.2. Interaction with Middleboxes

The interaction with the assumptions about TCP made by middleboxes is covered extensively elsewhere:

• Section 2.3.3 specifies forwarding behaviour for Inner Options;
• The following sections explain the Inner Space protocol approach to middlebox traversal:
  • Section 3.2.1 justifies extending TCP within the TCP Data;
  • Section 3.2.2 justifies the magic number approach;
  • Section 3.2.3 explains why the protocol will remain robust as middlboxes evolve;
  • Section 3.2.6 justifies including Inner Options in sequence;
  • Section 3.2.5) explains how the protocol will remain robust to resegmentation.

5.3. Interaction with the Pre-Existing TCP API

An aim of the Inner Space protocol is for legacy applications to continue to just work without modification. Therefore it is expected that the dual handshaking logic and placement of options within the TCP Data will be implemented beneath the well-known socket interface.

Inner Space implementations will need to comply with the following behaviours to ensure that legacy applications continue to receive predictable behaviour from the socket interface:getpeername() while the TCP server behind the socket is (unwittingly) engaged in a dual handshake, it will return the port of the remote client, even though this connection might subsequently be aborted. This is because a TCP server is not aware of whether it is part of a dual handshake.

Querying local port (TCP client):

If an application calls getsockname() while the TCP client behind the socket is engaged in a dual TCP handshake, the call SHOULD block until the local TCP has aborted one of the connections so it knows which of the two ports will continue to be used.

Binding to an explicit port:

If an application specifies that it wants the TCP client to use a specific port, the Inner Space capability can be used, but the dual handshake MUST be disabled, because the dual handshake has to try two ports. Therefore, if the app binds to a specific port, the upgraded SYN MUST be tried first on its own, then if that reveals that the server is not upgraded, the stack will abort that connection with a RST and use the same port to send an ordinary SYN. Use of a specific port might be necessary, for example in the FTP protocol, in a port-testing application or if the application wants to explicitly control all the handshaking logic of the Inner Space protocol itself.

Logging:

The dual handshake will show up as a specific signature in logs of network activity. Log formats might not be able to record two local ports against one socket, so logs might contain unexpected or erroneous data. Even if logs correctly track both connection attempts, log analysis software might not expect to see one socket attempt to use two different ports. {ToDo: All this needs to be turned into a predictability requirement.}

Note that Inner Space has no impact on queries for the remote port from a TCP server. If an application calls

Some applications interrogate the TCP stack to determine the path max transmission unit (PMTU), e.g. in order to optimize application message boundaries within the datastream. From the viewpoint of such applications, TCP options subtract the same amount from the PMTU whether they are Outer or Inner Options. However, the 8 (or 12) octet InSpace header and the alignment padding represent extra overhead. Therefore, for such applications, the TCP stack as seem through the socket API will seem similar to a tunnel that reduces the useful size of the PMTU. This could lead to fragmentation until such applications are updated. Nonetheless, most such applications already include code to adapt to PMTU reduction by tunnels.

It would be appropriate to enable the Inner Space protocol on a per-host or per-user basis. The necessary configuration switch does not need to be standardised, but it might allow the following three states:

Enabled:

The stack will enable Inner Space on any TCP connection that that needs Inner Space for its TCP options. The stack might still disable the Inner Space protocol autonomously after the initial handshake if it is not needed.

Forwarding:

The Forwarding mode is for TCP implementations on middleboxes that implement split TCP connections, as discussed in Section 2.3.3. Forwarding mode is similar to Disabled, except it forwards data in SYN without deferring it until the incoming connection is established.

Disabled:

Inner Space is not enabled by default on any connections, except those that specifically request it.

The socket API might also need to be extended for future applications that want to control the Inner Space protocol explicitly. Experience will determine the best API, so these ideas are merely informational suggestions at this stage:

Enabling/disabling Inner Space:

As well as the above per-host or per-user switches, the extended API might need to allow an application to disable Inner Options on a per-socket basis (e.g. for testing). A socket might need to be opened in one of three possible Inner Space modes: i) Enabled; ii) Enabled initially but can be disabled autonomously by the stack if redundant; iii) Enabled initially, then disables itself after the SYN/ACK; and iv) Disabled. It also ought to be possible for an application to disable Inner Options on-demand mid-connection.

Querying support for Inner Space:

An application might need to be able to determine whether the host supports Inner Space and in which mode it is enabled on a particular socket. For instance, an application might need to choose different socket options depending on how much space is available, which depends on whether Inner Space is enabled.

Latency vs Efficiency:

A socket that prefers efficient use of connection state over latency might use the optional explicit variant of the dual handshake (Appendix D). It is unlikely that a new option specific to Inner Space would be needed to express this preference, as many operating systems already offer a similar socket option.

Logging:

Log formats and log analysis software might need to be extended to distinguish between the deliberate RST within the dual handshake and an unexpected connection RST.

6. IANA Considerations

This specification requires IANA to allocate values from the TCP Option Kind name-space against the following names:

• "Inner Option Space Upgraded (InSpaceU)"
• "Inner Option Space Ordinary (InSpaceO)"
• "ModeSwitch"

Early implementation before the IANA allocation MUST follow [RFC6994] and use experimental option 254 and respective Experiment IDs:

• 0xUUUU (16 bits);
• 0xOOOO (16 bits);
• 0xMMMM (16 bits);

{ToDo: Values TBA and register them with IANA} then migrate to the assigned option after allocation.

7. Security Considerations

Certain cryptographic functions have different coverage rules for the TCP Header and TCP Payload. Placing some TCP options beyond the Data Offset could mean that they are treated differently from regular TCP options. This is a deliberate feature of the protocol, but application developers will need to be aware that this is the case.

A malicious host can send bogus SYN segments with a spoofed source IP address (a SYN flood attack). The Inner Space protocol does not alter the feasibility of this attack. However, the extra space for TCP options on a SYN allows the attacker to include more TCP options on a SYN than before, so it can make a server do more option processing before replying with a SYN/ACK. To mitigate this problme, a server under stress could deprioritise SYNs with longer option fields to focus its resources on SYNs that require less processing.

Each SYN in a SYN flood attack causes a TCP server to consume memory. The Inner Space protocol allows a potentially large amount of TCP option state to be negotiated during the SYN exchange, which could allow attackers to exhaust the TCP server's memory more easily. The EchoCookie TCP option (see Section 2.5) allows the server to place this state in a cookie and send it on the SYN/ACK to the purported address of the client—rather than hold it in memory. Then, as long as the client returns the cookie on the acknowledgement and the server verifies it, the server can recover its full record of all the TCP options it negotiated and continue the connection without delay. On the other hand, the server's responses to SYNs from spoofed addresses will scatter to those spoofed addresses and the server will not have consumed any memory while waiting in vain for them to reply. See the Security Considerations in [I-D.briscoe-tcpm-echo-cookie] for how the EchoCookie facility protects against reflection and amplification attacks.

Some security devices block data in an initial SYN segment, classifying it as the signature of an attack. Attackers might indeed use data-in-SYN to strengthen the force of a SYN flood attack, but it has also always been valid for clients to use data-in-SYN for low latency service as well (today data-in-SYN is used by TCP Fast Open, but data-in-SYN has been permitted for similar reasons right back to the days of RFC 793). On its own, data-in-SYN MUST NOT be considered a sufficient signature of an attack. It can only be considered an attack signature if seen in combination with other symptoms of a SYN flood attack. The logic that led to data-in-SYN alone being considered an attack was probably well-intentioned, but it actually turns a security device into an attack on innocent low latency services.

The optional extension for DPI traversal specified in Appendix C.3 might create a new attack vector. The attack was originally proposed (by David Mazieres) when an earlier draft required the optional extension to be applied at the start of both half-connections. As long as the DPI traversal extension no longer applies in the server-client direction the attack seems less feasible. Nonetheless, the attack in the server-client direction is described here anyway (in case it prompts someone to think of a similar feasible attack in the client-server direction):

Attack that used to be feasible in the server-client direction:

An attacker could have crafted content (e.g. a binary file such as a graphics object) such that it included the appropriate bits in the correct positions to match the Inner Space magic numbers and the expected format of some TCP options. It could have then uploaded this content to a legacy server for download by other clients (e.g. a public image archive). Then, if an upgraded Inner Space TCP client had accessed this legacy server, it would have seemed as if the server was upgraded. So the attacker could have theoretically conscripted the server into sending TCP options of its choice. Although the attacker would have been limited to TCP options relevant to those previously proposed by the client, some harm might have been possible. The attacker might also have been able to contrive the remainder of the content (after removing the apparent TCP options) to be some form of script or executable.

If the DPI traversal solution is to be used, and a feasible attack is developed in the client-server direction, a couple of directions to prevent such an attack could be explored:

• the magic number would somehow have to be complemented by another signal, perhaps out of band;
• the magic number would need to somehow include a cryptographic hash of material sent by the client, so that an attacker could not predict it.

8. Acknowledgements

The idea of this approach grew out of discussions with Joe Touch while developing draft-touch-tcpm-syn-ext-opt, and with Jana Iyengar and Olivier Bonaventure. Jana Iyengar also suggested the sender-only flow-control offset. The idea that it is architecturally preferable to place a protocol extension within a higher layer, and code its location into upgraded implementations of the lower layer, was originally articulated by Rob Hancock. {ToDo: Ref?} The following people provided useful comments: Joe Touch, Yuchung Cheng, John Leslie, Mirja Kuehlewind, Andrew Yourtchenko, Costin Raiciu, Marcelo Bagnulo Braun, Julian Chesterfield, Jaime Garcia, Ted Hardie and David Mazieres, Tim Shepard, Mark Handley.

Bob Briscoe's contribution is part-funded by the European Community under its Seventh Framework Programme through the Trilogy 2 project (ICT-317756) and the Reducing Internet Transport Latency (RITE) project (ICT-317700). The views expressed here are solely those of the author.

9. References

9.1. Normative References

9.2. Informative Reference

Appendix A. Zero Overhead Message Boundary Insertion (ZOMBI)

This appendix is normative and mandatory to implement for the Inner Space protocol. This encoding is relegated to an appendix merely because it is applicable more generally than for just Inner Space. Therefore, in a future revision, this appendix might be removed and replaced by a reference to a stand-alone document.

The Inner Space protocol requires the sender to add a marker in every segment at the first 4-octet aligned word from the start of the datastream. Then, even if the stream is subsequently resegmented, the receiver can recover segments and their associated TCP options as they were sent. The sender uses the value 0x0000 as the 2-octet marker at the start of the InSpace option header. It uses the ZOMBI encoding to remove all other occurrences of 0x0000, treating the segment as a sequence of 2-octet shorts. Then, a marker will unambiguously locate the InSpace option at the start of each segment. From this InSpace option, the receiver can find the length of the segment. Then it can decode the ZOMBI encoding to return the segment to its original form.

The sender applies the ZOMBI encoding as follows:

1. It places 0x0000 in the Marker and the ZOMBI fields of the InSpace option, and fills all the other fields of the InSpace option with the relevant sizes and offsets.
2. Treating the stream as a sequence of 2-octet shorts,starting from the ZOMBI field, it replaces each occurrence of 0x0000 with the offset (in shorts) to the next occurrence of 0x0000, or to just beyond the end of the segment when there are no more occurrences of 0x0000.

Because an offset can never be zero, this process naturally removes all occurrences of 0x0000 from the segment.

The receiver reverses the above encoding, assuming the worst case of a resegmented stream unless it finds otherwise:

1. If it is buffering undecoded bytes either side of the newly arrived segment in the sequence space, it coalesces them.
2. Scanning two octets at a time aligned on even numbers of octets from the ISN, it locates the next occurrence of an InSpace option by locating the next occurrence of 0x0000 in a segment.
3. Starting at the ZOMBI field, it points a variable (e.g. ptr) to a position in the stream, reads the short at that location, writes 0x0000 into the stream to replace it, then increments ptr by the value just read. It continually repeats the same read, replace and increment operations at each new location pointed to by ptr.
4. The receiver knows the size of the sent segment from the SDS field, so that it knows when to stop decoding. If the end of the received segment is reached before this, it implies the stream has been resegmented and the next segment has not been buffered yet. In this case, the receiver stores how much decoding is left.
5. If there are more undecoded octets buffered, the process repeats from step 1.

Below an implementation of the ZOMBI encode and decode algorithms is given in C. The decode algorithm would be preceded by marker-scanning code to find the location of the ZOMBI and SDS fields within the InSpace option. The SDS field will always be non-zero, therefore it will never be changed by the encoding, so the receiver can read it before starting to decode. In case length is odd, a non-zero pseudo-padding octet is considered to be appended to the segment while encoding or decoding (but it is not actually transmitted).

/* {ToDo: Test}
 * ZombiEncode encodes "length" bytes of data
 * starting directly after the marker pointed to by "ptr", where:
 *   length = sds - pad.
 */

void ZombiEncode(unsigned short *ptr, unsigned short length)
{
  const unsigned short *end = ptr + ++length>>1;  % /2 rounded up
  unsigned short *code_ptr = ++ptr;               % point to ZOMBI
  unsigned short code = 0x0001;

  while (++ptr < end) {                   % initialise after ZOMBI
    if (*ptr == 0) {
      *code_ptr = code;
      code_ptr = ptr;
      code = 0x0001;
    } else
      code++;
  }
}

/* {ToDo: Test}
 * ZombiDecode decodes "length" bytes of data
 * starting after the marker pointed to by "ptr", where
 *   length = sds - pad.
 * Returns number of shorts still to decode.
 */

short ZombiDecode(unsigned short *ptr, unsigned short length)
{
  const unsigned short *end = ptr++ + ++length>>1;  % /2 rounded up
  while (ptr < end) {                         % initialise to ZOMBI
    code = *ptr;
    *ptr = 0;
    ptr += code;
  }
  return (ptr - end);
}

The ZOMBI encoding always uses a marker that is larger than the maximum possible segment size. Therefore, for a jumbo segment Appendix C.2, the sender uses 0x00000000 (4 octets of zeros) as the marker; it pads the segment to a multiple of 4 octets; and it scans the stream in 4-octet words, replacing any occurrences of the marker with the offset in 4-octet words to the next marker.

The ZOMBI encoding is similar to consistent overhead byte stuffing (COBS [Cheshire97]). The main difference is that COBS markers are only one octet. Therefore, in COBS, whenever the distance between zero-bytes is greater than 0xFE, it has to insert an extra byte into the stream with the special value of 0xFF. When decoding, 0xFF is removed rather than replaced by 0x00. Therefore, as well as 2 extra delimiting octets, COBS introduces a variable number of extra octets, but no more than 1 in 254 (a more accurate name would have been capped overhead byte stuffing, because the overhead is variable, not consistent).

In contrast, ZOMBI introduces a predictable overhead of 4 delimiting octets per segment (or 5 for odd length segments), with no unpredictable variation. Therefore, space for the known overhead can be set aside in the InSpace option, and the ZOMBI encode and decode operation can be zero-copy, which is not possible with COBS. A more accurate name for ZOMBI would have been constant overhead message boundary insertion. Nonetheless, the encoding to replace markers once the message boundaries have been inserted actually is zero overhead, so the cool acronym is not totally contrived.

Appendix B. Generic Connection Mode Switching

This appendix is normative and mandatory to implement for the Inner Space protocol. This encoding is relegated to an appendix merely because, in a future revision, this appendix might be removed and replaced by a reference to a stand-alone document. It defines the new ModeSwitch TCP option illustrated in Figure 5. This option provides a facility to disable the Inner Space protocol for the remainder of a connection. It also provides a general-purpose facility for a TCP connection to co-ordinate between the endpoints before switching into a yet-to-be-defined mode.

 0                   1                   2
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
+---------------+---------------+-----------+-+-+
|  ModeSwitch   | Length=3      |Flags (CU) |I|R|
+---------------+---------------+-----------+-+-+

 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+---------------+---------------+
| Kind=InSpaceO | Length=2      |
+---------------+---------------+

 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+---------------+---------------+
| Kind=InSpaceU | Length=2      |
+---------------+---------------+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
|                            Marker                             |
+-----------------------------------------------------------+---+
|                          ZOMBI                            |Len|
+-----------------------------------------------------------+---+
|                      Sent Data Size (SDS)                     |
+-----------------------------------------------------------+-+-+
|                 Inner Options Offset (InOO)               CU|P|
+-----------------------------------------------------------+-+-+
|                 Suffix Options Offset (SOO)               |CU |
+-----------------------------------------------------------+---+

                                             TCP Data
                    .---------------------------'------------------.
                    |                 Inner Options                |
  a) SYN=1          |               .---------'---------.          |
+--------+----------+-------------+-+---------+---------+----------+
| BaseHdr| OuterOpts| Payload     | | PrefOpts| SuffOpts|InSpace#1 |
+--------+----------+-------------+-+---------+---------+----------+
|        DO         |             | |   SOO   |         |          |
`------------------>|             |P`-------->|         | Len = 3  |
|                   |             |a|           InOO    |<---------'
                                  |d|<------------------'          |

  b) First SYN=0 segment
+--------+----------+--------+-+---------+--------+--------+-------+
| BaseHdr| OuterOpts|Payload | |InSpace#2|PrefOpts|SuffOpts|Payload|
+--------+----------+--------+-+---------+--------+--------+-------+
|        DO         |        | |   Len   |  SOO            |       |
`------------------>|        |P`-------->`------->|        |       |
                    |        |a|         |          InOO   |       |
                    |        |d|         `---------------->|       |
                    | SDS#1  |                               SDS#2 |
                    `------->`------------------------------------>|
                    |        |                                     |