]> TCP RST Diagnostic Payload Orange
mohamed.boucadair@orange.com
Nokia
India kondtir@gmail.com
Tencent
kerneljasonxing@gmail.com
TCP Maintenance and Minor Extensions Service diagnostic This document specifies a diagnostic payload format returned in TCP RST segments. Such payloads are used to share with an endpoint the reasons for which a TCP connection has been reset. Sharing this information is meant to ease diagnostic and troubleshooting. This specification builds on provisions that are already present in RFC 9293 "Transmission Control Protocol (TCP)". As such, this document does not require any change to RFC 9293. Discussion Venues Discussion of this document takes place on the TCP Maintenance and Minor Extensions mailing list (tcpm@ietf.org), which is archived at . Source for this draft and an issue tracker can be found at .
Introduction A TCP connection can be reset by a peer for various reasons, e.g., received data does not correspond to an active connection. Also, a TCP connection can be reset by an on-path service function (e.g., Carrier Grade NAT (CGN) , NAT64 , or firewall) for several reasons. Typically, a Network Address Translator (NAT) function can generate an RST segment to notify an endpoint upon the expiry of the lifetime of the corresponding mapping entry or because an RST segment was received from a peer (). A TCP connection can also be closed by a user or an application at any time. However, the peer that receives an RST segment does not have any hint about the reason that led to terminating the connection. Likewise, the application that relies upon such a TCP connection may not easily identify the reason for the connection closure. Troubleshooting such events at the remote side of the connection that receives the RST segment may not be trivial. This document fills this void by specifying a format of the diagnostic payload that is returned in an RST segment. Returning such data is consistent with the provision in for RST segments, especially:
"TCP implementations SHOULD allow a received RST segment to include data (SHLD-2)."
This document does not change the conditions under which an RST segment is generated (). The generic procedure for processing an RST segment is specified in . Only the deviations from that procedure to insert and validate a diagnostic payload is provided in . provides a set of examples to illustrate the use of TCP RST diagnostic payloads. This document specifies the format and the overall approach to ease maintaining the list of codes while allowing for adding new codes as needed in the future and accommodating any existing vendor-specific codes. An initial version of error codes is available in Table 2. However, the authoritative source to retrieve the full list of error codes is the IANA-maintained registry (). Investigation based on some major CGN vendors revealed that RSTs with data are not discarded and are translated according to any matching mapping entry. Moreover, implementation and experimental validation in Linux are detailed in .
Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document makes use of the terms defined in . SEG.LEN is defined in .
RST Diagnostic Payload The format of the RST diagnostic payload is shown in .
Structure of the RST Diagnostic Payload
The RST diagnostic payload comprises a magic cookie that is used to unambiguously identify an RST payload that follows this specification. It MUST be set to 0x33AA. The descriptions of other fields shown in are as follows:
Length:
Indicates the total length, in octets, of the diagnostic payload that follows.
Reason Length:
Indicates the length, in octets, of the reason-description field.
If set to a non-null zero, this means that the reason code is not present.
reason-code:
This field, if present, takes a value from an available registry such as the "TCP Failure Causes" registry (). Value 0 is reserved and MUST NOT be used.
reason-description:
Includes a brief description of the reset reason encoded as UTF-8 .
This parameter MUST NOT be included if a reason code is supplied; Reason Length MUST be set to 0 for such a case.
This parameter is useful only for reset reasons that are not yet registered or for application-specific reset reasons.
pen:
Includes a Private Enterprise Number (PEN) .
This parameter MAY be included when the reason code is not taken from the IANA-maintained registry (), but from a vendor-specific registry. That is, if the "pen" parameter is present, then the reason code refers to the registry of the entity specified by the "pen" parameter.
The "pen" parameter MUST be omitted if a reason code refers to the IANA-maintained registry ().
The presence of this field is inferred from the values of Length and Reason Length fields. Specifically, 'Length' MUST be set to "'Reason Length' + 4" if no reason-code is supplied or "8" if a reason code is present.
At least one of "reason-code" and "reason-description" parameters MUST be included in an RST diagnostic payload. If SEG.LEN > Length + 4, the receiver processes the diagnostic payload but ignores the rest of the data in the segment payload. If SEG.LEN < Length + 4, the segment is consided as malformed RST. Malformed RST diagnostic payload messages that include the magic cookie MUST be silently ignored by the receiver. A peer that receives a valid diagnostic payload may pass the reset reason information to the local application in addition to the information (MUST-12) described in . That information may also be logged locally, unless a local policy specifies otherwise. How the information is passed to an application and how it is stored locally is implementation-specific.
Some Examples depicts an example of an RST diagnostic payload that is generated to inform the peer that the TCP connection is reset because an ACK was received from that peer while the connection is still in the LISTEN state ().
Example of an RST Diagnostic Payload with Reason Code
An RST diagnostic payload may also be sent by an on-path service function. For example, the following diagnostic payload is returned by a NAT function upon expiry of the mapping entry to which the TCP connection is bound ().
Example of an RST Diagnostic Payload to Report Connection Timeout
illustrates an RST diagnostic payload that is returned by a peer that resets a TCP connection for a reason code 1234 defined by a vendor with the private enterprise number 32473.
Example of an RST Diagnostic Payload to Report Vendor-Specific Reason Code
uses the Enterprise Number 32473 defined for documentation use .
IANA Considerations
New Registry for TCP Failure Causes This document requests IANA to create a new registry entitled "TCP Failure Causes" under the "Transmission Control Protocol (TCP) Parameters" registry group . Values are taken from the 1-65535 range. The assignment policy for this registry is "Expert Review" (). The designated experts may approve registration once they checked that the new requested code is not covered by an existing code and if the provided reasoning to register the new code is acceptable. A registration request may supply a pointer to a specification where that code is defined. However, a registration may be accepted even if no permanent and readily available public specification is available. The registry is initially populated with the values listed in . Initial TCP Failure Causes
Value Description Specification (if available)
0 Reserved [ThisDocument]
1 Illegal Option
2 Desynchronized state
3 New data is received after CLOSE is called Sections 3.6.1 and 3.10.7.1 of
4 ABORT Process
5 Unexpected ACK received by non-synchronized state connection
6 Unexpected SYN in the window
7 Unexpected security compartment
8 Malformed Message [ThisDocument]
9 Not Authorized [ThisDocument]
10 Resource Exceeded [ThisDocument]
11 Network Failure [ThisDocument]
12 Reset received from the peer [ThisDocument]
13 Destination Unreachable [ThisDocument]
14 Connection Timeout [ThisDocument]
15 Too much outstanding data
16 Unacceptable performance
17 Middlebox interference
Note that codes in the 8-14 range can be used by service functions (Carrier Grade NAT (CGN), firewall, proxy, etc.).
Operational Considerations
Multiple RSTs Per , one or more RST segments can be sent to reset a connection. Sending more RST segments to reset a connection can be used to mitigate deployment contexts where some on-path devices may discard RSTs with payload data. Whether a TCP endpoint elects to send more than one RST with only a subset of them that include the diagnostic payload is implementation-specific.
Manageability TCP server implementations should support the following parameters:
  • A parameter to control the activation of the RST diagnostic.
  • A parameter to accept/discard RSTs with diagnostic payload other than reason cause (i.e., accept or deny RSTs with reason-description).
  • A parameter to set a maximum length of acceptable reason-description, when enabled.
  • A parameter to control whether "empty" RSTs are also sent together with RST with diagnostic payload.
  • A rate-limit of RST with diagnostic payload.
  • Counters to track sent/received RSTs with diagnostic payload.
  • Counters to track received invalid RSTs with diagnostic payload.
Security Considerations discusses TCP-related security considerations. In particular, RST-specific attacks and their mitigations are discussed in . In addition to these considerations, it is RECOMMENDED to control the size of acceptable diagnostic payload and keep it as brief as possible. The RECOMMENDED acceptable maximum size of the RST diagnostic payload is 255 octets. Also, it is RECOMMENDED to avoid leaking privacy-related information as part of the diagnostic payload (e.g., including a description such as "user X resets explicitly the connection" is not recommended). The "reason-description" string, when present, MUST NOT include any private information that an observer would not otherwise have access to. The presence of vendor-specific reason codes (Section 3) may be used to fingerprint hosts. Such a concern does not apply if the reason codes are taken from the IANA-maintained registry. Implementers are, thus, encouraged to register new codes within IANA instead of maintaining specific registries. The reason description, when present, MUST NOT be displayed to end users but is intended to be consumed by applications. Such a description may carry a malicious message to mislead the end-user.
References Normative References Transmission Control Protocol (TCP) This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. UTF-8, a transformation format of ISO 10646 ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279. Guidelines for Writing an IANA Considerations Section in RFCs Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. TCP Extensions for Multipath Operation with Multiple Addresses TCP/IP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers. The simultaneous use of these multiple paths for a TCP/IP session would improve resource usage within the network and thus improve user experience through higher throughput and improved resilience to network failure. Multipath TCP provides the ability to simultaneously use multiple paths between peers. This document presents a set of extensions to traditional TCP to support multipath operation. The protocol offers the same type of service to applications as TCP (i.e., a reliable bytestream), and it provides the components necessary to establish and use multiple TCP flows across potentially disjoint paths. This document specifies v1 of Multipath TCP, obsoleting v0 as specified in RFC 6824, through clarifications and modifications primarily driven by deployment experience. Informative References Transmission Control Protocol (TCP) Parameters Private Enterprise Numbers Common Requirements for Carrier-Grade NATs (CGNs) This document defines common requirements for Carrier-Grade NATs (CGNs). It updates RFC 4787. Stateful NAT64: Network Address and Protocol Translation from IPv6 Clients to IPv4 Servers This document describes stateful NAT64 translation, which allows IPv6-only clients to contact IPv4 servers using unicast UDP, TCP, or ICMP. One or more public IPv4 addresses assigned to a NAT64 translator are shared among several IPv6-only clients. When stateful NAT64 is used in conjunction with DNS64, no changes are usually required in the IPv6 client or the IPv4 server. Updates to Network Address Translation (NAT) Behavioral Requirements This document clarifies and updates several requirements of RFCs 4787, 5382, and 5508 based on operational and development experience. The focus of this document is Network Address Translation from IPv4 to IPv4 (NAT44). This document updates RFCs 4787, 5382, and 5508. Enterprise Number for Documentation Use This document describes an Enterprise Number (also known as SMI Network Management Private Enterprise Code) for use in documentation. This memo provides information for the Internet community. The Constrained Application Protocol (CoAP) The Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained (e.g., low-power, lossy) networks. The nodes often have 8-bit microcontrollers with small amounts of ROM and RAM, while constrained networks such as IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs) often have high packet error rates and a typical throughput of 10s of kbit/s. The protocol is designed for machine- to-machine (M2M) applications such as smart energy and building automation. CoAP provides a request/response interaction model between application endpoints, supports built-in discovery of services and resources, and includes key concepts of the Web such as URIs and Internet media types. CoAP is designed to easily interface with HTTP for integration with the Web while meeting specialized requirements such as multicast support, very low overhead, and simplicity for constrained environments.
Implementation and Experimental Validation in Linux Questions and concerns have been raised regarding whether RST with payload affects the normal termination of flows across different software platforms, operating systems, middleboxes, etc. Even though explicitly allows this behavior, a full implementation is needed to widely verify if unexpected cases can happen in the real world. The overall design in Linux is to pre-allocate a large enough zeroed buffer, put a reset reason code in the first byte and sent it out to verify whether the RST with payload can be possibly declined by any equipment in between two sides and the other side successfully parses the RST with payload.
Implementation The following implementation is accomplished on top of Linux 6.16:
Payload Attachment:
Allocate a 1000-byte data payload attached to all generated RST packets.
Reason Code Encoding:
The first byte of the payload is used to store a predefined reset reason code that is listed in include/net/rstreason.h file, while the remainder of the payload is zero-padded. The reason code is generated by the existing mechanism called TCP reset reasons.
Handling of Reset Types:
The implementation distinguishes between the two primary reset scenarios in tcp_send_active_reset() and tcp_v4_send_reset() respectively:
  • For an Active Reset, initiated proactively by the local system, the payload is placed in the linear area of the socket buffer (sk_buff).
  • For a Passive Reset, sent in response to an unexpected or invalid incoming packet, the payload is stored in the non-linear (paged) area of the sk_buff.
Complete patch is shown in .
Complete Patch sk_mark = 0; ctl_sk->sk_priority = 0; } + memcpy(buffer, (char *)&rep, arg.iov[0].iov_len); + /* put rst reason into the first byte in payload */ + buffer[arg.iov[0].iov_len] = reason; + arg.iov[0].iov_len += PAYLOAD_LEN; ip_send_unicast_reply(ctl_sk, sk, skb, &TCP_SKB_CB(skb)->header.h4.opt, ip_hdr(skb)->saddr, ip_hdr(skb)->daddr, diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index b616776e3354..c07dd009a0de 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3628,12 +3628,14 @@ void tcp_send_fin(struct sock *sk) void tcp_send_active_reset(struct sock *sk, gfp_t priority, enum sk_rst_reason reason) { + u32 len = MAX_TCP_HEADER + PAYLOAD_LEN; + char payload[PAYLOAD_LEN]; struct sk_buff *skb; TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTRSTS); /* NOTE: No TCP options attached and we never retransmit this. */ - skb = alloc_skb(MAX_TCP_HEADER, priority); + skb = alloc_skb(len, priority); if (!skb) { NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED); return; @@ -3641,8 +3643,13 @@ void tcp_send_active_reset(struct sock *sk, gfp_t priority, /* Reserve space for headers and prepare control bits. */ skb_reserve(skb, MAX_TCP_HEADER); + skb_put(skb, PAYLOAD_LEN); tcp_init_nondata_skb(skb, tcp_acceptable_seq(sk), TCPHDR_ACK | TCPHDR_RST); + memset(payload, 0, PAYLOAD_LEN); + payload[0] = reason; + skb_store_bits(skb, 0, payload, PAYLOAD_LEN); + TCP_SKB_CB(skb)->end_seq += PAYLOAD_LEN; tcp_mstamp_refresh(tcp_sk(sk)); /* Send it off. */ if (tcp_transmit_skb(sk, skb, 0, priority)) ]]>
Experimental Validation To ensure a thorough evaluation, a multi-layered experimental methodology was designed, progressing from basic functional checks to complex, real-world compatibility and stability tests. The whole implementation has been deployed in Tencent's production environment for almost six months.
Functional Verification The basic functionality test is using iperf or iperf3 to construct a normal termination senario. The tcpdump tool with -X option effectively helps to show the [RST+] flag and the 1000-byte payload, confirming that the kernel correctly generated and transmitted the augmented RST packets. Two servers, designated as Client A and Server B. The test is conducted as following:
  1. Start the iperf3 server on Server B (iperf3 -s).
  2. Initiate a connection from Client A to Server B (iperf3 -c [IP_of_B]).
  3. After the connection is established, one of the iperf3 processes is terminated using the kill command, triggering the kernel to send an RST packet.
  4. Simultaneously, tcpdump is run on either host to capture the reset packet using the filter: 'tcp[tcpflags] & tcp-rst != 0' -X -nn -vv -S.
Compatibility Verification
Hardwares and Kernels:
Tests were conducted on various Linux distributions (e.g., Ubuntu or CentOS) with different kernel versions. The physical hosts were equipped with a range of network interface cards (NICs), including Intel i40e, ixgbe, and Mellanox mlx5.
Virtualization:
The mechanism was tested in a virtualized environment where the VM used a virtio_net driver and the host employed DPDK to redirect packets in the host.
Middleboxes:
Tests were performed with Layer 4 (L4) and Layer 7 (L7) gateways placed between the client and server to verify correct packet parsing and forwarding.
Wide Area Network (WAN):
The setup was tested over long-haul international links to simulate complex conditions, including China-to-Singapore (RTT > 30ms) and China-to-Germany (RTT > 200ms).
In conclusion, across all complex environment tests, the RST packets with payloads were successfully received by the peer. No instances of packets being dropped or mishandled by intermediate middleboxes, gateways, or diverse hardware and software configurations were observed.
Acknowledgments The "diagnostic payload" name is inspired by that was cited by Carsten Bormann in the tcpm mailing list. Thanks to Jon Shallow and Gleb Smirnoff for the comments. Thanks also to Li Jinghui for the discussion.