| tls | B. Schwartz |
| Internet-Draft | Google LLC |
| Intended status: Standards Track | October 31, 2019 |
| Expires: May 3, 2020 |
TLS Metadata for Load Balancers
draft-schwartz-tls-lb-02
A load balancer that does not terminate TLS may wish to provide some information to the backend server, in addition to forwarding TLS data. This draft proposes a protocol between load balancers and backends that enables secure, efficient delivery of TLS with additional information. The need for such a protocol has recently become apparent in the context of split mode ESNI.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 3, 2020.
Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
Data encodings are expressed in the TLS 1.3 presentation language, as defined in Section 3 of [TLS13].
A load balancer is a server or bank of servers that acts as an intermediary between the client and a range of backend servers. As the name suggests, a load balancer’s primary function is to ensure that client traffic is spread evenly across the available backend servers. However load balancers also serve many other functions, such as identifying connections intended for different backends and forwarding them appropriately, or dropping connections that are deemed malicious.
A load balancer operates at a specific point in the protocol stack, forwarding e.g. IP packets, TCP streams, TLS contents, HTTP requests, etc. Most relevant to this proposal are TCP and TLS load balancers. TCP load balancers terminate the TCP connection with the client and establish a new TCP connection to the selected backend, bidirectionally copying the TCP contents between these two connections. TLS load balancers additionally terminate the TLS connection, forwarding the plaintext to the backend server (typically inside a new TLS connection). TLS load balancers must therefore hold the private keys for the domains they serve.
When a TCP load balancer forwards a TLS stream, the load balancer has no way to incorporate additional information into the stream. Insertion of any additional data would cause the connection to fail. However, the load-balancer and backend can share additional information if they agree to speak a new protocol. The most popular protocol used for this purpose is currently the PROXY protocol [PROXY], developed by HAPROXY. This protocol prepends a plaintext collection of metadata (e.g. client IP address) onto the TCP socket. The backend can parse this metadata, then pass the remainder of the stream to its TLS library.
The PROXY protocol is effective and widely used, but it offers no confidentiality or integrity protection, and therefore might not be suitable when the load balancer and backend communicate over the public internet. It also does not offer a way for the backend to reply.
The proposed protocol supports a two-way exchange between a load balancer and a backend server. It works by prepending information to the TLS handshake:
+-----------+ +-----------+ +-----------+
| Backend A | | Backend B | | Backend C |
+-----------+ +-----------+ +-----------+
\/ /\
4. EncryptedProxyData[ \/ /\ 3. ClientHello (verbatim)
got SNI info] \/ /\ 2. EncryptedProxyData[
5. ServerHello, etc. \/ /\ SNI="secret.b",
\/ /\ client=2, etc.]
\/ /\
+---------------+
| Load balancer |
+---------------+
\/ /\
6. ServerHello, etc. \/ /\ 1. ClientHello[
(verbatim) \/ /\ ESNI=enc("secret.b")]
\/ /\
+-----------+ +-----------+ +-----------+
| Client 1 | | Client 2 | | Client 3 |
+-----------+ +-----------+ +-----------+
Figure 1: Data flow diagram
A ProxyExtension is identical in form to a standard TLS Extension (Section 4.2 of [TLS13]), with a new identifier space for the extension types.
struct {
ProxyExtensionType extension_type;
opaque extension_data<0..2^16-1>;
} ProxyExtension;
ProxyExtensions can be sent in an upstream (to the backend) or downstream (to the load balancer) direction
enum {
upstream(0),
downstream(1),
(255)
} ProxyDataDirection;
The ProxyData contains a set of ProxyExtensions.
struct {
ProxyDataDirection direction;
ProxyExtension proxy_data<0..2^16-1>;
} ProxyData;
The EncryptedProxyData structure contains metadata associated with the original ClientHello (Section 4.1.2 of [TLS13]), encrypted with a pre-shared key that is configured out of band.
struct {
opaque psk_identity<1..2^16-1>;
opaque nonce<8..2^16-1>
opaque encrypted_proxy_data<1..2^16-1>;
} EncryptedProxyData;
When the load balancer receives a ClientHello, it serializes any relevant metadata into an upstream ProxyData, then encrypts it with the ClientHello as additional_data to produce the EncryptedProxyData. The backend’s reply is a downstream ProxyData struct, also transmitted as an EncryptedProxyData, using the upstream EncryptedProxyData as additional_data. Recipients in each case MUST verify that ProxyData.direction has the expected value, and discard the connection if it does not.
The downstream ProxyData SHOULD NOT contain any ProxyExtensionType values that were not present in the upstream ProxyData.
Like a standard TLS Extension, a ProxyExtension is identified by a uint16 type number. Load balancers MUST only include extensions that are registered for use in ProxyData. Backends MUST ignore any extensions that they do not recognize.
There are initially seven type numbers allocated:
enum {
padding(0),
client_address(1),
destination_address(2),
esni_inner(3),
certificate_padding(4),
overload(5),
ratchet(6),
(65535)
} ProxyExtensionType;
The “padding” extension functions as described in [RFC7685]. It is used here to avoid leaking information about the other extensions. It can be used in upstream and downstream ProxyData.
The “client_address” extension functions as described in [I-D.kinnear-tls-client-net-address]. It conveys the client IP address observed by the load balancer. Backends that make use of this extension SHOULD include an empty “client_address” extension in the downstream ProxyData.
The “destination_address” extension is identical to the “client_address” extension, except that it contains the load balancer’s server IP address that received this connection.
The “esni_inner” extension is only sent upstream, and can only be used if the ClientHello contains the encrypted_server_name extension [ESNI]. The extension_data is the ClientESNIInner (Section 5.1.1 of [ESNI]), which contains the true SNI and nonce. This is useful when the load balancer knows the ESNI private key and the backend does not, i.e. split mode ESNI.
The “certificate_padding” extension always contains a single uint32 value. The upstream value conveys the padding granularity G, and the downstream value indicates the unpadded size of the Certificate struct (Section 4.4.2 of [TLS13]).
To pad the Handshake message (Section 4 of [TLS13]) containing the Certificate struct, the backend SHOULD select the smallest length_of_padding (Section 5.2 of [TLS13]) such that Handshake.length + length_of_padding is a multiple of G.
The load balancer SHOULD include this extension whenever it sends the “esni_inner” extension.
Padding certificates from many backends to the same length is important to avoid revealing which backend is responding to a ClientHello. Load balancer operators SHOULD ensure that no backend has a unique certificate size after padding, and MAY set G large enough to make all responses have equal size.
In the upstream ProxyData, the “overload” extension contains a single uint16 indicating the approximate proportion of connections that are being routed to this server as a fraction of 65535. If there is only one server, load balancers SHOULD set the value to 65535.
In the downstream ProxyData, the value is an OverloadValue:
enum {
accepted(0),
overloaded(1),
rejected(2),
(255)
} OverloadState;
struct {
OverloadState state;
uint16 load;
uint32 ttl;
} OverloadValue;
When OverloadValue.state is “accepted”, the backend is accepting connections normally. The “overloaded” state indicates that the backend is accepting this connection, but would prefer not to receive additional connections. A value of “rejected” indicates that the backend did not accept this connection. When sending a “rejected” response, the backend SHOULD close the connection without sending a ServerHello.
OverloadValue.load indicates the load fraction of the responding backend server, with 65535 indicating maximum load.
The load balancer SHOULD treat this information as valid for OverloadValue.ttl seconds, or until it receives another OverloadValue from that server.
Load balancers that have multiple available backends for an origin SHOULD avoid connecting to servers that are in the “overloaded” or “rejected” state. When a connection is rejected, the load balancer MAY retry that connection by sending the ClientHello to a different backend server. When multiple servers are in the “accepted” state, the load balancer MAY use OverloadValue.load to choose among them.
When there is a server in an unknown state (i.e. a new server or one whose last TTL has expired), the load balancer SHOULD direct at least one connection to it, in order to refresh its OverloadState.
If all servers are in the “overloaded” or “rejected” state, the load balancer SHOULD drop the connection.
If the backend server is reachable without traversing the load balancer, and an adversary can observe packets on the link between the load balancer and the backend, then that adversary can execute a replay flooding attack, sending the backend server duplicate copies of observed EncryptedProxyData and ClientHello. This attack can waste server resources on the Diffie-Hellman operations required to process the ClientHello, resulting in denial of service.
The “ratchet” extension reduces the impact of such an attack on the backend server by allowing the backend to reject these duplicates after decrypting the ProxyData. (This decryption uses only a symmetric cipher, so it is expected to be much faster than typical Diffie-Hellman operations.) Its upstream payload consists of a RatchetValue:
struct {
uint64 index;
uint64 floor;
} RatchetValue;
A RatchetValue is scoped to a single backend server and psk_identity. Within that scope, the load balancer initializes index to a random value, and executes the following procedure:
The backend server initializes floor to the first RatchetValue.floor it receives (under a psk_identity), and then executes the following procedure for each incoming connection:
With these measures in place, replays can be rejected without processing the ClientHello.
In principle, this replay protection fails after 2^64 connections when the floor value wraps. On a backend server that averages 10^9 new connections per second, this would occur after 584 years. To avoid this replay attack, load balancers and backends SHOULD establish a new PSK at least this often.
Backends that are making use of the “ratchet” extension SHOULD include an empty “ratchet” extension in their downstream ProxyData.
When forwarding a TLS stream over TCP, the load balancer SHOULD prepend a TLSPlaintext whose content_type is XX (proxy_header) and whose fragment is the EncryptedProxyData.
Following this proxy header, the load balancer MUST send the full contents of the TCP stream, exactly as received from the client. The backend will observe the proxy header, immediately followed by a TLSPlaintext containing the ClientHello. The backend will decrypt the EncryptedProxyData using the ClientHello as associated data, and process the ClientHello and the remainder of the stream as standard TLS.
Similarly, the backend SHOULD reply with the downstream EncryptedProxyData in a proxy header, followed by the normal TLS stream, beginning with a TLSPlaintext frame containing the ServerHello. If the downstream ProxyHeader is not present, has an unrecognized version number, or produces an error, the load balancer SHOULD proxy the rest of the stream regardless.
This protocol is intended to provide both parties with a strong guarantee of integrity for the metadata they receive. For example, an active attacker cannot take metadata intended for one stream and attach it to another, because each stream will have a unique ClientHello, and the metadata is bound to the ClientHello by AEAD.
One exception to this protection is in the case of an attacker who deliberately reissues identical ClientHello messages. An attacker who reuses a ClientHello can also reuse the metadata associated with it, if they can first observe the EncryptedProxyData transferred between the load balancer and the backend. This could be used by an attacker to reissue data originally generated by a true client (e.g. as part of a 0-RTT replay attack), or it could be used by a group of adversaries who are willing to share a single set of client secrets while initiating different sessions, in order to reuse metadata that they find helpful.
Backends that are sensitive to this attack SHOULD implement the “ratchet” mechanism in Section 6.7, including the optional defenses.
This protocol is intended to maintain confidentiality of the metadata transferred between the load balancer and backend, especially the ESNI plaintext and the client IP address. An observer between the client and the load balancer does not observe this protocol at all, and an observer between the load balancer and backend observes only ciphertext.
However, an adversary who can monitor both of these links can easily observe that a connection from the client to the load balancer is shortly followed by a connection from the load balancer to a backend, with the same ClientHello. This reveals which backend server the client intended to visit. In many cases, the choice of backend server could be the sensitive information that ESNI is intended to protect.
Connections to different domains might be distinguishable by the cleartext contents of the ServerHello, such as cipher_suite and server_share.group. Load balancer operators with ESNI support SHOULD provide backend operators with a list of cipher suites and groups to support, and a preference order, to avoid different backends having distinctive behaviors.
IANA will be directed to add the following allocation to the TLS ContentType registry:
| Value | Description | DTLS-OK | Reference |
|---|---|---|---|
| XX | proxy_header | N | This document |
IANA will be directed to create a new “TLS ProxyExtensionType Values” registry on the TLS Extensions page. Values less than 0x8000 will be subject to the “RFC Required” registration procedure, and the rest will be “First Come First Served”. To avoid codepoint exhaustion, proxy developers SHOULD pack all their nonstandard information into a single ProxyExtension.
| [ESNI] | Rescorla, E., Oku, K., Sullivan, N. and C. Wood, "Encrypted Server Name Indication for TLS 1.3", Internet-Draft draft-ietf-tls-esni-04, July 2019. |
| [I-D.kinnear-tls-client-net-address] | Kinnear, E., Pauly, T. and C. Wood, "TLS Client Network Address Extension", Internet-Draft draft-kinnear-tls-client-net-address-00, March 2019. |
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997. |
| [RFC7685] | Langley, A., "A Transport Layer Security (TLS) ClientHello Padding Extension", RFC 7685, DOI 10.17487/RFC7685, October 2015. |
| [RFC8174] | Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017. |
| [TLS13] | Rescorla, E., "The Transport Layer Security (TLS) Protocol Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018. |
| [PROXY] | Tarreau, W., "The PROXY protocol", March 2017. |
This is an elaboration of an idea proposed by Eric Rescorla during the development of ESNI. Thanks to David Schinazi, David Benjamin, and Piotr Sikora for suggesting important improvements.