Internet DRAFT - draft-rousskov-icap-trailers
draft-rousskov-icap-trailers
Network Working Group A. Rousskov
Internet-Draft The Measurement Factory
Updates: 3507 (if approved) October 10, 2016
Intended status: Informational
Expires: April 13, 2017
ICAP Trailers
draft-rousskov-icap-trailers-01
Abstract
This document defines an ICAP trailer feature which allows ICAP
agents to reliably send message metadata after the message body. The
ICAP trailer is independent from the HTTP trailer that might also be
encapsulated in an ICAP message. ICAP changes defined here are
backward compatible and address a long-standing ICAP specification
errata entry.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 13, 2017.
Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
Rousskov Expires April 13, 2017 [Page 1]
Internet-Draft ICAP Trailers October 2016
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Motivation and Design Choices . . . . . . . . . . . . . . . . 2
2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Overall Operation . . . . . . . . . . . . . . . . . . . . . . 4
4. Notations . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5. Extended Use of the Allow Header Field . . . . . . . . . . . 5
6. Message Syntax . . . . . . . . . . . . . . . . . . . . . . . 6
7. Trailer Field Syntax . . . . . . . . . . . . . . . . . . . . 6
8. Client Requirements . . . . . . . . . . . . . . . . . . . . . 7
9. Server Requirements . . . . . . . . . . . . . . . . . . . . . 8
10. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 9
11. Security Considerations . . . . . . . . . . . . . . . . . . . 11
12. Normative References . . . . . . . . . . . . . . . . . . . . 12
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12
1. Motivation and Design Choices
ICAP [RFC3507] specification says that the Trailer header field is
"defined in ICAP the same as [...] in HTTP". Unfortunately, that
phrase alone is not enough for trailer-related interoperability in
the ICAP context because of the following conflicting
interpretations, requirements, and needs:
o Both ICAP and HTTP message headers might contain a Trailer field.
o HTTP messages might contain HTTP trailers (that ICAP servers could
be interested in receiving or even sending). An HTTP trailer can
be present with or without an HTTP Trailer header field.
o ICAP agents need to distinguish an HTTP trailer from an ICAP
trailer.
o HTTP uses Chunked Transfer Coding [RFC7230] to transmit trailers.
The chunked coding is applied to the entire HTTP message body.
This choice places an HTTP trailer inside an HTTP message body.
o It is possible to interpret the ICAP specification as placing the
ICAP trailer either inside the HTTP message body or inside the
ICAP message body.
o ICAP does not chunk-encode ICAP message bodies. Instead, ICAP
message bodies contain a combination of zero, one, or two HTTP
headers possibly followed by a chunked-encoded HTTP message body.
Rousskov Expires April 13, 2017 [Page 2]
Internet-Draft ICAP Trailers October 2016
o Chunked coding does not support multiple trailers: Chunked HTTP
messages always contain exactly one (but possibly empty) trailer
part.
o HTTP effectively restricts trailers usage to messages with bodies,
presumably because, without a body, the information in the trailer
can usually be placed in the message header. In ICAP context, it
is not obvious whether trailers ought to be restricted to messages
with HTTP bodies (embedded in ICAP bodies) or to messages with
ICAP bodies (that might only contain HTTP headers and no HTTP
body).
These problems led to a ban on ICAP trailers [Errata].
Several designs were considered for introducing proper ICAP trailers
support:
1. Extend chunked coding to support multiple trailers (one for HTTP
and one for ICAP). This option was rejected because many ICAP
agents use existing HTTP-focused libraries to parse embedded HTTP
bodies. As anecdotal evidence related to the ICAP-only "ieof"
chunk extension support shows, it would be difficult to extend
those libraries to handle a complicated ICAP-only extension.
Also, this design would make it difficult to send an ICAP trailer
when processing large HTTP messages without bodies.
2. Embed ICAP trailer fields inside the chunked HTTP message body
trailer, using an ICAP-specific field name prefix (e.g.,
"ICAP-"). This option was rejected because it would either allow
malicious HTTP messages to inject ICAP trailers or require ICAP
clients to hide conflicting HTTP trailer fields from the ICAP
server. This design also badly violates layering boundaries by
mixing HTTP- and ICAP-level information in the same protocol
structure.
3. Extend Encapsulated header with a "trailer" token. This option
was rejected because the Encapsulated header describes embedded
HTTP message parts and an ICAP trailer is not a part of any HTTP
message. In other words, ICAP trailers do not get encapsulated.
4. Clarify the ICAP Trailer semantics (and transfer mechanism)
without introducing any new trailer support-negotiation
mechanism. This option was rejected because trailers affect
message framing and many existing ICAP agent implementations
cannot parse any form of trailers.
5. Add a new trailer support-negotiation mechanism (e.g., "Allow:
trailers") and a new trailer presence-signaling mechanism (e.g.,
Rousskov Expires April 13, 2017 [Page 3]
Internet-Draft ICAP Trailers October 2016
Trailer2) while leaving the poorly defined Trailer header
semantics as is. This option was narrowly rejected because a new
trailer support-negotiation mechanism alone was deemed sufficient
to resolve conflicts between this specification and any
reasonable existing implementation of the poorly defined Trailer
semantics.
6. Add a new trailer support-negotiation mechanism and only clarify
Trailer header semantics (and transfer mechanism) upon successful
negotiation, while reusing the well-known Trailer header field
name as the trailer presence-signaling mechanism (requiring
successful support negotiation). This specification documents
this design.
2. Use Cases
Trailers allow an ICAP agent to transmit metadata after the message
body. Such delayed transmission is useful when the same information
was not available at the start of the message transmission. For
example:
o A client uses an ICAP trailer to relay the current HTTP/1.1
connection or HTTP/2 stream status after transmitting a large HTTP
message. A server uses that information to optimize message
analysis (e.g., skip or abort analysis of HTTP requests sent by
already disconnected HTTP clients).
o A server uses an ICAP trailer to relay audit information about
viruses present at the end of a large HTTP message.
o A server uses an ICAP trailer to relay prefetching information
about HTML parts referenced from a large HTTP message.
3. Overall Operation
This section informally describes the overall feature operation.
This description is deliberately imprecise and cannot be used to
build compliant implementations. The following sections contain
actual protocol requirements.
To announce feature support, ICAP agents exchange "Allow: trailers"
settings during an OPTIONS transaction. To send a trailer at the end
of a particular REQMOD or RESPMOD transaction, the agent first sends
both "Allow: trailers" and "Trailer" header fields. The Trailer
field lists header field names expected in the message trailer
section. After sending the entire ICAP message body, the agent sends
the trailer section (a.k.a. "trailer"). The trailer section is
syntactically equivalent to the ICAP message header section. The
Rousskov Expires April 13, 2017 [Page 4]
Internet-Draft ICAP Trailers October 2016
trailer section does not have to contain any of the promised fields
and might even have no fields at all.
A trailer makes sense only in an ICAP message with a body. However,
a trailer could be sent if the ICAP message body encapsulates just
HTTP headers.
If a trailer is sent, its bytes are always the last bytes sent during
the entire ICAP transaction. Thus, a client never sends a trailer at
the end of Preview unless it sent the "ieof" chunk extension as well.
Similarly, a server never a sends trailer with a 100 (Continue)
control message.
4. Notations
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
Conformance criteria and error handling considerations are defined in
Section 2.5 of [RFC7230].
This specification uses the Augmented Backus-Naur Form (ABNF)
notation of [RFC5234] with a list extension defined in Section 7 of
[RFC7230]. All syntax rules not explicitly defined in this
specification (e.g., header-field and CRLF) are defined in (or
included by reference from) [RFC7230].
The "Allow/X" notation is defined in Section 5.
5. Extended Use of the Allow Header Field
The use of the Allow header field defined in sections 4.6 and 4.10.2
of [RFC3507] is extended while preserving its original syntax and
general semantics (i.e., a comma-separated list of tokens, each
identifying an ICAP feature supported by the sending agent):
Allow = 1#token
[RFC3507] defines Allow usage in OPTIONS requests and responses.
This specification extends Allow usage to other messages and adds
"trailers" to the list of possible Allow value tokens.
An agent MUST treat multiple Allow header fields as one Allow field
with a comma-separated list of individual field value tokens,
concatenated in the order of their appearance in the ICAP header. An
ICAP agent MUST ignore Allow header tokens it does not understand.
This document does not specify the significance of Allow tokens order
and impact of repeated tokens.
Rousskov Expires April 13, 2017 [Page 5]
Internet-Draft ICAP Trailers October 2016
An agent MAY send an Allow header in any message. Such header
contains a list of ICAP features supported by the sending agent. For
example, a client could send an "Allow: trailers" in a REQMOD request
and receive an "Allow: 204, trailers, 206" header field in response.
The exact meaning of each Allow token in a context of its message is
defined by the corresponding feature specification. By sending
Allow, the agent indicates compliance with each listed feature's
specification but does not necessarily commit to offer or use any of
the listed features during future transactions.
To keep this specification succinct, we introduce "Allow/X" notation
to mean an ICAP Allow header field which value contains an "X" token,
possibly among other tokens. For example, "Allow/trailers" stands
for an Allow header field with a "trailers" token and possibly other
tokens. This notation covers multi-field Allow headers as well
because they are equivalent to a combined single-field Allow header.
The same approach to extending Allow header usage was successfully
applied to the ICAP 206 extension (XXX: reference our expired ICAP
icap-ext-partial-content draft). This specification is compatible
with the ICAP 206 extension.
6. Message Syntax
An ICAP message with a trailer is a concatenation of a regular ICAP
message and an trailer section. The trailer section syntax is
identical to the ICAP header syntax:
ICAP-message-without-trailer = <see [RFC3507]>
ICAP-message-with-trailer = ICAP-message-without-trailer
trailer-section
trailer-section = *( trailer-field CRLF )
CRLF
trailer-field = header-field
Note that any trailer, even a trailer without fields, ends with CRLF.
That terminating sequence is essential for proper message framing on
persistent ICAP connections.
A sender MUST NOT generate a trailer section that contains a field
necessary for message framing (e.g., Encapsulated, Preview, and
Trailer), routing (e.g., Host), or authentication.
7. Trailer Field Syntax
The ICAP Trailer header value syntax is identical to the HTTP Trailer
syntax [RFC7230]:
Rousskov Expires April 13, 2017 [Page 6]
Internet-Draft ICAP Trailers October 2016
Trailer = 1#field-name
A Trailer header field sender SHOULD enumerate the names of all
expected trailer fields. This a priori knowledge of trailer fields
might help the recipient with trailer processing (e.g., certain
message annotation actions could be delayed in anticipation of the
trailer section). However, enumerating all expected trailer fields
can be impractical or even impossible in some environments. A
Trailer sender MAY send a trailer section with a set of field names
that differs from the set of field names listed in the Trailer header
field.
This specification does not place any restriction on the order of
field names in the Trailer header field. Senders SHOULD NOT generate
duplicate names for the Trailer header field.
8. Client Requirements
A client compliant with this specification SHOULD send Allow/trailers
in each OPTIONS request. A non-authenticating server cannot be
expected to mark an Allow/trailers-sending client specially, but this
support announcement requirement is meant to minimize
interoperability problems associated with servers sending Allow/
trailers in OPTIONS responses. Some servers that do not support
trailers might not be able to ignore Allow/trailers in OPTIONS
requests; therefore, a client SHOULD offer a configuration option or
other means of disabling sending Allow/trailers in OPTIONS requests.
An ICAP service sending Allow/trailers in OPTIONS response is called
a trailers-supporting service. That service designation, maintained
by the client, starts upon receiving the OPTIONS service response
carrying Allow/trailers and lasts until OPTIONS expiration or a new
OPTIONS response from that service.
A client compliant with this specification SHOULD send Allow/trailers
in each request to a trailers-supporting service. Doing so allows
the service to respond with a trailer (and is also necessary for
sending a client trailer, as detailed further below).
A client receiving both a Trailer header field and Allow/trailers in
the response MUST expect a trailer section in that response. In all
other cases, a client MUST use the usual trailer-free ICAP response
syntax. A client receiving a Trailer header field without Allow/
trailers in a response MAY treat the response as syntactically
malformed and, regardless of this response treatment, MUST NOT reuse
the connection for any other messages (including pending pipelined
requests, if any).
Rousskov Expires April 13, 2017 [Page 7]
Internet-Draft ICAP Trailers October 2016
A client MAY send a trailer in any request that satisfies all of
these conditions:
1. the request is sent to a trailers-supporting service;
2. the request has a body.
A client MUST NOT send a trailer in any other request.
To send a trailer, the client MUST send Allow/trailers and a Trailer
header field in the same request. A client MUST NOT send a
combination of those two header fields without sending a trailer.
9. Server Requirements
A server compliant with this specification SHOULD send Allow/trailers
in each successful response to an OPTIONS request carrying Allow/
trailers. Although ICAP/1.0 [RFC3507] allows a list of features in
the Allow header, some ICAP clients might not be able to handle an
Allow header other than "Allow: 204"; therefore, a server SHOULD NOT
send Allow/trailers in a response to a request without Allow/
trailers.
A server compliant with this specification MAY send Allow/trailers in
a response without a trailer to a request with Allow/trailers. The
client receiving a no-trailer REQMOD or RESPMOD response with Allow/
trailers ought to ignore Allow/trailers. Nevertheless, the server is
allowed to respond with Allow/trailers in this context because doing
so might simplify server implementation and configuration.
A server receiving both a Trailer header field and Allow/trailers in
the request MUST expect a trailer in that request. In all other
cases, a server MUST use the usual trailer-free ICAP request syntax.
A server receiving a Trailer header field without Allow/trailers in a
request MAY treat this request as syntactically malformed and,
regardless of the request treatment, MUST NOT reuse the connection
for any future requests. A previously received pipelined request is
not a "future request", even if the server has not finished
responding to it yet.
A server MAY send a trailer in any response that satisfies all of
these conditions:
1. the response is for a request containing Allow/trailers;
2. the response has a body.
A server MUST NOT send a trailer in any other response.
Rousskov Expires April 13, 2017 [Page 8]
Internet-Draft ICAP Trailers October 2016
To send a trailer, the server MUST send Allow/trailers and a Trailer
header field in the same response. A server MUST NOT send a
combination of those two header fields without sending a trailer.
10. Examples
The following examples illustrate trailer exchanges between ICAP
agents compliant with this specification. To clarify message and
message part boundaries, all CRLF sequences after major message parts
are shown as "\r\n" lines, while boundaries between ICAP requests and
responses are signified by empty lines. CRLF sequences at the end of
other lines are implied. Unimportant low-level details such as
irrelevant HTTP and ICAP header fields or Encapsulated offsets are
shown as "...".
Figure 1 shows an OPTIONS request and response when both the client
and the server are compliant with this specification. The client
also supports the ICAP 206 extension, but the server does not.
OPTIONS icap://example.net/sample-service ICAP/1.0
...
Allow: 204, trailers, 206
\r\n
ICAP/1.0 200 OK
...
Allow: 204
Allow: trailers
\r\n
Figure 1: OPTIONS handshake
The OPTIONS response in Figure 1 contains two Allow headers to
illustrate one of several possible implementations. A compliant
server can also send a single Allow header with a list of values,
just like the client does in the above example.
Figure 2 is a RESPMOD request with an ICAP trailer that the client
can send after receiving an OPTIONS response with Allow/trailers
shown in Figure 1. Note that the Trailer field value ("TBD") does
not match the actual field names in the trailer.
Rousskov Expires April 13, 2017 [Page 9]
Internet-Draft ICAP Trailers October 2016
RESPMOD icap://example.net/sample-service ICAP/1.0
...
Allow: 204, trailers
Trailer: TBD
Encapsulated: req-hdr=0, res-hdr=..., res-body=...
\r\n
GET /origin-resource HTTP/1.1
...
\r\n
HTTP/1.1 200 OK
...
Content-Length: 24
\r\n
18
Origin server sent this.
0
\r\n
X-Client-Log-Lineno: 15612570
X-Client-Status: disconnected (at 1470262108)
\r\n
Figure 2: Request Trailer
Figure 3 is a RESPMOD response with both HTTP and ICAP trailers
present. The ICAP trailer contains X-Threat-Found and Connection
fields. The ICAP trailer Connection field overwrites the Connection
field in the ICAP header. To respond with this ICAP trailer, the
server ought to receive Allow/trailers in the corresponding ICAP
request (e.g., like the request shown in in Figure 2).
Rousskov Expires April 13, 2017 [Page 10]
Internet-Draft ICAP Trailers October 2016
ICAP/1.0 200 OK
...
Connection: keep-alive
Encapsulated: res-hdr=0, res-body=...
Allow: trailers
Trailer: X-ICAP-Log-Lineno, X-Threat-Found
\r\n
HTTP/1.1 200 OK
...
Content-Length: 22
Trailer: X-Content-Checksum
\r\n
16
Origin server sent \0.
0
X-Content-Checksum: sha1-short=183caa016
\r\n
X-Threat-Found: Type=0; Resolution=1; Threat=backslash
Connection: close
\r\n
Figure 3: Two Trailers
TODO: Figure out how to keep figure descriptions and figures on the
same page without inserting empty space into HTML rendering of the
draft.
11. Security Considerations
Proper trailer support reduces old ICAP [RFC3507] security concerns
because implementations unaware of trailer complexities are arguably
more likely to misbehave when receiving HTTP or ICAP trailers.
Sending request headers with multiple Allow tokens could crash poor-
quality ICAP servers unaware of this specification. Trailer support
negotiation rules partially mitigate that risk by restricting unaware
implementations exposure; such implementations are exposed only
during OPTIONS exchanges. Since OPTIONS transaction has to precede
any HTTP message processing, and since virtually all ICAP client-
server relationships are stable, most poor-quality implementations
would be detected early and reliably.
Rousskov Expires April 13, 2017 [Page 11]
Internet-Draft ICAP Trailers October 2016
A naive implementation of trailer interpretation logic might update
an already "frozen" or "committed" (at header parsing time) state of
the ICAP transaction or connection, resulting in crashes and other
problems. For example, such an implementation could panic after
discovering a Connection trailer field value that contradicts the
Connection header field value that has already been received and
processed at the beginning of the same transaction.
12. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/
RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC3507] Elson, J. and A. Cerpa, "Internet Content Adaptation
Protocol (ICAP)", RFC 3507, DOI 10.17487/RFC3507, April
2003, <http://www.rfc-editor.org/info/rfc3507>.
[RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/
RFC5234, January 2008,
<http://www.rfc-editor.org/info/rfc5234>.
[RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer
Protocol (HTTP/1.1): Message Syntax and Routing", RFC
7230, DOI 10.17487/RFC7230, June 2014,
<http://www.rfc-editor.org/info/rfc7230>.
[Errata] Various Authors, , "RFC 3507 Errata",
<http://www.measurement-factory.com/std/icap/>.
Author's Address
Alex Rousskov
The Measurement Factory
Email: rousskov@measurement-factory.com
URI: http://www.measurement-factory.com/
Rousskov Expires April 13, 2017 [Page 12]
