Enhanced RDMA Connection Establishment
draft-ietf-storm-mpa-peer-connect-02

Abstract

Extensions to MPA are specified for RDMA Connection establishment. The first extension extends RFC5043, enabling peer-to-peer connection establishment over MPA/TCP. The second extension extends both RFC5043 and RFC5044, by providing an option for standardized exchange of RDMA-layer connection configuration.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

This Internet-Draft will expire on May 27, 2011.

Copyright Notice

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1.  Introduction
    1.1.  Summary of changes from RFC 5044
    1.2.  Summary of changes from RFC 5043
2.  Requirements Language
3.  Definitions
4.  Motivations
    4.1.  Enabling MPA Mode
    4.2.  Lack of Explicit RTR in MPA Request/Reply Exchange
    4.3.  Limitations on ULP Workaround
        4.3.1.  Transport Neutral APIs
        4.3.2.  Work/Completion Queue Accounting
        4.3.3.  Host-based Implementation of MPA Fencing
    4.4.  Standardized RDMA Parameter Negotiation
5.  MPA Connection Setup
6.  Enhanced MPA Request/Reply Frames
7.  Enhanced SCTP Session Control Chunks
8.  Enhanced RDMA Connection Establishment Data
9.  Interoperability
10.  IANA Considerations
11.  Security Considerations
12.  Acknowledgements
13.  References
    13.1.  Normative References
    13.2.  Informative References
§  Authors' Addresses

1.  Introduction

When used over TCP, the current RDDP suite of protocols relies on Marker PDU Alignment (MPA) [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.) protocol for both connection establishment and for markers for TCP layering. Currently MPA only supports a client-server model for connection establishment, forcing peer-to-peer applications to interact as though they had a client/server relationship. In addition negotiation of some of Remote Direct Memory Access Protocol (RDMAP) [RFC5040] (Recio, R., Metzler, B., Culley, P., Hilland, J., and D. Garcia, “A Remote Direct Memory Access Protocol Specification,” October 2007.) specific parameters are left to ULP negotiation. Providing an optional ULP-independent format for exchanging these parameters would be of benefit to transport neutral RDMA applications.

1.1.  Summary of changes from RFC 5044

This draft extends [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.) MPA connection setup protocol. First, it add exchange and negotiation of maximum number of RDMA Read Incoming (IRD) and Outgoing messages (ORD). Second, it adds one more Ready to Receive (RTR) frame from requestor to responder as the last message of connection establishment and adds negotiation of RTR frame message type into MPA request/response frames.

1.2.  Summary of changes from RFC 5043

This draft extends [RFC5043] (Bestler, C. and R. Stewart, “Stream Control Transmission Protocol (SCTP) Direct Data Placement (DDP) Adaptation,” October 2007.) by adding new Enhanced Session Control Chunks that extend the currently defined Chunks with the addition of IRD and ORD negotiation.

2.  Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.).

3.  Definitions

FULPDU:

Framed Upper Layer Protocol PDU. See [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.).

Completion Queue (CQ):

A consumer accessible queue where the RDMA device reports completions of Work Requests. A Consumer is able to reap completions from a CQ without requiring per transaction support from the kernel or other privileged entity. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .).

Completion Queue Entry (CQE):

Transport and device specific representation of a Work Completion. A Completion Queue holds CQEs. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .).

Inbound RDMA Read Queue Depth (IRD):

The maximum number of incoming outstanding RDMA Read Request Messages an RDMA connection can handle. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .).

IRD:

See Inbound RDMA Read Queue Depth.

ORD:

See Outbound RDMA Read Queue Depth.

Outbound RDMA Read Queue Depth (ORD):

The maximum number of outstanding RDMA Read Requests that can be issued for the RDMA connection. This should be less than or equal to the peer's IRD. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .).

Queue Pair (QP):

The traditional name for a local Endpoint in a [VIA] (Compaq, Intel, Microsoft, “Virtual Interface Architecture Specification,” 1997.) derived local interface. A Queue Pair is the set of Work Queues associated exclusively with a single Endpoint. The Send Queue (SQ), Receive Queue (RQ) and Inbound RDMA Read Queue (IRQ) are considered to be part of the Queue Pair. The potentially shared Completion Queue (CQ) and Shared Receive Queue (SRQ) are not. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .).

Shared Receive Queue(SRQ):

A shared pool of Receive Work Requests posted by the Consumer that can be allocated by multiple RDMA endpoints (Queue Pair). See [DAPL] (, “Direct Access Programming Library,” .).

Work Queue:

An element of a [VIA] (Compaq, Intel, Microsoft, “Virtual Interface Architecture Specification,” 1997.) derived local interface that allows user-space applications to submit Work Requests directly to network hardware. Specific Work Queues include the Send Queue (SQ) for transmit requests, Receive Queue (RQ) for receive requests specific to a single Endpoint and Shared Receive Queues (SRQs) for receive requests that can be allocated by one or more Endpoints. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .).

Work Queue Element (WQE):

Transport and device specific representation of a Work Request. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .)

Work Request:

An elementary object used by Consumers to enqueue a requested operation (WQEs) onto a Work Queue. See [RDMAC] (, “RDMA Protocol Verbs Specification (Version 1.0),” .).

4.  Motivations

The goal of this draft is twofold. One is to extend support from the current client-server model for RDMA connection setup to a peer-to-peer model. The second is to add negotiation of RDMA Read queue size for both sides of an RDMA connection.

4.1.  Enabling MPA Mode

MPA defines encoding of DDP Segments in FULPDUs (Framed Upper Layer Protocol PDUs). Generation of FULPDUs requires the ability to periodically insert MPA Markers and to generate the MPA CRC-32c for each frame. Reception may require parsing/removing the markers after using them to identify MPA Frame boundaries, and validation of the MPA-CRC32c.

A major design objective for MPA was to ensure that the resulting TCP stream would be a fully compliant TCP stream for any and all TCP-aware middle-boxes. The challenge is that while only some TCP payload streams are a valid stream of MPA FULPDUs, any sequence of bytes is a valid TCP payload stream. The determination that a given stream is in a specific MPA mode cannot be made at the MPA or TCP layer. Therefore enabling of MPA mode is handled by the ULP.

The MPA protocol can be viewed as having two parts.

• a specification of generation and reception of MPA FULPDUs. This is unchanged by Enhanced RDMA Connection Establishment.
• a pre-MPA exchange of messages to enable a specific MPA mode for the TCP connection. Enhanced RDMA Connection Establishment extends this protocol with two new features.

In typical implementations, generation and reception of MPA FULPDUs is handled by hardware. The exchange of the MPA Request and Reply frames is then handled by host software. As will be explained, this implementation split prevents applications from working around the client-server assumptions in the current MPA Request/Reply exchange.

4.2.  Lack of Explicit RTR in MPA Request/Reply Exchange

The exchange of MPA Request and Reply messages to place a TCP connection in MPA mode is specified in [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.). This protocol provides many benefits to the design of MPA FULPDU hardware:

• The ULP is responsible for specifying the exact MPA Mode (Markers enabled or disabled, CRC-32c enabled or suppressed) and the point in the TCP streams (inbound and outbound) where MPA frames will begin.
• Before the first MPA frame is transmitted, all pre-MPA mode TCP payload will have been acknowledged by the peer. Therefore it is never necessary to generate a retransmission that mixes pre-MPA and MPA payload.
• Before MPA reception is enabled, all incoming pre-MPA mode TCP payload will have been acknowledged. Therefore the host will never receive a TCP segment that mixes pre-MPA and MPA payload.

The limitation of the current MPA Request/Reply exchange is that it does not define a Ready to Receive (RTR) message that the active side would send, so that the passive side can know that the last non-MPA payload (the MPA Reply) had been received.

Instead, the role of an RTR message is piggy-backed on the first MPA FULPDU sent by the active side. This is actually a valuable optimization for all applications that fit the classic client/server model. The client only initiates the connection when it has a request to send to the server, and the server has nothing to send until it has received and processed the client request.

Even applications where the server sends some configuration data immediately can easily send the same information as application private data in the MPA Reply. So the currently defined exchange works for almost all applications.

Many peer-to-peer applications, especially those involving cluster calculations (frequently using MPI [UsingMPI] (MIT Press, “Using MPI-2: Advanced Features of the Message Passing Interface,” 1999.), or [RDS] (Open Fabrics Association, “Reliable Datagram Socket,” 2008.)), have no natural client or server roles ([PPMPI] (Morgan Kaufmann Publishers Inc., “Parallel Programming with MPI,” 2008.), [OpenMP] (McGraw-Hill, “Parallel Programming in C with MPI and OpenMP,” 2003.)). Typically one member of the cluster is arbitrarily selected to initiate the connection when the distributed task is launched, while the other accepts it. At startup time, however, there is no way to predict which node will have the first message to actually send. Establishing the connections immediately, however, is valuable because it reduces latency once results are ready to transmit and it validates connectivity throughout the cluster.

The lack of an explicit RTR message in the MPA Request/Reply exchange forces all applications to have a first message from the connection initiator, whether this matches the application communication model or not.

4.3.  Limitations on ULP Workaround

The requirement that the RDMA connection initiator sends the first message does not appear to be that onerous on first examination. The natural question is why the application layer would not simply generate a "nop" message when there was no other message to submit.

There are three factors that make this workaround unsuitable for many peer-to-peer applications.

• Transport Neutral APIs.
• Work/Completion Queue Accounting.
• Host-based implementation of MPA Fencing.

4.3.1.  Transport Neutral APIs

Many of these applications access RDMA services using a transport neutral API such as [DAPL] (, “Direct Access Programming Library,” .) or [OFA] (, “OFA verbs & APIs,” .). Only MPA has a first message requirement. Other RDMA transports, including SCTP and InfiniBand, do not.

Application or middleware communications can be expressed as transport neutral RDMA operations, allowing lower software layers to translate to transport and device specifics. Having a distinct extra message that is required only for one transport undermines the application's goal of being transport neutral.

4.3.2.  Work/Completion Queue Accounting

RDMA local APIs conventionally use work queues to submit requests (work queue elements or WQEs) and to asynchronously receive completions (in completion queues or CQs).

Each work request can generate a completion queue entries (CQE). Completions for successful transmit work requests are frequently suppressed, but the completion queue capacity must account for the possibility that each will complete in error. A completion queue can receive completions from multiple work queues.

Completion Queues are defined so as to allow hardware RDMA implementations to generate CQEs directly to a user-space mapped buffer. This enables a user-space RDMA consumer to reap completions without requiring kernel intervention.

A hardware RDMA implementation cannot reasonably wait for an available slot in the completion queue. The queue must be sized such that an overflow will not occur. When an overflow does occur it is considered catastrophic and will typically require tearing down all RDMA connections using that CQ.

This style of interface is very efficient, but places a burden on the application to properly size each Completion Queue to match the Work Queues that feed it.

While the format of both WQEs and CQEs is transport and device dependent, a transport neutral API can deal with WQEs and CQEs as abstract transport and device neutral objects. Therefore the number of WQEs and CQEs required for an application can be transport and device neutral.

The capacity of the work queues and completion queues can be calculated in an abstract transport/device neutral fashion. Lower layers of the protocol stack cannot disrupt these calculations by inserting a dummy "nop" Work Request and filtering out the matching completion. The lower layer does not know the usage model on which the queue sizes are built, nor does it know how frequently an insertion will be required.

4.3.3.  Host-based Implementation of MPA Fencing

Many hardware implementations of iWARP using MPA/TCP do not handle the MPA Request/Reply exchange in hardware, rather they are handled by the host processor in software. With such designs it is common for the MPA Fencing to be implemented in the user-space device-specific library (commonly referred to as a 'User Verbs' library or module).

When the generation and reception of MPA FULPDUs is already dedicated to hardware, a Work Completion can only be generated by an untagged message.

4.4.  Standardized RDMA Parameter Negotiation

Most RDMA applications are developed using a transport neutral API to access RDMA services based on a "queue pair" paradigm as originally defined by the Virtual Interface Architecture [VIA] (Compaq, Intel, Microsoft, “Virtual Interface Architecture Specification,” 1997.), refined by the Direct Access Programming Library [DAPL] (, “Direct Access Programming Library,” .) and most commonly deployed with the OpenFabrics API [OFA] (, “OFA verbs & APIs,” .).

These transport neutral APIs seek to provide a common set of RDMA services whether the underlying transport is, for example, iWARP over MPA, iWARP over SCTP or InfiniBand.

The common model for establishing an RDMA connection has the following steps:

• The passive side ULP listens for connection requests.
• The active side ULP submits a connection request using an RDMA endpoint ("queue pair"), the desired destination and the parameters to be used for the connection. Those parameters include both RDMA layer characteristics, such as the RDMA Read credits to be allowed and application specific data (typically referred to as "private data").
• The passive side ULP receives the Connection Request, which includes the identity of the active side and the requested connection characteristics. The passive side ULP uses this information to decide whether to accept the connection, and if it is to be accepted, how to create and/or configure the RDMA endpoint.
• If accepting, the passive side ULP submits its acceptance of the Connection Request. This local accept operation includes the RDMA endpoint to be used and the connection characteristics (both the RDMA configuration and any application specific private data to be returned).
• The active side receives confirmation that the connection has been accepted, what the configured connection characteristics are, and any application supplied private data.

As currently defined, DDP connection establishment requires the ULP to encode the RDMA configuration in the application specific private data. This results undesirable duplication of logic to cover both InfiniBand and iWARP, and to specify the extraction of the RDMA characteristics from the ULP for each specific Upper Layer Protocol.

A standard definition of the RDMA characteristics within the private data section would enable common connection establishment APIs to format the RDMA characteristics based on the same API information used when establishing an InfiniBand connection. The application would then only have to indicate that it was using this standard format to enable common connection establishment procedures to apply common code to properly parse these fields and configure the RDMA endpoints accordingly.

5.  MPA Connection Setup

Below we provide overview of Enhanced Connection Setup. The goal is to allow standard negotiation of ORD/IRD setting on both sides of the RDMA connection and/or to negotiate the initial data transfer operation by an initiator when the existing 'client sends first' rule does not match application requirements.

The RDMA connection initiator sends an MPA Request, as specified in [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.); the new format defined here allows for:

• Standardized negotiation of ORD and IRD.
• Negotiation of an RTR message.

The RDMA connection responder processes the MPA Request and generates an MPA Reply, as specified in [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.); the new format completes the negotiation.

The local interface SHOULD require the ULP to explicitly configure on a per-application or per-connection basis when an explicit RTR message will be required. Piggy-backing the RTR on a Client's first message is a valuable optimization for most connections.

The RDMA connection initiator MUST NOT allow any later FULPDUs to be transmitted before the RTR message. One method to achieve that is to delay notifying the ULP that the RDMA connection has been established until after any required RTR Message has been transmitted.

All MPA exchanges are performed via TCP prior to RDMA establishment, and are therefore signaled via TCP and not via RDMA completion.

6.  Enhanced MPA Request/Reply Frames

Enhanced RDMA Connection Establishment uses an alternate format for MPA Requests and Replies, as follows:

    0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 0  |                                                               |
    +         Key (16 bytes containing "MPA ID Req Frame")          +
 4  |      (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65)        |
    +         Or  (16 bytes containing "MPA ID Rep Frame")          +
 8  |      (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65)        |
    +                                                               +
 12 |                                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 16 |M|C|R|S| Res   |     Rev       |          PD_Length            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                                                               |
    ~                                                               ~
    ~                   Private Data                                ~
    |                                                               |
    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Key:

Unchanged from [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.).

M:

Unchanged from [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.).

C:

Unchanged from [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.).

R:

Unchanged from [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.).

S:

One if the Private Data begins with the Enhanced RDMA Connection Establishment Data. Zero otherwise.

Res:

One bit smaller than in [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.), otherwise unchanged.

Rev:

This field contains the revision of MPA. To use any Enhanced Connection Establishment feature this MUST be set to two, If no Enhanced Connection Establishment features are desired it MAY be set to one. A host accepting MPA connections SHOULD continue to accept MPA Requests with version one even if it supports version two.

PD_Length:

Unchanged from [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.). This is the total length of the Private Data field, including the Enhanced RDMA Connection Establishment Data if present.

Private Data:

Unchanged from [RFC5044] (Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, “Marker PDU Aligned Framing for TCP Specification,” October 2007.). However, if the 'S' flag is set, Private Data begins with Enhanced RDMA Connection Establishment Data.

7.  Enhanced SCTP Session Control Chunks

The type of the SCTP Session Control Chunk is defined by a Function Code. [RFC5043] (Bestler, C. and R. Stewart, “Stream Control Transmission Protocol (SCTP) Direct Data Placement (DDP) Adaptation,” October 2007.) already defines codes for 'DDP Stream Session Initiate' and 'DDP Stream Session Accept', which are equivalent to a MPA Request Frame and an accepting MPA Reply Frame.

Enhanced RDMA Connection Establishment requires three additional Function codes, as follows:

Enhanced DDP Stream Session Initiate:

0x05

Enhanced DDP Stream Session Accept:

0x06

Enhanced DDP Stream Session Reject:

0x07

The Enhanced Reject function code SHOULD be used to indicate a configuration that would have been accepted.

It should be noted that [RFC5043] (Bestler, C. and R. Stewart, “Stream Control Transmission Protocol (SCTP) Direct Data Placement (DDP) Adaptation,” October 2007.) already supports either side sending the first DDP Message. The Payload Protocol Identifier (PPID) already distinguishes between Session Establishment and DDP Segments.

8.  Enhanced RDMA Connection Establishment Data

    0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 0  |A|B|        IRD                |C|D|        ORD                |
 4  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

IRD:

In request: the Initiator requested responder IRD for the connection. In reply: the depth the Responder will support. An all ones value (0x3FFF) indicates that automatic negotiation of the IRD is not desired, and that the ULP will be responsible for doing this configuration.

ORD:

In request: the Initiator initial ORD setting for the connection. In reply: the depth the Responder will support. An all ones value (0x3FFF) indicates that automatic negotiation of the IRD is not desired, and that the ULP will be responsible for doing this configuration.

A:

Control Flag for using a zero length ULPDU as the RTR message.

B:

Control Flag for using a zero length RDMA Write as the RTR message.

C:

Control Flag for using a zero length RDMA Read as the RTR message.

D:

Reserved. Must be sent as zero and ignored when received.

In the MPA Request, the Initiator SHOULD set the A, B and C Control Flags respectively to TRUE for each of the options it supports.

In the MPA Reply, the Control Flag is set for the set of options that the passive side will accept as an RTR message. This response MUST include all options that the responder will support without requiring a connection specific enabling action. For example, if the responder will always unblock an MPA connection when it receives a zero length MPA Write, it should indicate so without regard to what was in the MPA Request. Options which require connection specific enabling actions SHOULD NOT be set unless the corresponding flag was set in the MPA Request. The respondent MAY choose to limit the number of modes that it enables.

If there is no Standard RDMAP Configuration Data in the MPA Reply Frame, and the Enhanced Connection Establishment version number is used, it is the equivalent of setting 'A', 'B' and 'C'.

Setting no Control Flags in the MPA Reply indicates that an RDMA Send message will be required. As this option will require the initiator ULP to be involved it SHOULD NOT be used unless necessary.

9.  Interoperability

An initiator SHOULD NOT use the Enhanced DDP Connection Establishment formats or function codes when no enhanced functionality is desired.

A responder SHOULD continue to accept the unenhanced connection requests.

10.  IANA Considerations

This document has no IANA considerations.

11.  Security Considerations

The security considerations from RFC 5044 apply and the changes in this document do not introduce new security considerations.

12.  Acknowledgements

The authors wish to thank Sean Hefty, Dave Minturn, Tom Talpey and David Black for their valuable contributions and review of this document.

13.  References

13.1. Normative References

13.2. Informative References

Authors' Addresses