Internet DRAFT - draft-wood-storm-rdmap-ext-v2
draft-wood-storm-rdmap-ext-v2
Storage Maintenance (storm) Working Group Donald Wood
Internet Draft Robert Sharp
Intended status: Standards Track Kenneth Keels
Expires: May 2014 Intel Corporation
November 26, 2013
RDMA Protocol Extensions V2
draft-wood-storm-rdmap-ext-v2-00.txt
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with
the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on May 26, 2014.
Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
Wood et al. Expires May 26, 2014 [Page 1]
Internet-Draft RDMA Protocol Extensions November 2013
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Abstract
This document specifies extensions to the IETF Remote Direct Memory
Access Protocol and RDMA Protocol Extensions. RDMAP provides read
and write services directly to applications and enables data to be
transferred directly into Upper Layer Protocol (ULP) Buffers without
intermediate data copies. The extensions specified in this document
provide the following capabilities and/or improvements to eliminate
application visible differences with other RDMA technologies: Send
with Immediate Data Operations and a new form of RDMA Read
Operation.
Table of Contents
1. Introduction...................................................3
1.1. RDMAP Extensions Implementation Requirements..............3
2. Requirements Language..........................................4
3. Glossary.......................................................4
4. Header Format Extensions.......................................5
4.1. RDMAP Control and Invalidate STag Fields..................5
4.2. RDMA Message Definitions..................................9
5. Send with Immediate Data Operations...........................10
5.1. RDMAP Interactions with ULP for Immediate Data...........10
5.2. Ordering and Completions.................................11
5.3. Send with Immediate Data Operations......................11
6. RDMA Read V2..................................................11
6.1. RDMA Read V2 Request Header Format.......................12
6.2. RDMA Read V2 Response Header Format......................12
6.3. Ordering and Completions.................................12
7. Ordering and Completions Table................................13
8. Error Processing..............................................15
8.1. Errors Detected at the Local Peer........................15
8.2. Errors Detected at the Remote Peer.......................15
9. Security Considerations.......................................16
10. IANA Considerations..........................................16
11. References...................................................16
Wood et al. Expires May 26, 2014 [Page 2]
Internet-Draft RDMA Protocol Extensions November 2013
11.1. Normative References....................................16
11.2. Informative References..................................17
12. Acknowledgments..............................................17
Appendix A. DDP Segment Formats for RDMA Messages................18
A.1. DDP Segment for RDMA Read V2 Request.....................18
A.2. DDP Segment for RDMA Read V2 Response....................18
A.3. DDP Segment for Send with Immediate Data and Send with
Solicited Event and Immediate Data............................19
A.4. DDP Segment for Send with Invalidate and Immediate Data and
Send with Invalidate and Solicited Event and Immediate Data...21
1. Introduction
The RDMA Protocol [RFC5040] and RDMA Protocol Extensions[RFCYYYY]
provides capabilities for zero copy data communications that
preserve memory protection semantics, enabling more efficient
network protocol implementations. This document specifies the
following extensions to the RDMA Protocol (RDMAP):
o Send with Immediate Data operations allow the ULP at the sender
to provide a small amount of immediate data with an RDMA Message.
o RDMA Read with untagged RDMA Read Response. Support for RDMA Read
requests and responses without exposing the data sink buffer
address(es) on the wire.
Other RDMA transport protocols define the functionality added by
these extensions leading to differences in RDMA applications and/or
Upper Layer Protocols. Removing these differences in the transport
protocols simplifies these applications and ULPs and is the main
motivation for the extensions specified in this document.
The changes in this RFC require the RDMAP version to move to version
2.
1.1. RDMAP Extensions Implementation Requirements
The discovery and negotiation of the RDMA Extensions and the RDMAP
Extensions is covered in MPA Extensions [RFCZZZZ]. The following
paragraphs describe the implementation that advertises RDMAP
Extensions and negotiates their use.
When the RDMAP version is negotiated to 2, all features described in
RDMA Extensions [RFCYYYY] and RDMA Extensions V2 [RFCXXXX] are
Wood et al. Expires May 26, 2014 [Page 3]
Internet-Draft RDMA Protocol Extensions November 2013
supported. This requirement allows applications to count on all the
features being available. These include:
* Atomic Operations
* Immediate Data and Immediate Data with SE Operations
* Send with Immediate Data, Send with Solicited Event and
Immediate Data, Send with Invalidate and Immediate Data, and Send
with Invalidate and Solicited Event and Immediate Data Operations.
* RDMA Read V2 Request and RDMA Read V2 Response Operations.
2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119 [RFC2119].
3. Glossary
This document is an extension of [RFC5040] and key words are defined
in the glossary of the referenced document.
Immediate Data - a small fixed size portion of data sent from the
Data Source to a Data Sink
Requester - the sender of an RDMA Operation request.
Responder - the receiver of an RDMA Operation request.
Wood et al. Expires May 26, 2014 [Page 4]
Internet-Draft RDMA Protocol Extensions November 2013
4. Header Format Extensions
The control information of RDMA Messages is included in DDP protocol
[RFC5041] defined header fields, with the following new formats:
. Four new RDMA Messages carry additional RDMAP headers for Send
with Immediate Data variants. The Send with Immediate Data, Send
with Solicited Event and Immediate Data, Send with Invalidate and
Immediate Data, and Send with Invalidate and Solicited Event and
Immediate Data messages include 8 bytes of data following the
RDMAP header.
. Two new RDMA Messages carry additional RDMAP headers for the new
RDMA Read Request and RDMA Read Response messages. The new RDMA
Read Request message includes a revised definition following the
untagged RDMAP header that removes the Data Sink STag and Tagged
Offset fields. The new RDMA Read Response is an untagged message.
4.1. RDMAP Control and Invalidate STag Fields
Figure 1 depicts the format of the DDP Control and RDMAP Control
fields, in the style and convention of [RFC5040]. The DDP Control
Field consists of the T,L, Resrv and DV fields [RFC5041]. The RDMAP
Control Field consists of the RV, Rsv and Opcode fields [RFC5040].
When the RDMAP version has been negotiated to 2, the RDMAP RV field
MUST be set to 2 for all Operations.
The RDMA Messages defined by this specification use all 8 bits of
the RDMAP Control Field. The first octet reserved for ULP use in the
DDP Protocol MUST be used by the RDMAP to carry the RDMAP Control
Field.
The ordering of the bits in the first octet MUST be as shown in
Figure 1.
Wood et al. Expires May 26, 2014 [Page 5]
Internet-Draft RDMA Protocol Extensions November 2013
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|T|L| Resrv | DV| RV|R| Opcode |
| | | | | |s| |
| | | | | |v| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Invalidate STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1 DDP Control and RDMAP Control Fields
Figure 2 defines the values of RDMA Opcode field that MUST be used
for the RDMA Messages defined in this specification. The Opcode
field has been extended one additional bit to accommodate the
additional opcodes.
In RFC 5040, bit 7 was reserved and required to be zero. The
expansion of the opcode field size is compatible with RFC 5040 since
only the new opcodes will set this bit to one.
Figure 2 also defines when the STag, Tagged Offset, and Queue Number
fields MUST be provided for the RDMA Messages defined in this
specification.
All RDMA Messages defined in this specification MUST have:
The RDMA Version (RV) field: 10b.
Opcode field: See Figure 2.
To support RV field set to 10b, an RNIC MUST:
* Implement [RFCYYYY] RDMA Protocol Extensions (Atomic, Immediate
Data and Immediate Data with SE Operations).
* Implement Send with Immediate Data, Send with Solicited Event
and Immediate Data, Send with Invalidate and Immediate Data, and
Send with Invalidate and Solicited Event and Immediate Data
operations.
* Implement RDMA Read V2 Request and RDMA Read V2 Response
messages.
Wood et al. Expires May 26, 2014 [Page 6]
Internet-Draft RDMA Protocol Extensions November 2013
For Send with Invalidate and Immediate Data, and Send with
Invalidate and Solicited Event and Immediate Data, the second
through fifth octets of the DDP RsvdULP field MUST be used by RDMAP
to carry the Invalidate STag.
Wood et al. Expires May 26, 2014 [Page 7]
Internet-Draft RDMA Protocol Extensions November 2013
-------+-----------+-------+------+-------+-----------+--------------
RDMA | Message | Tagged| STag | Queue | Invalidate| Message
Opcode | Type | Flag | and | Number| STag | Length
| | | TO | | | Communicated
| | | | | | between DDP
| | | | | | and RDMAP
-------+-----------+-------+------+-------+-----------+--------------
-------+-----------+-------------------------------------------------
10001b | RDMA Read | 0 | N/A | 1 | N/A | Yes
| Request V2| | | | |
-------+-----------+-------------------------------------------------
10010b | RDMA Read | 0 | N/A | 3 | N/A | Yes
| Response | | | | |
| V2 | | | | |
-------+-----------+-------------------------------------------------
10011b | Send with | 0 | N/A | 0 | N/A | Yes
| Immediate | | | | |
| Data | | | | |
-------+-----------+-------------------------------------------------
10100b | Send with | 0 | N/A | 0 | Yes | Yes
| Invalidate| | | | |
| and | | | | |
| Immediate | | | | |
| Data | | | | |
-------+-----------+-------------------------------------------------
10101b | Send with | 0 | N/A | 0 | N/A | Yes
| SE and | | | | |
| Immediate | | | | |
| Data | | | | |
-------+-----------+-------------------------------------------------
10110b | Send with | 0 | N/A | 0 | Yes | Yes
| Invalidate| | | | |
| and SE and| | | | |
| Immediate | | | | |
| Data | | | | |
-------+-----------+-------------------------------------------------
Figure 2 Additional RDMA Usage of DDP Fields
Note: N/A means Not Applicable.
All other DDP and RDMAP control fields MUST be set as described in
[RFC5040] except for the RV field which MUST be 2.
Wood et al. Expires May 26, 2014 [Page 8]
Internet-Draft RDMA Protocol Extensions November 2013
4.2. RDMA Message Definitions
The following figure defines which RDMA Headers MUST be used on each
new RDMA Message and which new RDMA Messages are allowed to carry
ULP payload:
-------+-----------+-------------------+-------------------------
RDMA | Message | RDMA Header Used | ULP Message allowed in
Message| Type | | the RDMA Message
OpCode | | |
| | |
-------+-----------+-------------------+-------------------------
-------+-----------+-------------------+-------------------------
10001b | RDMA Read | RDMA Read Request | No
| Request V2| Header V2 |
-------+-----------+-------------------+-------------------------
10010b | RDMA Read | None | No
| Response | |
| V2 | |
-------+-----------+-------------------+-------------------------
10011b | Send with | Immediate Data | Yes
| Immediate | header |
| Data | in the last ULPDU |
-------+-----------+-------------------------------------------------
10100b | Send with | Immediate Data | Yes
| Invalidate| header |
| and | in the last ULPDU |
| Immediate | |
| Data | |
-------+-----------+-------------------------------------------------
10101b | Send with | Immediate Data | Yes
| SE and | header |
| Immediate | in the last ULPDU |
| Data | |
-------+-----------+-------------------------------------------------
10110b | Send with | Immediate Data | Yes
| Invalidate| header |
| and SE and| in the last ULPDU |
| Immediate | |
| Data | |
-------+-----------+-------------------------------------------------
Figure 3 RDMA Message Definitions
Wood et al. Expires May 26, 2014 [Page 9]
Internet-Draft RDMA Protocol Extensions November 2013
This extension uses RDMAP Queue Number 3 for Untagged Buffers for
RDMA Read V2 Responses. This queue is used for tracking outstanding
RDMA Read V2 Requests.
5. Send with Immediate Data Operations
The Send with Immediate Data operations are used to improve ULP
processing efficiency by allowing 8 bytes of immediate data to be
delivered to the ULP at the Remote Peer.
5.1. RDMAP Interactions with ULP for Immediate Data
For Send with Immediate Data operations, the following are the
interactions between the RDMAP Layer and the ULP:
. At the Data Source:
. The ULP passes to the RDMAP Layer the following:
. Eight bytes of ULP Immediate Data
. When the Immediate Data operation Completes, an indication
of the Completion results.
. At the Data Sink:
. When any of the Send with Immediate Data variants Completes
successfully, the RDMAP Layer passes the following
information to the ULP Layer:
. Eight bytes of Immediate Data
. An Event, if the Data Sink is configured to generate an
Event.
. When any of the Send with Immediate Data variants Completes
in error, the Data Sink RDMAP Layer will pass up the
corresponding error information to the Data Sink ULP and
send a Terminate Message to the Data Source RDMAP Layer. The
Data Source RDMAP Layer will then pass up the Terminate
Message to the ULP.
Wood et al. Expires May 26, 2014 [Page 10]
Internet-Draft RDMA Protocol Extensions November 2013
5.2. Ordering and Completions
Ordering and completion rules for Immediate Data are the same as
those for a Send operation as described in section 5.5 of RFC 5040.
5.3. Send with Immediate Data Operations
The Send with Immediate Data, Send with Invalidate and Immediate
Data, Send with Solicited Event and Immediate Data, and Send with
Solicited Event and Invalidate and Immediate Data messages follow
the same model as described in section 5.3 of RFC 5040 with the
following differences:
. The opcodes for the Send with Immediate Data variants have
different values from the RDMA Send (without Immediate Data
Messages).
. The L bit MUST be set to 1 only in the ULPDU that contains
immediate data.
. The immediate data MUST be the only data allowed in the ULPDU
with the L bit set to 1.
. At the data sink, if the operation is completed successfully,
the RDMAP Layer MUST pass the eight bytes of immediate data to
the ULP Layer.
When any Send with Immediate Data variant has a message length of
zero, the only ULDPU MUST have the immediate data and the L bit MUST
be set to 1.
6. RDMA Read V2
The definition of RDMA Read in RFC 5040 requires exposure of the
Data Sink STag and Data Sink Tagged Offset. Applications have to be
aware of this exposure since it is different from other RDMA
technologies. Dealing with the exposure in a secure manner causes
application overhead in the fast path operations specific to RDMAP.
Two new RDMA messages are needed to remove the exposure of the Data
Sink buffer information. The new RDMA Read message is composed of an
RDMA Read V2 Request that removes the SinkSTag and SinkTO
parameters. The RDMA Read V2 Response is a simple untagged DDP
message targeting queue number 3.
RDMA Read V2 Response Messages MUST use the Untagged Buffer model
with QN=3. Queue number 3 MUST be used to track outstanding RDMA
Read V2 Request messages at the Requester. When the RDMA Read V2
Wood et al. Expires May 26, 2014 [Page 11]
Internet-Draft RDMA Protocol Extensions November 2013
Response message is received, the MSN MUST be used to locate the
corresponding RDMA Read V2 request in order to place the data and
Complete the Operation.
6.1. RDMA Read V2 Request Header Format
The RDMA Read V2 Request Message carries an RDMA Read V2 Request
Header that describes the Data Source Buffers used by the RDMA Read
operation. The RDMA Read Request Header immediately follows the DDP
header. The RDMAP layer passes to the DDP layer an RDMAP Control
Field. The following figure depicts the RDMA Read V2 Request Header
that MUST be used for all RDMA Read V2 Request Messages:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RDMA Read Message Size (RDMARDSZ) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Source STag (SrcSTag) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Source Tagged Offset (SrcTO) |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4 RDMA Read V2 Message Header
The RDMA Read Message Size, Data Source STag, and Data Source Tagged
Offset parameters are the same as defined in RFC 5040 section 4.4.
6.2. RDMA Read V2 Response Header Format
The RDMA Read V2 Response Message does not include an RDMAP header.
The RDMAP layer passes to the DDP layer an RDMAP Control Field. The
RDMA Read V2 Response Message is fully described by the DDP Headers
of the DDP Segments associated with the Message.
6.3. Ordering and Completions
Ordering and completion rules for RDMA Read V2 Request are the same
as those for a Send operation as described in section 5.2.1 of RFC
5040 with one exception.
Wood et al. Expires May 26, 2014 [Page 12]
Internet-Draft RDMA Protocol Extensions November 2013
. The Local Peer is no longer required to use multiple RDMA Read
Request Messages if the Local Peer wishes to reference multiple
data sink buffers. This is because RDMA Read V2 Response is
untagged.
Ordering and completion rules for the RDMA Read V2 Response are the
same as the Send operation described in section 5.3 of RFC 5040 for
a Remote Peer with one exception.
. The queue number field for the DDP layer MUST be 3.
7. Ordering and Completions Table
For reference, the following table is copied from [RFCYYYY] and pared
down to show only Send and Read Operations. It summarizes the ordering
relationships for from the standpoint of Local Peer issuing the
Operations. Note that in the table that follows, Send includes all
variants: Send, Send with Invalidate, Send with Solicited Event, and
Send with Solicited Event and Invalidate with and without immediate
data. Finally note that RDMA Read includes the original RDMA Read and
the newly defined RDMA Read V2.
----------+------------+-------------+-------------+-------------------
First | Second | Placement | Placement | Ordering
Operation | Operation | Guarantee at| Guarantee at| Guarantee at
| | Remote Peer | Local Peer | Remote Peer
----------+------------+-------------+-------------+-------------------
Immediate | Send | No Placement| Not | Completed in
Data | | Guarantee | Applicable | Order
| | between Send| |
| | Payload and | |
| | Immediate | |
| | Data | |
----------+------------+-------------+-------------+-------------------
Immediate | RDMA | No Placement| RDMA Read | RDMA Read
Data | Read | Guarantee | Response | Response
| | between | will not be | Message will
| | Immediate | Placed until| not be
| | Data and | Immediate | generated
| | RDMA Read | Data is | until
| | Request | Placed at | Immediate Data
Wood et al. Expires May 26, 2014 [Page 13]
Internet-Draft RDMA Protocol Extensions November 2013
| | | Remote Peer | has been
| | | | Completed
----------+------------+-------------+-------------+-------------------
Immediate | Immediate | No Placement| Not | Completed in
Data or | Data | Guarantee | Applicable | Order
Send | | | |
----------+------------+-------------+-------------+-------------------
RDMA Read | Immediate | No Placement| Immediate | Not Applicable
| Data | Guarantee | Data MAY be |
| | between | Placed |
| | Immediate | before |
| | Data and | RDMA Read |
| | RDMA Read | Response is |
| | Request | generated |
----------+------------+-------------+-------------+-------------------
Atomic | Send | No Placement| Send Payload| Not Applicable
| | Guarantee | MAY be |
| | between Send| Placed |
| | Payload and | before |
| | Atomic | Atomic |
| | Request | Response is |
| | | generated |
----------+------------+-------------+-------------+-------------------
Atomic | RDMA | No Placement| No Placement| RDMA Read
| Read | Guarantee | Guarantee | Response
| | between | between | Message will
| | Atomic | Atomic | not be
| | Request and | Response | generated
| | RDMA Read | and RDMA | until Atomic
| | Request | Read | Response Message
| | | Response | has been
| | | | generated
----------+------------+-------------+-------------+-------------------
Send | Atomic | No Placement| Atomic | Atomic Response
| | Guarantee | Response | Message will not
| | between Send| will not be | be generated until
| | Payload and | Placed at | Send has been
| | Atomic | the Local | Completed
| | Request | Peer Until |
| | | Send Payload|
| | | is Placed |
| | | at the |
| | | Remote Peer |
----------+------------+-------------+-------------+-------------------
RDMA | Atomic | No Placement| No Placement| Atomic Response
Wood et al. Expires May 26, 2014 [Page 14]
Internet-Draft RDMA Protocol Extensions November 2013
Read | | Guarantee | Guarantee | Message will
| | between | between | not be generated
| | Atomic | Atomic | until RDMA
| | Request and | Response | Read Response
| | RDMA Read | and RDMA | has been
| | Request | Read | generated
| | | Response |
----------+------------+-------------+-------------+-------------------
8. Error Processing
In addition to error processing described in section 7 of [RFC5040],
the following rules apply for the new RDMA Messages defined in this
specification.
8.1. Errors Detected at the Local Peer
The Local Peer MUST send a Terminate Message for each of the
following cases:
1. For errors detected while creating a RDMA Read V2 Request, RDMA
Read V2 Response, or Immediate Data with SE Message, or other
reasons not directly associated with an incoming Message, the
Terminate Message and Error code are sent instead of the Message.
In this case, the Error Type and Error Code fields are included
in the Terminate Message, but the Terminated DDP Header and
Terminated RDMA Header fields are set to zero.
2. For errors detected on an incoming RDMA Read V2 Request, RDMA
Read V2 Response, or Immediate Data with Solicited Event (after
the Message has been Delivered by DDP), the Terminate Message is
sent at the earliest possible opportunity, preferably in the next
outgoing RDMA Message. In this case, the Error Type, Error Code,
and Terminated DDP Header fields are included in the Terminate
Message, but the Terminated RDMA Header field is set to zero.
8.2. Errors Detected at the Remote Peer
Wood et al. Expires May 26, 2014 [Page 15]
Internet-Draft RDMA Protocol Extensions November 2013
9. Security Considerations
This document specifies extensions to the RDMA Protocol
specification in [RFC5040], and as such the Security Considerations
discussed in Section 8 of [RFC5040] apply.
10. IANA Considerations
IANA is requested to add the following entries to the "RDMAP Message
Operation Codes" registry of "RDDP Registries":
0x11, RDMA Read V2 Request, [RFCXXXX]
0x12, RDMA Read V2 Response, [RFCXXXX]
0x13, Send with Immediate Data, [RFCXXXX]
0x14, Send with Invalidate and Immediate Data, [RFCXXXX]
0x15, Send with Solicited Event and Immediate Data, [RFCXXXX]
0x16, Send with Invalidate and Solicited Event and Immediate Data,
[RFCXXXX]
RFC Editor: Please replace XXXX in all instances of [RFCXXXX] above
with the RFC number of this document and remove this note.
RFC Editor: Please replace YYYY in all instances of [RFCYYYY] above
with the RFC number of the RDMA Extensions document and remove this
note.
RFC Editor: Please replace ZZZZ in all instances of [RFCZZZZ] above
with the RFC number of the MPA Extensions document and remove this
note.
11. References
11.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
Wood et al. Expires May 26, 2014 [Page 16]
Internet-Draft RDMA Protocol Extensions November 2013
[RFC5040] Recio, R. et al., "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007.
[RFC5041] Shah, H. et al., "Direct Data Placement over Reliable
Transports", RFC 5041, October 2007.
[RFC5226] T. Narten and H. Alvestrand, "Guidelines for Writing an
IANA Considerations Section in RFCs", RFC 5226, BCP 26,
May 2008.
[RFCYYYY] Shah, H. et al., "RDMA Protocol Extensions", RFC YYYY,
October 2013.
[RFCZZZZ] "MPA Extensions" (not yet written)
11.2. Informative References
12. Acknowledgments
The authors would like to acknowledge the following contributors who
provided valuable comments and suggestions.
This document was prepared using 2-Word-v2.0.template.dot.
Wood et al. Expires May 26, 2014 [Page 17]
Internet-Draft RDMA Protocol Extensions November 2013
Appendix A. DDP Segment Formats for RDMA Messages
This appendix is for information only and is NOT part of the
standard. It simply depicts the DDP Segment format for the various
RDMA Messages.
A.1. DDP Segment for RDMA Read V2 Request
The following figure depicts an RDMA Read V2 Request DDP Segment:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read V2 Request) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read V2 Request) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read V2 Request) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RDMA Read Message Size (RDMARDSZ) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Source STag (SrcSTag) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Source Tagged Offset (SrcTO) |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5 RDMA Read V2 Request, DDP Segment Format
A.2. DDP Segment for RDMA Read V2 Response
The following figure depicts an RDMA Read V2 Response, DDP Segment:
Wood et al. Expires May 26, 2014 [Page 18]
Internet-Draft RDMA Protocol Extensions November 2013
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read V2 Response) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read V2 Response) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP (RDMA Read V2 Response) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6 RDMA Read V2 Response, DDP Segment Format
A.3. DDP Segment for Send with Immediate Data and Send with Solicited
Event and Immediate Data
The following figure depicts a Send with Immediate Data and Send
with Solicited and Immediate Data Request, DDP Segment:
Wood et al. Expires May 26, 2014 [Page 19]
Internet-Draft RDMA Protocol Extensions November 2013
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
0 1 2 3
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Send ULP Payload |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7 Send with Immediate Data and Send with Solicited Event and
Immediate Data (L bit = 0, payload only), DDP Segment Format
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
0 1 2 3
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved (Not Used) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Immediate Data |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8 Send with Immediate Data and Send with Solicited Event and
Immediate Data Operations (L bit = 1, Immediate Data only), DDP
Segment Format
Wood et al. Expires May 26, 2014 [Page 20]
Internet-Draft RDMA Protocol Extensions November 2013
This is the same format that is used in section A.4 of RFC 5040. The
differences are:
. The opcodes for the immediate data operations are different.
. The L bit will be set to 1 only in the ULPDU that contains
immediate data.
. The only data allowed in the ULPDU with the L bit set to 1 is
the immediate data itself.
If the send message length is zero, the only ULPDU sent has the
immediate data (also, the L bit is set to 1).
A.4. DDP Segment for Send with Invalidate and Immediate Data and Send
with Invalidate and Solicited Event and Immediate Data
The following figure depicts a Send with Invalidate and Immediate
Data and Send with Invalidate and Solicited and Immediate Data
Request, DDP Segment:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Invalidate STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Send ULP Payload |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 9 Send with Invalidate and Immediate Data and Send with
Invalidate and Solicited Event and Immediate Data (L bit = 0,
payload only), DDP Segment Format
Wood et al. Expires May 26, 2014 [Page 21]
Internet-Draft RDMA Protocol Extensions November 2013
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DDP Control | RDMA Control |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Invalidate STag |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Queue Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Send) Message Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Immediate Data |
+ +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 10 Send with Invalidate and Immediate Data and Send with
Invalidate and Solicited Event and Immediate Data (L bit = 1,
Immediate Data only), DDP Segment Format
This is the same format that is used in section A.5 of RFC 5040. The
differences are:
. The opcodes for the immediate data operations are different.
. The L bit will be set to 1 only in the ULPDU that contains
immediate data.
. The only data allowed in the ULPDU with the L bit set to 1 is
the immediate data itself.
If the send message length is zero, the only ULPDU sent has the
immediate data (also, the L bit is set to 1).
Wood et al. Expires May 26, 2014 [Page 22]
Internet-Draft RDMA Protocol Extensions November 2013
Authors' Addresses
Donald Wood
Intel Corporation
1300 South Mopac Expressway, Mailstop: AN4-4B
Austin, TX 78746
Phone: 1-512-362-1440
Email: donald.e.wood@intel.com
Robert Sharp
Intel Corporation
1300 South Mopac Expressway, Mailstop: AN4-4B
Austin, TX 78746
Phone: 1-512-362-1407
Email: robert.o.sharp@intel.com
Kenneth Keels
Intel Corporation
1300 South Mopac Expressway, Mailstop: AN4-4B
Austin, TX 78746
Phone: 1-512-362-1419
Email: kenneth.g.keels@intel.com
Wood et al. Expires May 26, 2014 [Page 23]
