RPC-over-RDMA Extensions to Reduce Internode Round-trips
draft-dnoveck-nfsv4-rpcrdma-rtrext-01

Abstract

It is expected that a future version of the RPC-over-RDMA transport will allow protocol extensions to be defined. This would provide for the specification of OPTIONAL features allowing participants who implement such features to cooperate as specified by that extension, while still interoperating with participants who do not support that extension.

A particular extension is described herein, whose purpose is to reduce the latency due to inter-node round-trips needed to effect operations which involve direct data placement or which transfer RPC messages longer than the fixed inline buffer size limit.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on June 6, 2017.

Copyright Notice

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Preliminaries
• 1.1. Requirements Language
• 1.2. Introduction
• 1.3. Prerequisites
• 1.4. Role Terminology
• 2. Extension Overview
• 3. Direct Data Placement Features
• 3.1. Current Situation
• 3.2. RDMA_MSGP
• 3.3. Send-based DDP
• 3.4. Other DDP-Related Extensions
• 4. Message Continuation Feature
• 4.1. Current Situation
• 4.2. Message Continuation Changes
• 4.3. Message Continuation and Credits
• 5. Using Protocol Additions
• 5.1. New Operation Support
• 5.2. Message Continuation Support
• 5.3. Send-based DDP Support
• 5.4. Error Reporting
• 6. XDR Preliminaries
• 6.1. Message Continuation Preliminaries
• 6.2. Data Placement Preliminaries
• 7. Data Placement Structures
• 7.1. Data Placement Overview
• 7.2. Buffer Structure Definition
• 7.3. Message DDP Structures
• 7.4. Response Direction DDP Structures
• 8. Transport Properties
• 8.1. Property List
• 8.2. RTR Support Property
• 8.3. Receive Buffer Structure Property
• 8.4. Request Transmission Receive Limit Property
• 8.5. Response Transmission Send Limit Property
• 9. New Operations
• 9.1. Operations List
• 9.2. Transmit Request Operation
• 9.3. Transmit Response Operation
• 9.4. Transmit Continue Operation
• 9.5. Error Reporting Operation
• 10. XDR
• 10.1. Code Component License
• 10.2. XDR Proper for Extension
• 11. Security Considerations
• 12. IANA Considerations
• 13. References
• 13.1. Normative References
• 13.2. Informative References
• Appendix A. Acknowledgements
• Author's Address

1. Preliminaries

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

1.2. Introduction

This document describes a potential extension to the RPC-over-RDMA protocol, which would allow participating implementations to have more flexibility in how they use RDMA sends and receives to effect necessary transmission of RPC requests and replies.

In contrast to existing facilities defined in RPC-over-RDMA Version One in which the mapping between RPC messages and RPC-over-RDMA messages is strictly one-to-one and DDP is effected only through use of explicit RDMA operations, the following features are made available through this extension:

• The ability to effect Direct Data Placement in the context of a single RPC-over-RDMA transmission, rather than requiring explicit RDMA operations to effect the necessary placement.
• The ability to continue an RPC request or reply over multiple RPC-over-RDMA transmissions

1.3. Prerequisites

This document is written assuming that certain underlying facilities will be made available to build upon, in the context of a future version of RPC-over-RDMA. It is most likely that such facilities will be first available in Version Two of RPC-over-RDMA.

• A protocol extension mechanism is needed to enable the extensions to RPC-over-RDMA described here.
  
  This document is currently written to conform to the extension model for the proposed RPC-over-RDMA Version Two as described in [rpcrdmav2].
• An existing means of communicating transport properties between the RPC-over-RDMA endpoints is assumed.
  
  This document is currently written assuming the transport property model defined in [rpcrdmav2] will be available and can be extended to meet the needs of this extension.

As the document referred to above is currently a personal Internet Draft, and subject to change, adjustments to this document are expected to be necessary when and if the needed facilities are defined in one or more working group documents.

1.4. Role Terminology

A number of different terms are used regarding the roles of the two participants in an RPC-over-RMA connection. Some of these roles last for the duration of a connection while others vary from request to request or from message to message.

The roles of the client and server are fixed for the lifetime of the connection, with the client defined as the endpoint which initiated the connection.

The roles of requester and responder often parallel those of client and server, although this is not always the case. Most requests are made in the forward direction, in which the client is the requester and the server is the responder. However, backward direction requests are possible, in which case the server is the requester and the client is the responder. As a result clients and servers may both act as requesters and responders for different requests issued on the same connection.

The roles of sender and receiver vary from message to messages. With regard to the messages described in this document, the sender may act as a requester by sending RPC requests or a responder by sending RPC requests or as both at the same time by sending a mix of the two.

2. Extension Overview

This extension is intended to function as part of RPC-over-RDMA and implementations should successfully interoperate with existing RPC-over-RDMA Version One implementations. Nevertheless, this extension seeks to take a somewhat different approach to high-performance RPC operation than has been used previously in that it seeks to de-emphasize the use of explicit RDMA operations. It does this in two ways:

• By implementing a send-based form of Direct Data Placement (see Section 3), use of explicit RDMA operations can be avoided in many common cases in which data is directly placed.
• Use of explicit RDMA to support reply chunks and position-zero read chunks can be avoided by allowing a single message to be split into multiple transmissions. This can be used to avoid many instances of the only existing use of explicit RDMA operations not associated with Direct Data Placement.

While use of explicit RDMA operations allows the cost of the actual data transfer to be offloaded from the client and server CPUs to the RNIC, there are ancillary costs in setting up the transfer that cannot be ignored. As a result, send-based functions are often preferable, since the RNIC also uses DMA to effect these operations. In addition, the cost of the additional inter-node round trips required by explicit RDMA operation can be an issue, which can becomes increasingly troublesome as internode distances increase. Once one moves from in-machine-room to campus-wide or metropolitan-area distances the additional round-trip delay of 16 microseconds per mile becomes an issue impeding use of explicit RDMA operations.

3. Direct Data Placement Features

3.1. Current Situation

Although explicit RDMA operations are used in the existing RPC-over-RDMA protocol for purposes unrelated to Direct Data Placement, all DDP is effected using explicit RDMA operations.

As a result, many operations involving Direct Data Placement involve multiple internode round trips.

3.2. RDMA_MSGP

Although this was not stated explicitly, it appears that RDMA_MSGP (defined in [RFC5666], removed from RPC-over-RDMA Version One by [rfc5666bis]), was an early attempt to effect correct placement of bulk data within a single RPC-over-RDMA transmission.

As things turned out, the fields within the RDMA_MSGP header were not described in [RFC5666] in a way that allowed this message type to be implemented.

In attempting to provide DDP functionality, we have to keep in mind and avoid the problems that led to failure of RDMA_MSGP. It appears that the problems go deeper than neglecting to write a few relevant sentences. It is helpful to note that:

• The inline message size limits eventually adopted were too small to allow RDMA_MSGP to be used effectively. This is true of both the 1K limit in Version One [rfc5666bis] and the 4K limit specified in [rpcrdmav2].
  
  On the other hand, there is text within [RFC5667] that suggests that much longer messages were anticipated at some points during the evolution of RPC-over-RDMA.
• The fact that NFSv4 COMPOUNDs often have additional operations beyond the one including the bulk data means that the RDMA_MSGP model cannot be extended to NFSv4. As a result, the bulk data needs to be excised from the data stream just as chunks are, so that the payload stream can include non-bulk data both before and after the logical position of the excised bulk data.
• In order for the sender to determine the appropriate amount of padding necessary within a transmission to place the bulk data at the proper position within receive buffer, the server must know more about the structure of the receiver's buffers. Since the padding needs to bring the bulk data to a position within the buffer that is appropriate to receive the bulk data, the sender needs to know where within the receive buffers such DDP-eligible areas are located.
• While appropriate padding could place the bulk data within a large WRITE into an appropriately aligned buffer or set of buffer, there is no corresponding provision for the bulk data associated with a READ. In short, there is no way to indicate to the responder that it should use RDMA_MSGP to appropriately place bulk data in the response.
• There is no explicit discussion of the required padding's use in effecting proper data placement or connection with the ULB's specification of DDP-eligible XDR.

To summarize, RDMA_MSGP was an attempt to properly place data which was thought of as a local optimization and insufficient attention was given to it to make it successful. As a result, as RPC-over-RDMA Version One was developed, Direct Data Placement was identified with the use of explicit RDMA operations, and the possibility of Data Placement within sends was not recognized.

3.3. Send-based DDP

In this exension we will describe a more cmplete way to provide send-based data placement, as follows:

• By defining the structure of receive buffers as a transport property available to be interrogated by the peer implementation.
• By treating positioning of bulk data within a message as an instance of DDP, causing the bulk data to be excised from the payload XDR stream, as is the case with other forms of DDP.
• By defining new DDP control data structures that support both send-based DDP and the form of DDP using explicit RDMA operations that was specified in RPC-over-RDMA Version One. These new control structures, described in Section 7.1 are organized differently from the chunk-based structures described in [rfc5666bis].

3.4. Other DDP-Related Extensions

In order to support send-based DDP, new DDP-related data structures have been defined, as described in Sections 7.3 and 7.4.

These new data structures support both send-based and RDMA-operation-based DDP. In addition, because of the restructuring described in Section 7.1, a number of additional facilities are made available:

• The ability to restrict entries regarding DDP in response data to XDR data items generated in response to performing particular constituent operations within a given RPC request (e.g. specific operations within an NFSv4 COMPOUND).
• The ability to make use of DDP contingent on the actual length of a DDP-eligible data item in the response.
• The ability to specify whether use of DDP for a particular DDP-eligible data item is required or optional.

These additional facilities will be available to implementations that do not support send-based DDP, as long as both parties support the OPTIONAL Header types that include these new structures. For more information about the relationships among, the new transport properties, operations, and features, see Section 5.

4. Message Continuation Feature

4.1. Current Situation

Within RPC-over-RDMA Version One [rfc5666bis], each transmission of a request or reply involves sending a single RDMA send message and conversely each message-related transmission involves only a single RPC request or reply.

This strict one-to-one model leads to some potential performance issues.

• Because of RDMA's use of fixed-size receives, some requests and replies will inevitably not fit in the limited space available, even if they do not contain any DDP-eligible bulk data.
  
  Such cases will raise performance issues because, to deal with them, the server is interrupted twice to receive a single request and all the necessary transfers are serialized. In particular, there are two server interrupt latencies involved before the server can process the actual request, in addition to the OTW round-trip latencies.
• In the case of replies, there may be cases in which reply chucks need to be allocated and registered even if the actual reply would fit within the fixed receive-size limit. Because the decision to create a reply chunk is made at the time the request is sent, even an extremely low probability of a longer reply will trigger allocation of a reply chunk.
  
  Because this decision is made in conformance with ULB rules, which, by their nature, may only reference a limited set of data, a reply chunk may be required even when the actual probability of a long reply is exactly zero. For example a GETATTR request can generate a long reply due to a long ACL, and thus COMPOUND with this operation might allocate a reply chunk, even if the specific file system being interrogated only supports ACLs of limited sizes, or the GETATTR in question does not interrogate one of the ACL attributes. Also, the OWNER attribute is a string and may be impossible to determine a priori that the owner of any particular file has no chance of requiring more than 4K bytes of space, for example. The assumption there are no such user names, while it probably is valid, is not a fact that RPC-over-RDMA implementations can depend on.

4.2. Message Continuation Changes

Continuing a single RPC request or reply is addressed by defining separate optional header types to begin and to continue sending a single RPC message. This is instead of creating a header with a continuation bit. In this approach, all of the DDP-related fields, which include support for send-based DDP, appear in the starting header (of types ROPT_XMTREQ and ROPT_XMTRESP) and apply to the RPC message as a whole.

Later RPC-over-RDMA messages (of type ROPT_XMTCONT) may extend the payload stream and/or provide additional buffers to which bulk data can be directed.

In this case, all of the RPC-over-RDMA messages used together are referred to as a transmission group and must be received in order without any intervening message.

In implementations using this optional facility, those decoding RPC messages received using RPC-over-RDMA no longer have the assurance that that each RPC message is in a contiguous buffer. As most XDR implementations are built based on the assumption that input will not be contiguous, this will not affect performance in most cases.

4.3. Message Continuation and Credits

Using multiple transmissions to send a single request or response can complicate credit management. In the case of the message continuation feature, deadlocks can be avoided because use of message continuation is not obligatory. The requester or responder can use explicit RDMA operations if sufficient credits to use message continuation are not available.

A requester is well positioned to make this choice with regard to the sending of requests. The requester must know, before sending a request, how long it will be, and therefore, how many credits it would require to send the request using message continuation. If these are not available, it can avoid message continuation by either creating read chunks sufficient to make the payload stream fit in a single transmission or by creating a position-zero read chunk.

With regard to the response, the requester is not in position to know exactly how long the response will be. However, the ULB will allow the maximum response length to be determined based on the request. This value can be used:

• To determine the maximum number of receive buffers that might be required to receive any response sent.
• To allocate and register a reply chunk to hold a possible large reply.

The requester can avoid doing the second of these if the responder has indicated it can use message continuation to send the response. In this case, it makes sure that the buffers will be available and indicates to the responder how many additional buffers (in the form of pre-posted reads have been made available to accommodate continuation transmissions.

When the responder processes the request, those additional receive buffers may be used or not, or used only in part. This may be because the response is shorter than the maximum possible response, or because a reply chunk was used to transmit the response.

After the first or only transmission associated with the response is received by the requester, it can be determined how many of the additional buffers were used for the response. Any unused buffers can be made available for other uses such as expanding the pool of receive buffers available for the initial transmissions of response or for receiving opposite direction requests. Alternatively, they can be kept in reserve for future uses, such as being made available to future requests which have potentially long responses.

5. Using Protocol Additions

In using existing RPC-over-RDMA facilities for protocol extension, interoperability with existing implementations needs to be assured. Because this document describes support for multiple features, we need to clearly specify the various possible extensions and how peers can determine whether certain facilities are supported by both ends of the connection.

5.1. New Operation Support

Note that most of the new operations defined in this extension are not tightly tied to a specific feature. XOPT_XMTREQ and XOPT_XMTRESP are designed to support implementations that support either or both Send-based DDP or message continuation. However, the converse is not the case and these header types can be implemented by those not supporting either of these features. For example, implementations may only need support for the facilities described in Section 3.4.

Implementations may determine whether a peer implementation supports XOPT_XMTREQ, XOPT_XMTREQ, or XOPT_XMTCONT by attempting these operations. An alternative is to interrogate the RTR Support Property for information about which operations are supported.

5.2. Message Continuation Support

Implementations may determine and act based on the level of peer implementation of support for message continuation as follows:

• To deal with issues relating to sending the peer multi-transmission requests, the requester can interrogate the peer's value of the Request Transmission Receive Limit (Section 8.4). In cases in which the property is not provided or has the value one, the requester implementation can avoid sending multi-transmission requests, and use the equivalent of position-zero read chunks to convey a request larger than the receive buffer limit.
  
  Similarly, if the request is longer than can fit in a set of transmissions given that limit, the request can be conveyed in the same fashion,
• To deal with issues relating to sending the peer multi-transmission responses, responders will only send multi-transmission responses for requests conveyed using XOPT_XMTREQ where the number of response transmissions is less than or equal to buffer reservation count (in the field optxrq_rsbuf). The requester can avoid receiving a message consisting of too many transmissions by setting this field appropriately. This includes the case in which the requester cannot handle any multi-transmission responses.
• To avoid reserving receive buffers that the responder is not prepared to use, the requester can interrogate the peer's value of the Response Transmission Send Receive Limit (Section 8.5). In cases in which it is possible that a request might result in a response too large for this set of buffers, the requester, the requester can provide a reply chunk to receive the response, which the responder can use if the count of buffers provided is insufficient.

5.3. Send-based DDP Support

Implementations may determine and adapt to the level of peer implementation support for send-based DDP as described below. Note that an implementation may be able to send messages containing bulk data items placed using send-based DDP while not being prepared to receive them, or the reverse.

• The requester can interrogate the responder's Receive Buffer Structure Property. In cases in which the property is not provided or shows no DDP-targetable buffer segments, an implementation knows that messages containing bulk data may not be sent using send-based DDP. In such cases, when XOPT_XMTREQ is used to send a request, bulk items may be transferred by setting the associated DDP information to indicate that the bulk data is to be fetched using explicit RDMA operations.
• In cases in which a requester is unprepared to accept messages using send-based DDP, its Receive Buffer Structure Property will make this clear to the responder. Nevertheless, the requester will generally indicate to the responder that bulk data items are to be returned using explicit RDMA operations. As a result, requesters may use XOPT_XMTREQ (and get the benefit of the DDP-related features discussed in Section 3.4 even if they support neither message continuation nor send-based DDP.
• Since it is possible for a responder to generate responses containing bulk data using send-based DDP even if it is not prepared to send such message, a requester who is prepared to accept such messages should specify in the request that the responses are to contain (or may contain) bulk data placed in this way. In deciding whether this is to be done the requester can interrogate the responder's RTR Support Property for information about which whether the peer can send responses in this form. It can do this without regard to whether the responder can accept messages containing bulk data items placed using send-based DDP.

In determining whether bulk data will be placed using send-based DDP or via explicit RDMA operations, the level of support for message continuation will have a role. This is because DDP using explicit RDMA will reduce message size while send-based DDP reduces the size of the payload stream by rearranging the message, leaving the message size the same. As a result, the considerations discussed in Section 4.3 will have to be attended to by the sender in determining which form of DDP is to be used.

5.4. Error Reporting

The more extensive transport layer functionality described in this document requires its own means of reporting errors, to deal with issues that are distinct from:

• Errors (including XDR errors) in the XDR stream as received by responder or requester.
• XDR errors detected in the XDR headers defined by the base protocol.
• XDR errors detected in the new operations defined in this document.

Beyond the above, the following sorts of errors will have to be dealt with, depending on which of the features of the extension are implemented.

• Information associated with send-based DDP may be inconsistent or otherwise invalid, even though it conforms to the XDR definition.
• There may be problems with the organization of transmission groups in that there are missing or extraneous transmissions.

In each of the above cases, the problem will be reported to the sender using the Error Reporting operation which needs to be supported by every endpoint that sends ROPT_XMTREQ, ROPT_XMTRESP, or ROPT_XMTCONT. This includes cases in which the problem is one with a reply. The function of the Error Reporting operation is to aid in diagnosing transport protocol errors and allowing the sender to recover or decide recovery is not possible. Reporting failure to the requesting process is dealt with indirectly. For example,

• When the transmissions used to send a request are ill-formed, the requestor can respond to the error indication by proceeding to send the request using existing (i.e. non-extended) facilities. If it chooses not to do so, the requestor can report an RPC request failure to the initiator of the RPC.
• When the transmissions used to send a response are ill-formed, the responder need to know about the problem since it will otherwise assume that the transmissions succeeded. It can proceed to resend the reply using existing (i.e. non-extended) facilities. If it chooses not to do so, the requester will not see a response and eventually an RPC timeout will occur.

6. XDR Preliminaries

6.1. Message Continuation Preliminaries

<CODE BEGINS>

typedef uint32  xms_grpxn;
typedef uint32  xms_grpxc;
struct xms_id {
        uint32         xmsi_xid;
        msg_type       xmsi_dir;
        xms_grpxn      xmsi_seq;
} 

<CODE ENDS>
          

In order to implement message continuation, we have occasion to refer to particular RPC-over-RDMA transmissions within a transmission group or to characteristics of a later transmission group.

An xms_grpxn designates a particular RPC-over-RDMA transmission within a set of transmissions devoted to sending a single RPC message.

An xms_grpxc specifies the number of RPC-over-RDMA transmissions in a potential group of transmissions devoted to sending a single RPC message.

6.2. Data Placement Preliminaries

<CODE BEGINS>

typedef uint32  xmddp_itemlen;
typedef uint32  xmddp_pldisp;
typedef uint32  xmddp_vsdisp;

typedef uint32  xmddp_tbsn;
 
enum xmddp_type {
        XMDTYPE_EXRW = 1,
        XMDTYPE_TBSN = 2,
        XMDTYPE_CHOOSE = 3,
        XMDTYPE_BYSIZE = 4,
        XMDTYPE_TOOSHORT = 5,
        XMDTYPE_NOITEM = 6
};
                
<CODE ENDS>
          

Data structures related to data placement use a number of XDR typedefs to help clarify the meaning of fields in the data structures which use these typedefs.

An xmddp_itemlen specifies the length of XDR item. Because items excised from the XDR stream are XDR items, lengths of items excised from the XDR stream are denoted by xmddp_itemlens.

An xmddp_pldisp specifies a specific displacement with the payload stream associated with a single RPC-over-RDNA transmission or a group of such transmissions. Note that when multiple transmissions are used for a single message, all of the payload streams within a transmission group are considered concatenated.

An xmddp_vsdisp specifies a displacement within the virtual XDR stream associates with the set of RPC messages transferred by single RPC-over-RDNA transmission or a group of such transmissions. The virtual XDR stream includes bulk data excised from the payload stream and so displacements within it reflect those of the corresponding objects in the XDR stream that might be sent and received if no bulk data excision facilities were involved in the RPC transmission.

An xmddp_tbsn designates a particular target buffer segment within a (trivial or non-trivial) RPC-over-RDMA transmission group. Each DDP-targetable buffer segment is assigned a number starting with zero and proceeding through all the buffer segments for all the RPC-over-RDMA transmissions in the group. This includes buffer segments not actually used because transmission are shorter than the maximum size and those in which a DDP-targetable buffer segment is used to hold part of the payload XDR stream rather than bulk data.

An xmddp_type allows a selection between DDP using explicit RDMA operations and that using send-based DDP. It is used in a number of contexts. The specific context governs which subset of the types is valid:

• In request messages, they indicate where each of the directly-placed data items within the request has been placed. In this case, xmddp_type appears as the discriminator within an xmddp_loc which is part of an xmddp_mitem that is an element within a request's optxrq_ddp field.
• In request messages, they direct the responder as to where potential directly-placed items are to be placed. In this case, xmddp_type appears as the discriminator within an xmddp_rsdloc which is part of an xmddp_rsditem that is an element within a request's optxrq_rsd field.
• In response messages, they indicate how each of the potential directly-placed items has been dealt with. A subset of these are directly-placed data items and are presented in the same form as that used for directly-placed data items within a request. In this case, xmddp_type appears as the discriminator within an xmddp_loc which is part of an xmddp_mitem that is an element within a response's optxrs_ddp field.

A number of these type are valid in all of these contexts, since they specify use of a specific mode of direct placement which is to be used or has been used.

• XMDTYPE_EXRW selects DDP using explicit RDMA reads and writes.
• XMDTYPE_TBSN selects use of send-based DDP in which DDP-eligible data is located in DDP-targetable buffer segments.

Another set of types is used to direct the use of specific sets of types but cannot specify an actual choice that has been made.

• XMDTYPE_CHOICE indicates that the responder may use either send-based DDP or chunk-based DDP using explicit RDMA operations, with a place for the latter having been provided by the requester.
• XMDTYPE_BYSIZE indicates that the responder is to use either send-based DDP or chunk-based DDP using explicit RDMA operations, with the choice between the two governed by the actual size of the associated DDP-eligible XDR item.

The following types are used when no actual direct placement has occurred. They are used in responses to indicate ways in which a direction to govern DDP in a reply was responded to without resulting in direct placement.

• XMDTYPE_TOOSHORT indicates that the corresponding entry in an xmddp_rsdset was matched with a DDP-eligible item which was too small to be handled using direct placement, resulting in the DDP-eligible item being placed inline.
• XMDTYPE_NOITEM indicates that the corresponding entry in an xmddp_rsdset was not matched with a DDP-eligible item in the reply.

The following table indicates which of the above types is valid in each of the contexts in which these types may appear. For valid occurrences, it distinguishes those which give sender-generated information about the message, and those that direct reply construction, from those that indicate how those directions governed the construction of a reply. For invalid occurrences, we distinguish between those that result in XDR decode errors and those which are valid from the XDR point of view but are semantically invalid.

<CODE BEGINS>

struct xmrbs_seg {
        uint32          xmrseg_length;
        uint32          xmrseg_align;
        uint32          xmrseg_flags;
};

const uint32    XMRSFLAG_DDP = 0x01;


struct xmrbs_group {
        uint32          xmrgrp_count;
        xmrbs_seg       xmrgrp_info;
};

struct xmrbs_buf {
        uint32          xmrbuf_length;
        xmrbs_group     xmrbuf_groups<>;
};

<CODE ENDS>
          

<CODE BEGINS>

union xmddp_loc switch(xmddp_type type)

        case XMDTPE_EXRW:
                rpcrdma1_segment        xmdl_ex<>;
        case XMDTYPE_TBSN:
                xmddp_itemlen           xmdl_offset;
                xmddp_tbsn              xmdl_bsnum<>;
        case XMDTYPE_TOOSHORT:
        case XMDTYPE_NOITEM:
                void;
};


struct xmddp_mitem {
        xmddp_vsdisp    xmdmi_disp;
        xmddp_itemlen   xmdmi_length;
        xmddp_loc       xmdmi_where;
};

typedef xmddp_mitem     xmddp_grpinfo<>;
         
<CODE ENDS>
          

<CODE BEGINS>

union xmddp_rsdloc switch(xmddp_type type)

        case XMDTPE_EXRW:
        case XMDTPE_CHOICE:
                rpcrdma1_segment        xmdrsdl_ex<>;
        case XMDTPE_BYSIZE:
                xmddp_itemlen           xmdrsdl_dsdov;
                rpcrdma1_segment        xmdrsdl_bsex<>;
        case XMDTYPE_TBSN:
                void;
};

struct xmddp_rsdrange {
        xmddp_vsdisp    xmdrsdr_begin;
        xmddp_vsdisp    xmdrsdr_end;
};

struct xmddp_rsditem {
        xmddp_itemlen   xmdrsdi_minlen;
        xmddp_rsdloc    xmdrsdi_loc;
};

struct xmddp_rsdset {
        xmddp_rsdrange  xmdrsds_range;
        xmddp_rsditem   xmdrsds_items<>;
};

typedef xmddp_rsdset    xmddp_rsdgroup<>;

<CODE ENDS>
          

<CODE BEGINS>

const uint32           XPROP_RTRSUPP = 3;
typedef uint32         xpr_rtrs;

const uint32           RTRS_XREQ = 1;
const uint32           RTRS_XRESP = 2;
const uint32           RTRS_XCONT = 4;

<CODE ENDS>
          

<CODE BEGINS>

const uint32           XPROP_RBSTRUCT = 4;
typedef xmrbs_buf      xpr_rbs;

<CODE ENDS>
          

<CODE BEGINS>

const uint32           XPROP_REQRXLIM = 5;
typedef uint32         xpr_rqrxl;

<CODE ENDS>
         

<CODE BEGINS>

const uint32           XPROP_RESPSXLIM = 6;
typedef uint32         xpr_rssxl;

<CODE ENDS>
          

<CODE BEGINS>

const uint32     ROPT_XMTREQ = 1;

struct optxmt_req {
        xmddp_grpinfo   optxrq_ddp;
        xmddp_rsdgroup  optxrq_rsd;
        xms_grpxc       optxrq_count;
        xms_grpxc       optxrq_rsbuf;
        xmddp_pldisp    optxrq_pslen;

};        
 
<CODE ENDS>
          

<CODE BEGINS>

const uint32     ROPT_XMTRESP = 2;

struct optxmt_resp {
        xmddp_grpinfo   optxrs_ddp;
        xms_grpxn       optxrs_count;
        xmddp_pldisp    optxrs_pslen;

};        
 
<CODE ENDS>
          

<CODE BEGINS>

const uint32     ROPT_XMTCONT = 3;

struct optxmt_cont {
        xms_grpxn       optxc_xnum;
        uint32          optxc_itype;
        xmddp_pldisp;   optxc_pslen;
};        
 
<CODE ENDS>
          

<CODE BEGINS>

enum optr_err {
        OPTRERR_BADHMT = 1,
        OPTRERR_BADOMT = 2,
        OPTRERR_BADCONT = 3,
        OPTRERR_BADSEQ = 4,
        OPTRERR_BADXID = 5,
        OPTRERR_BADOFF = 6,
        OPTRERR_BADTBSN = 7,
        OPTRERR_BADPL = 8
}

union optr_info switch(optr_err optre_which) {

  case OPTRERR_BADHMT:
  case OPTRERR_BADOMT:
  case OPTRERR_BADSEQ:
  case OPTRERR_BADXID:
        uint32          optri_expect;
        uint32          optri_current;

  case OPTRERR_BADCONT:
        void;


  case OPTRERR_BADTBSN:
  case OPTRERR_BADOFF:
  case OPTRERR_BADPL:
        uint32          optri_value;
        uint32          optri_min;
        uint32          optri_max;

};

<CODE ENDS>
          

<CODE BEGINS>

const uint32     ROPT_REPTERR = 4;

struct optrept_err {
        xms_id          optre_bad;
        xms_id          *optre_lead;
        optr_info       optre_info;
};        
 
<CODE ENDS>
          

<CODE BEGINS>

#!/bin/sh
grep '^ *///' | sed 's?^ /// ??' | sed 's?^ *///$??'

<CODE ENDS>

       

<CODE BEGINS>

sh extract.sh < ext.txt > xmitext.x

<CODE ENDS>

        

<CODE BEGINS>

/// /*
///  * Copyright (c) 2010, 2016 IETF Trust and the persons
///  * identified as authors of the code.  All rights reserved.
///  *
///  * The author of the code is: D. Noveck.
///  *
///  * Redistribution and use in source and binary forms, with
///  * or without modification, are permitted provided that the
///  * following conditions are met:
///  *
///  * - Redistributions of source code must retain the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer.
///  *
///  * - Redistributions in binary form must reproduce the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer in the documentation and/or other
///  *   materials provided with the distribution.
///  *
///  * - Neither the name of Internet Society, IETF or IETF
///  *   Trust, nor the names of specific contributors, may be
///  *   used to endorse or promote products derived from this
///  *   software without specific prior written permission.
///  *
///  *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS
///  *   AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
///  *   WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
///  *   IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
///  *   FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
///  *   EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
///  *   LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
///  *   EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
///  *   NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
///  *   SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
///  *   INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
///  *   LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
///  *   OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
///  *   IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
///  *   ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
///  */

<CODE ENDS>

          

<CODE BEGINS>
/// /*******************************************************************
///  *******************************************************************
///  ** 
///  **  XDR for OPTIONAL protocol extension.
///  ** 
///  **  Includes support for both message continuation and send-based 
///  **  DDP. The latter is supported by a new structure for the 
///  **  specification of data placements which can be used for both 
///  **  send-based DDP and DDP using explicit RDMA operations.
///  ** 
///  **  Extensions include:
///  ** 
///  **     o Four new transport properties.
///  **     o Four new OPTIONAL message types
///  **     
///  *******************************************************************
///  ******************************************************************/
///
/// /*******************************************************************
///  *
///  *                   Core XDR Definitions
///  *
///  ******************************************************************/

/// /* 
///  * General XDR preliminaries for these features,
///  */
/// typedef uint32  xms_grpxn;
/// typedef uint32  xms_grpxc;
/// 
/// /* 
///  * Basic XDR typedefs for the new approach to DDP Specification.
///  */
/// typedef uint32  xmddp_itemlen;
/// typedef uint32  xmddp_pldisp;
/// typedef uint32  xmddp_vsdisp;
/// typedef uint32  xmddp_tbsn;
///  
/// /* 
///  * Define the possible types of DDP items.
///  */
/// enum xmddp_type {
///         XMDTYPE_EXRW = 1,
///         XMDTYPE_TBSN = 2,
///         XMDTYPE_CHOOSE = 3,
///         XMDTYPE_BYSIZE = 4,
///         XMDTYPE_TOOSHORT = 5,
///         XMDTYPE_NOITEM = 6
/// };
/// 
/// /*
///  * XDR defining the placemebnt of bulk items in the message being 
///  * sent.
///  */
/// union xmddp_loc switch(xmddp_type type)
/// 
///         case XMDTPE_EXRW:
///                 rpcrdma1_segment        xmdl_ex<>;
///         case XMDTYPE_TBSN:
///                 xmddp_itemlen           xmdl_offset;
///                 xmddp_tbsn              xmdl_bsnum<>;
///         case XMDTYPE_TOOSHORT:
///         case XMDTYPE_NOITEM:
///                 void;
/// };
/// 
/// 
/// 
/// struct xmddp_mitem {
///         xmddp_vsdisp    xmdmi_disp;
///         xmddp_itemlen   xmdmi_length;
///         xmddp_loc       xmdmi_where;
/// };
/// 
/// typedef xmddp_mitem     xmddp_grpinfo<>;
/// 
/// /*
///  * XDR defining the placement of bulk items in the response to the
///  * message being sent.
///  */
/// union xmddp_rsdloc switch(xmddp_type type)
/// 
///         case XMDTPE_EXRW:
///         case XMDTPE_CHOICE:
///                 rpcrdma1_segment        xmdrsdl_ex<>;
///         case XMDTPE_BYSIZE:
///                 xmddp_itemlen           xmdrsdl_dsdov;
///                 rpcrdma1_segment        xmdrsdl_bsex<>;
///         case XMDTYPE_TBSN:
///                 void;
/// };
/// 
/// struct xmddp_rsdrange {
///         xmddp_vsdisp    xmdrsdr_begin;
///         xmddp_vsdisp    xmdrsdr_end;
/// };
/// 
/// struct xmddp_rsditem {
///         xmddp_itemlen   xmdrsdi_minlen;
///         xmddp_rsdloc    xmdrsdi_loc;
/// };
/// 
/// struct xmddp_rsdset {
///         xmddp_rsdrange  xmdrsds_range;
///         xmddp_rsditem   xmdrsds_items<>;
/// };
/// 
/// typedef xmddp_rsdset    xmddp_rsdgroup<>;
///
/// /*******************************************************************
///  *
///  *                     New Transport Properties           
///  *
///  ******************************************************************/
///
/// /* 
///  * New Transport Property codes 
///  */ 
/// const uint32           XPROP_RTRSUPP = 3;
/// const uint32           XPROP_RBSTRUCT = 4;
/// const uint32           XPROP_REQRXLIM = 5;
/// const uint32           XPROP_RESPSXLIM = 6;
/// 
/// /*
///  * XDR relating to RTR Support Property
///  */
/// typedef uint32         xpr_rtrs;
/// 
/// const uint32           RTRS_XREQ = 1;
/// const uint32           RTRS_XRESP = 2;
/// const uint32           RTRS_XCONT = 4;
/// 
/// /* 
///  * Items related to Receive Buffer Structure Property
///  */
/// struct xmrbs_seg {
///         uint32          xmrseg_length;
///         uint32          xmrseg_align;
///         uint32          xmrseg_flags;
/// };
/// 
/// const uint32    XMRSFLAG_DDP = 0x01; 
/// 
/// struct xmrbs_group {
///         uint32          xmrgrp_count;
///         xmrbs_seg       xmrgrp_info;
/// };
/// 
/// struct xmrbs_buf {
///         uint32          xmrbuf_length;
///         xmrbs_group     xmrbuf_groups<>;
/// };
/// typedef xmrbs_buf      xpr_rbs;
/// 
/// /*
///  * XDR relating to tranismission limit properties
///  */
/// typedef uint32         xpr_rqrxl;
/// 
/// typedef uint32         xpr_rssxl;
///
/// /*******************************************************************
///  *
///  *                     New OPTIONAL Message Types           
///  *
///  ******************************************************************/
///
/// /* 
///  * New message type codes 
///  */ 
/// const uint32     ROPT_XMTREQ = 1;
/// const uint32     ROPT_XMTRESP = 2;
/// const uint32     ROPT_XMTCONT = 3;
/// const uint32     ROPT_REPTERR = 4;
/// 
/// 
/// /*
///  * New message type to do the initial transmission of a request.
///  */ 
/// struct optxmt_req {
///         xmddp_grpinfo   optxrq_ddp;
///         xmddp_rsdgroup  optxrq_rsd;
///         xms_grpxc       optxrq_count;
///         xms_grpxc       optxrq_rsbuf;
///         xmddp_pldisp    optxrq_pslen;
/// 
/// };        
///  
/// /*
///  * New message type to do the initial transmission of a response.
///  */ 
/// struct optxmt_resp {
///         xmddp_grpinfo   optxrs_ddp;
///         xms_grpxn       optxrs_count;
///         xmddp_pldisp    optxrs_pslen;
/// 
/// };        
/// 
/// /*
///  * New message type to transmit the continuation of a request or
///  * response.
///  */ 
/// struct optxmt_cont {
///         xms_grpxn       optxc_xnum;
///         uint32          optxc_itype;
///         xmddp_pldisp;   optxc_pslen;
/// };        
/// 
/// /*
///  * XDR definitions to support error reporting.
///  */ 
/// enum optr_err {
///         OPTRERR_BADHMT = 1,
///         OPTRERR_BADOMT = 2,
///         OPTRERR_BADCONT = 3,
///         OPTRERR_BADSEQ = 4,
///         OPTRERR_BADXID = 5,
///         OPTRERR_BADOFF = 6,
///         OPTRERR_BADTBSN = 7,
///         OPTRERR_BADPL = 8
/// }
/// 
/// union optr_info switch(optr_err optre_which) {
/// 
///   case OPTRERR_BADHMT:
///   case OPTRERR_BADOMT:
///   case OPTRERR_BADSEQ:
///   case OPTRERR_BADXID:
///         uint32          optri_expect;
///         uint32          optri_current;
/// 
///   case OPTRERR_BADCONT:
///         void;
/// 
/// 
///   case OPTRERR_BADTBSN:
///   case OPTRERR_BADOFF:
///   case OPTRERR_BADPL:
///         uint32          optri_value;
///         uint32          optri_min;
///         uint32          optri_max;
/// 
/// };
/// 
/// struct xms_id {
///         uint32         xmsi_xid;
///         msg_type       xmsi_dir;
///         xms_grpxn      xmsi_seq;
/// };
///
/// /*
///  * New message type for error reporting.
///  */ 
/// struct optrept_err {
///         xms_id          optre_bad;
///         xms_id          *optre_lead;
///         optr_info       optre_info;
/// };        
///  
/// 
<CODE ENDS>