operation	code	XDR type	msg	section
Specify Properties at Connection	1	optinfo_connprop	No	5.3.1
Request Property Modification	2	rpcrdma2_reqprop	No	5.3.2
Respond to Modification Request	3	rpcrdma2_resprop	No	5.3.3
Report Updated Properties	4	rpcrdma2_updprop	No	5.3.4

Support for all of the operations above is OPTIONAL. RPC-over-RDMA Version Two implementations that receive an operation that is not supported MUST respond with RDMA_ERROR message with an error code of RDMA_ERR_INVAL_OPTION.

The only operation support requirements are as follows:

• Implementations which send RDMA2_REQPROP messages must support RDMA2_RESPROP messages.
• Implementations which support RDMA2_RESPROP or RDMA2_UPDPROP messages must also support RDMA2_CONNPROP messages.

5.3.1. RDMA2_CONNPROP: Specify Properties at Connection

The RDMA2_CONNPROP message type allows an RPC-over-RDMA participant, whether client or server, to indicate to its partner relevant transport properties that the partner might need to be aware of.

<CODE BEGINS>

struct rpcrdma2_connprop {
        rpcrdma2_propset rdma_start;
        rpcrdma2_propsubset rdma_nochg;
};        
 
<CODE ENDS>

          

The message definition for this operation is as follows:

All relevant transport properties that the sender is aware of should be included in rdma_start. Since support of this request is OPTIONAL, and since each of the properties is OPTIONAL as well, the sender cannot assume that the receiver will necessarily take note of these properties and so the sender should be prepared for cases in which the partner continues to assume that the default value for a particular property is still in effect.

Values of the subset of transport properties specified by rdma_nochg is not expected to change during the lifetime of the connection.

Generally, a participant will send a RDMA2_CONNPROP message as the first message after a connection is established. Given that fact, the sender should make sure that the message can be received by partners who use the default Receive Buffer Size. The connection's initial receive buffer size is typically 1KB, but it depends on the initial connection state of the RPC-over-RDMA version in use.

Properties not included in rdma_start are to be treated by the peer endpoint as having the default value and are not allowed to change subsequently. The peer should not request changes in such properties.

Those receiving an RDMA2_CONNPROP may encounter properties that they do not support or are unaware of. In such cases, these properties are simply ignored without any error response being generated.

5.3.2. RDMA2_REQPROP: Request Modification of Properties

The RDMA2_REQPROP message type allows an RPC-over-RDMA participant, whether client or server, to request of its partner that relevant transport properties be changed.

The rdma_xid field allows the request to be tied to a corresponding response of type RDMA2_RESPROP (See Section 5.3.3.) In assigning the value of this field, the sender does not need to avoid conflict with xid's associated with RPC messages or with RDMA2_REQPROP messages sent by the peer endpoint.

The partner need not change the properties as requested by the sender but if it does support the message type, it will generate a RDMA2_RESPROP message, indicating the disposition of the request.

<CODE BEGINS>

struct rpcrdma2_reqprop {
       rpcrdma2_propset rdma_want;
};        
 
<CODE ENDS>

          

The message definition for this operation is as follows:

The rpcrdma2_propset rdma_want is a set of transport properties together with the desired values requested by the sender.

5.3.3. RDMA2_RESPROP: Respond to Request to Modify Transport Properties

The RDMA2_RESPROP message type allows an RPC-over-RDMA participant to respond to a request to change properties by its partner, indicating how the request was dealt with.

<CODE BEGINS>

struct rpcrdma2_resprop {
        rpcrdma2_propsubset rdma_done;
        rpcrdma2_propsubset rdma_rejected;
        rpcrdma2_propset rdma_other;                        
};        
 
<CODE ENDS>

          

The message definition for this operation is as follows:

The rdma_xid field of this message must match that used in the RDMA2_REQPROP message to which this message is responding.

The rdma_done field indicates which of the requested transport property changes have been effected as requested. For each such property, the receiver is entitled to conclude that the requested change has been made and that future transmissions may be made based on the new value.

The rdma_rejected field indicates which of the requested transport property changes have been rejected by the sender. This may be because of any of the following reasons:

• The particular property specified is not known or supported by the receiver of the RDMA2_REQPROP message.
• The implementation receiving the RDMA2_REQPROP message does not support modification of this property.
• The implementation receiving the RDMA2_REQPROP message has chosen to reject the modification for another reason.

The rdma_other field contains new values for properties where a change is requested. The new value of the property is included and may be a value different from the original value in effect when the change was requested and from the requested value. This is useful when the new value of some property is not as large as requested but still different from the original value, indicating a partial satisfaction of the peer's property change request.

The sender MUST NOT include rpcrdma2_propval items within rdma_other that are for properties other than the ones for which the corresponding property request has requested a change. If the receiver finds such a situation, it MUST ignore the erroneous rpcrdma2_propval items.

The subsets of properties specified by rdma_done, rdma_rejected, and included in rdma_other MUST NOT overlap, and when ored together, should cover the entire set of properties specified by rdma_want in the corresponding request. If the receiver finds such an overlap or mismatch, it SHOULD treat properties missing or within the overlap as having been rejected.

5.3.4. RDMA2_UPDPROP: Update Transport Properties

The RDMA2_UPDPROP message type allows an RPC-over-RDMA participant to notify the other participant that a change to the transport properties has occurred. This is because the sender has decided, independently, to modify one or more transport properties and is notifying the receiver of these changes.

<CODE BEGINS>
 
struct rpcrdma2_updprop {
        rpcrdma2_propset rdma_now;
};        

<CODE ENDS>

          

The message definition for this operation is as follows:

rdma_now defines the new property values to be used.

5.4. Extensibility

5.4.1. Additional Properties

The set of transport properties is designed to be extensible. As a result, once new properties are defined in standards track documents, the operations defined in this document may reference these new transport properties, as well as the ones described in this document.

A standards track document defining a new transport property should include the following information paralleling that provided in this document for the transport properties defined herein.

• The rpcrdma2_propid value used to identify this property.
• The XDR typedef specifying the form in which the property value is communicated.
• A description of the transport property that is communicated by the sender of RDMA2_CONNPROP and RDMA2_UPDPROP and requested by the sender of RDMA2_REQPROP.
• An explanation of how this knowledge could be used by the participant receiving this information.
• Information giving rules governing possible changes of values of this property.

The definition of transport property structures is such as to make it easy to assign unique values. There is no requirement that a continuous set of values be used and implementations should not rely on all such values being small integers. A unique value should be selected when the defining document is first published as an internet draft. When the document becomes a standards track document working group should insure that:

• rpcrdma2_propid values specified in the document do not conflict with those currently assigned or in use by other pending working group documents defining transport properties.
• rpcrdma2_propid values specified in the document do not conflict with the range reserved for experimental use, as defined in Section 5.4.2.

Documents defining new properties fall into a number of categories.

• Those defining new properties and explaining (only) how they affect use of existing message types.
• Those defining new OPTIONAL message types and new properties applicable to the operation of those new message types.
• Those defining new OPTIONAL message types and new properties applicable both to new and existing message types.

When additional transport properties are proposed, the review of the associated standards track document should deal with possible security issues raised by those new transport properties.

5.4.2. Experimental Properties

Given the design of the transport properties data structure, it possible to use the operations to implement experimental, possibly unpublished, transport properties.

rpcrdma2_propid values in the range from 4,294,967,040 to 4,294,967,295 are reserved for experimental use and these values should not be assigned to new properties in standards track documents.

When values in this range are used there is no guarantee if successful interoperation among independent implementations.

6. XDR Protocol Definition

This section contains a description of the core features of the RPC-over-RDMA Version Two protocol, expressed in the XDR language [RFC4506].

This description is provided in a way that makes it simple to extract into ready-to-compile form. The reader can apply the following shell script to this document to produce a machine-readable XDR description of the RPC-over-RDMA Version One protocol without any OPTIONAL extensions.

<CODE BEGINS>

#!/bin/sh
grep '^ *///' | sed 's?^ /// ??' | sed 's?^ *///$??'

<CODE ENDS>

That is, if the above script is stored in a file called "extract.sh" and this document is in a file called "spec.txt" then the reader can do the following to extract an XDR description file:

<CODE BEGINS>

sh extract.sh < spec.txt > rpcrdma_corev2.x

<CODE ENDS>

Optional extensions to RPC-over-RDMA Version Two, published as Standards Track documents, will have similar means of providing XDR that describes those extensions. Once XDR for all desired extensions is also extracted, it can be appended to the XDR description file extracted from this document to produce a consolidated XDR description file reflecting all extensions selected for an RPC-over-RDMA implementation.

6.1. Code Component License

<CODE BEGINS>

/// /*
///  * Copyright (c) 2010, 2016 IETF Trust and the persons
///  * identified as authors of the code.  All rights reserved.
///  *
///  * The authors of the code are:
///  * B. Callaghan, T. Talpey, C. Lever, and D. Noveck.
///  *
///  * Redistribution and use in source and binary forms, with
///  * or without modification, are permitted provided that the
///  * following conditions are met:
///  *
///  * - Redistributions of source code must retain the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer.
///  *
///  * - Redistributions in binary form must reproduce the above
///  *   copyright notice, this list of conditions and the
///  *   following disclaimer in the documentation and/or other
///  *   materials provided with the distribution.
///  *
///  * - Neither the name of Internet Society, IETF or IETF
///  *   Trust, nor the names of specific contributors, may be
///  *   used to endorse or promote products derived from this
///  *   software without specific prior written permission.
///  *
///  *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS
///  *   AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
///  *   WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
///  *   IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
///  *   FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
///  *   EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
///  *   LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
///  *   EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
///  *   NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
///  *   SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
///  *   INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
///  *   LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
///  *   OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
///  *   IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
///  *   ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
///  */

<CODE ENDS>

Code components extracted from this document must include the following license text. When the extracted XDR code is combined with other complementary XDR code which itself has an identical license, only a single copy of the license text need be preserved.

6.2. RPC-Over-RDMA Version Two XDR

<CODE BEGINS>

/// /* From RFC 5531, Section 9 */
/// enum msg_type {
///         CALL = 0,
///         REPLY = 1
/// };
///
/// struct rpcrdma2_segment {
///         uint32 rdma_handle;
///         uint32 rdma_length;
///         uint64 rdma_offset;
/// };
///
/// struct rpcrdma2_read_segment {
///         uint32                  rdma_position;
///         struct rpcrdma2_segment rdma_target;
/// };
///
/// struct rpcrdma2_read_list {
///         struct rpcrdma2_read_segment rdma_entry;
///         struct rpcrdma2_read_list    *rdma_next;
/// };
///
/// struct rpcrdma2_write_chunk {
///         struct rpcrdma2_segment rdma_target<>;
/// };
///
/// struct rpcrdma2_write_list {
///         struct rpcrdma2_write_chunk rdma_entry;
///         struct rpcrdma2_write_list  *rdma_next;
/// };
///
/// struct rpcrdma2_chunk_lists {
///         enum msg_type               rdma_direction;
///         uint32                      rdma_inv_handle;
///         struct rpcrdma2_read_list   *rdma_reads;
///         struct rpcrdma2_write_list  *rdma_writes;
///         struct rpcrdma2_write_chunk *rdma_reply;
/// };
///
/// enum rpcrdma2_errcode {
///         RDMA2_ERR_VERS = 1,
///         RDMA2_ERR_BAD_XDR = 2,
///         RDMA2_ERR_REPLY_RESOURCE = 3,
///         RDMA2_ERR_INVAL_PROC = 4,
///         RDMA2_ERR_INVAL_OPTION = 5
///         RDMA2_ERR_SYSTEM = 6,
/// };
///
/// struct rpcrdma2_err_vers {
///         uint32 rdma_vers_low;
///         uint32 rdma_vers_high;
/// };
///
/// struct rpcrdma2_err_reply {
///         uint32 rdma_segment_index;
///         uint32 rdma_length_needed;
/// };
///
/// union rpcrdma2_error switch (rpcrdma2_errcode rdma_err) {
///         case RDMA2_ERR_VERS:
///           rpcrdma2_err_vers rdma_vrange;
///         case RDMA2_ERR_BAD_XDR:
///           void;
///         case RDMA2_ERR_REPLY_RESOURCE:
///           rpcrdma2_err_reply rdma_reply;
///         case RDMA2_ERR_INVAL_PROC:
///           void;
///         case RDMA2_ERR_INVAL_OPTION:
///           void;
///         case RDMA2_ERR_SYSTEM:
///           void;
/// };
///
/// struct rpcrdma2_optional {
///         enum msg_type rdma_optdir;
///         uint32 rdma_opttype;
///         opaque rdma_optinfo<>;
/// };
///
/// typedef rpcrdma2_propid uint32;
///
/// struct rpcrdma2_propval {
///         rpcrdma2_propid rdma_which;
///         opaque          rdma_data<>;
/// };
///
/// typedef rpcrdma2_propval rpcrdma2_propset<>;
/// typedef uint32 rpcrdma2_propsubset<>;
///
/// struct rpcrdma2_connprop {
///         rpcrdma2_propset rdma_start;
///         rpcrdma2_propsubset rdma_nochg;
/// };        
///
/// struct rpcrdma2_reqprop {
///        rpcrdma2_propset rdma_want;
/// };                
///
/// struct rpcrdma2_resprop {
///        rpcrdma2_propsubset rdma_done;
///        rpcrdma2_propsubset rdma_rejected;
///        rpcrdma2_propset rdma_other;                        
/// };                
///
/// struct rpcrdma2_updprop {
///        rpcrdma2_propset rdma_now;
/// };        

/// enum rpcrdma2_proc {
///         RDMA2_MSG = 0,
///         RDMA2_NOMSG = 1,
///         RDMA2_ERROR = 4,
///         RDMA2_OPTIONAL = 5,
///         RDMA2_CONNPROP = 6,
///         RDMA2_REQPROP = 7,
///         RDMA2_RESPROP = 8,
///         RDMA2_UPDPROP = 9
/// };
///
/// union rpcrdma2_body switch (rpcrdma2_proc rdma_proc) {
///         case RDMA2_MSG:
///           rpcrdma2_chunk_lists rdma_chunks;
///         case RDMA2_NOMSG:
///           rpcrdma2_chunk_lists rdma_chunks;
///         case RDMA2_ERROR:
///           rpcrdma2_error rdma_error;
///         case RDMA2_OPTIONAL:
///           rpcrdma2_optional rdma_optional;
///         case RDMA2_CONNPROP:
///           rpcrdma2_connprop rdma_connprop;
///         case RDMA2_REQPROP:
///           rpcrdma2_reqprop rdma_reqprop;
///         case RDMA2_RESPROP:
///           rpcrdma2_resprop rdma_resprop;
///         case RDMA2_UPDPROP:
///           rpcrdma2_updprop rdma_updprop;
/// };
///
/// struct rpcrdma2_xprt_hdr {
///           uint32 rdma_xid;
///           uint32 rdma_vers;
///           uint32 rdma_credit;
///           rpcrdma2_body rdma_body;
/// };
///
/// /*
///  * Transport propid values for basic properties
///  */
/// const uint32 RDMA2_PROPID_RBSIZ = 1;
/// const uint32 RDMA2_PROPID_BRS = 2;
///
/// /*
///  * Transport property typedefs
///  */
/// typedef uint32 rpcrdma2_prop_rbsiz;
/// typedef rpcrdma2_bkreqsup rpcrdma2_prop_brs;                
///
/// enum rpcrdma2_bkreqsup {
///         RDMA2_BKREQSUP_NONE = 0,
///         RDMA2_BKREQSUP_INLINE = 1,
///         RDMA2_BKREQSUP_GENL = 2
/// };

<CODE ENDS>

The XDR defined in this section is used to encode the Transport Header Stream in each RPC-over-RDMA Version Two message. The terms "Transport Header Stream" and "RPC Payload Stream" are defined in Section 4 of [I-D.ietf-nfsv4-rfc5666bis].

6.2.1. Presence Of Payload

• When the rdma_proc field has the value RDMA2_MSG, an RPC Payload Stream MUST follow the Transport Header Stream in the Send buffer.
• When the rdma_proc field has the value RDMA2_ERROR, an RPC Payload Stream MUST NOT follow the Transport Header Stream.
• When the rdma_proc field has the value RDMA2_OPTIONAL, all, part of, or no RPC Payload Stream MAY follow the Transport header Stream in the Send buffer.

6.2.2. Message Direction

Implementations of RPC-over-RDMA Version Two are REQUIRED to support backwards direction operation as described in [I-D.ietf-nfsv4-rpcrdma-bidirection]. RPC-over-RDMA Version Two introduces the rdma_direction field in its transport header to optimize the process of distinguishing between forward- and backwards-direction messages.

The rdma_direction field qualifies the value contained in the transport header's rdma_xid field. This enables a receiver to reliably avoid performing an XID lookup on incoming backwards-direction Call messages.

In general, when a message carries an XID that was generated by the message's receiver (that is, the receiver is acting as a requester), the message's sender sets the rdma_direction field to REPLY (1). Otherwise the rdma_direction field is set to CALL (0). For example:

• When the rdma_proc field has the value RDMA2_MSG or RDMA2_NOMSG, the value of the rdma_direction field MUST be the same as the value of the associated RPC message's msg_type field.
• When the rdma_proc field has the value RDMA2_OPTIONAL and a whole or partial RPC message payload is present, the value of the rdma_optdir field MUST be the same as the value of the associated RPC message's msg_type field.
• When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC message payload is present, a Requester MUST set the value of the rdma_optdir field to CALL, and a Responder MUST set the value of the rdma_optdir field to REPLY. The Requester chooses a value for the rdma_xid field from the XID space that matches the message's direction. Requesters and Responders set the rdma_credit field in a similar fashion: a value is set that is appropriate for the direction of the message.
• When the rdma_proc field has the value RDMA2_ERROR, the direction of the message is always Responder-to-Requester (REPLY).

6.2.3. Remote Invalidation

To request Remote Invalidation, a requester MUST set the value of the rdma_inv_handle field in an RPC Call's transport header to a non-zero value that matches one of the rdma_handle fields in that header. If none of the rdma_handle values in the Call may be invalidated by the responder, the requester MUST set the RPC Call's rdma_inv_handle field to the value zero.

If the responder chooses not to use Remote Invalidation for this particular RPC Reply, or the RPC Call's rdma_inv_handle field contains the value zero, the responder MUST use RDMA Send to transmit the matching RPC reply.

If a requester has provided a non-zero value in the RPC Call's rdma_inv_handle field and the responder chooses to use Remote Invalidation for the matching RPC Reply, the responder MUST use RDMA Send With Invalidate to transmit that RPC reply, and MUST use the value in the RPC Call's rdma_inv_handle field to construct the Send With Invalidate Work Request.

6.2.4. Transport Errors

Error handling works the same way in RPC-over-RDMA Version Two as it does in RPC-over-RDMA Version One, with the addition of several new error codes. Version One error handling is described in Section 5 of [I-D.ietf-nfsv4-rfc5666bis].

In all cases below, the sender copies the values of the rdma_xid and rdma_vers fields from the incoming transport header that generated the error to transport header of the error response. The rdma_proc field is set to RDMA2_ERROR.

RDMA2_ERR_VERS

This is the equivalent of ERR_VERS in RPC-over-RDMA Version One. The error code value, semantics, and utilization are the same.

RDMA2_ERR_INVAL_PROC

This is a new error code in RPC-over-RDMA Version Two. If a receiver recognizes the value in the rdma_vers field, but it does not recognize the value in the rdma_proc field, it MUST send RDMA2_ERR_INVAL_PROC.

RDMA2_ERR_BAD_XDR

This is the equivalent of ERR_CHUNK in RPC-over-RDMA Version One, with a few extra restrictions. The error code value is the same.

If a receiver recognizes the values in the rdma_vers and rdma_proc fields, but the incoming RPC-over-RDMA transport header cannot be parsed, the receiver MUST send RDMA2_ERR_BAD_XDR before Upper Layer Protocol processing starts.

RDMA2_ERR_REPLY_RESOURCE

This is a new error code in RPC-over-RDMA Version Two. If the RPC-over-RDMA transport header is otherwise correct, but the requester has not provided enough Write or Reply chunk resources to transmit the reply, the responder MUST send RDMA2_ERR_REPLY_RESOURCE.

The responder MUST set the rdma_segment_index field to point to the first segment in the transport header that is too short, or to zero to indicate that it was not possible to determine which segment was too small. Indexing starts at one (1), which represents the first segment in the first Write chunk (in either the Write list or Reply chunk). The responder MUST set the rdma_length_needed to the number of bytes needed in that segment in order to convey the reply.

Upon receipt of this error code, a responder MAY choose to terminate the operation (for instance, if the responder set the index and length fields to zero), or it MAY send the request again using the same XID and larger reply resources.

RDMA2_ERR_INVAL_OPTION

This is a new error code in RPC-over-RDMA Version Two. A receiver MUST send RDMA2_ERR_INVAL_OPTION when an RDMA2_OPTIONAL message is received and the receiver does not recognize the value in the rdma_opttype field.

RDMA2_ERR_SYSTEM

This is a new error code in RPC-over-RDMA Version Two. If some problem occurs on a receiver that does not fit into the above categories, the receiver MAY report it to the sender using the error code RDMA2_ERR_SYSTEM. This is a permanent error: a requester that receives this error MUST terminate the RPC transaction associated with the XID value in the rdma_xid field.

7. Protocol Version Negotiation

When an RPC-over-RDMA Version Two client establishes a connection to a server, the first order of business is to determine the server's highest supported protocol version.

As with RPC-over-RDMA Version One, a client MUST assume the ability to exchange only a single RPC-over-RDMA message at a time until it receives a valid non-error RPC-over-RDMA message from the server that reports the server's credit limit.

First, the client sends a single valid RPC-over-RDMA message with the value two (2) in the rdma_vers field. Because the server might support only RPC-over-RDMA Version One, this initial message can be no larger than the Version One default inline threshold of 1024 bytes.

7.1. Server Does Support RPC-over-RDMA Version Two

If the server does support RPC-over-RDMA Version Two, it sends RPC-over-RDMA messages back to the client with the value two (2) in the rdma_vers field. Both peers may use the default inline threshold value for RPC-over-RDMA Version Two connections (4096 bytes).

7.2. Server Does Not Support RPC-over-RDMA Version Two

If the server does not support RPC-over-RDMA Version Two, it MUST send an RPC-over-RDMA message to the client with the same XID, with RDMA2_ERROR in the rdma_proc field, and with the error code RDMA2_ERR_VERS. This message also reports a range of protocol versions that the server supports. To continue operation, the client selects a protocol version in the range of server-supported versions for subsequent messages on this connection.

If the connection is lost immediately after an RDMA2_ERROR / RDMA2_ERR_VERS message is received, a client can avoid a possible version negotiation loop when re-establishing another connection by assuming that particular server does not support RPC-over-RDMA Version Two. A client can assume the same situation (no server support for RPC-over-RDMA Version Two) if the initial negotiation message is lost or dropped. Once the negotiation exchange is complete, both peers may use the default inline threshold value for the transport protocol version that has been selected.

7.3. Client Does Not Support RPC-over-RDMA Version Two

If the server supports the RPC-over-RDMA protocol version used in Call messages from a client, it MUST send Replies with the same RPC-over-RDMA protocol version that the client uses to send its Calls.

7.4. Security Considerations

The security considerations for RPC-over-RDMA Version Two are the same as those for RPC-over-RDMA Version One.

7.4.1. Security Considerations (Transport Properties)

Like other fields that appear in each RPC-over-RDMA header, property information is sent in the clear on the fabric with no integrity protection, making it vulnerable to man-in-the-middle attacks.

For example, if a man-in-the-middle were to change the value of the Receive buffer size or the Requester Remote Invalidation boolean, it could reduce connection performance or trigger loss of connection. Repeated connection loss can impact performance or even prevent a new connection from being established. Recourse is to deploy on a private network or use link-layer encryption.

8. IANA Considerations

There are no IANA considerations at this time.

9. References

9.1. Normative References

[RFC2119]	Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997.
[RFC4506]	Eisler, M., "XDR: External Data Representation Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 2006.
[RFC5531]	Thurlow, R., "RPC: Remote Procedure Call Protocol Specification Version 2", RFC 5531, DOI 10.17487/RFC5531, May 2009.

9.2. Informative References

[I-D.ietf-nfsv4-rfc5666bis]	Lever, C., Simpson, W. and T. Talpey, "Remote Direct Memory Access Transport for Remote Procedure Call, Version One", Internet-Draft draft-ietf-nfsv4-rfc5666bis-08, November 2016.
[I-D.ietf-nfsv4-rpcrdma-bidirection]	Lever, C., "Bi-directional Remote Procedure Call On RPC-over-RDMA Transports", Internet-Draft draft-ietf-nfsv4-rpcrdma-bidirection-05, June 2016.
[IB]	InfiniBand Trade Association, "InfiniBand Architecture Specifications"
[RFC5040]	Recio, R., Metzler, B., Culley, P., Hilland, J. and D. Garcia, "A Remote Direct Memory Access Protocol Specification", RFC 5040, DOI 10.17487/RFC5040, October 2007.
[RFC5041]	Shah, H., Pinkerton, J., Recio, R. and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, DOI 10.17487/RFC5041, October 2007.
[RFC5661]	Shepler, S., Eisler, M. and D. Noveck, "Network File System (NFS) Version 4 Minor Version 1 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010.
[RFC5662]	Shepler, S., Eisler, M. and D. Noveck, "Network File System (NFS) Version 4 Minor Version 1 External Data Representation Standard (XDR) Description", RFC 5662, DOI 10.17487/RFC5662, January 2010.
[RFC5666]	Talpey, T. and B. Callaghan, "Remote Direct Memory Access Transport for Remote Procedure Call", RFC 5666, DOI 10.17487/RFC5666, January 2010.

Appendix A. Acknowledgments

The authors gratefully acknowledge the work of Brent Callaghan and Tom Talpey on the original RPC-over-RDMA Version One specification [RFC5666]. The authors also wish to thank Bill Baker, Greg Marsden, and Matt Benjamin for their support of this work.

The extract.sh shell script and formatting conventions were first described by the authors of the NFSv4.1 XDR specification [RFC5662].

Special thanks go to nfsv4 Working Group Chair Spencer Shepler and nfsv4 Working Group Secretary Thomas Haynes for their support.

Authors' Addresses