Internet DRAFT - draft-mzhang-nfsv4-sequence-id-calibration
draft-mzhang-nfsv4-sequence-id-calibration
Network File System Version 4 M. Zhang, Ed.
Internet-Draft J. Yang, Ed.
Intended status: Standards Track C. Tangudu, Ed.
Expires: 23 October 2023 K. Parambattu, Ed.
Huawei Technologies Co.
21 April 2023
Sequence ID calibration for mis-ordered requests
draft-mzhang-nfsv4-sequence-id-calibration-03
Abstract
This document updates RFC7862, Network File System (NFS) version 4
minor version 2, by adding two operations to prevent the client from
destroying session when getting the reply of a mis-ordered request
with NFS4ERR_SEQ_MISORDERED.
In NFSv4 minor version 1, sequence ID is used to ensure that the size
of the needed reply cache is tightly bounded. If the server gets a
mis-ordered request, the client will often break the session and
establish a new session with the server. This approach results in a
significant burden on the client and the server. During the process
of session rebuilding, IO performance will be affected. This is
especially troublesome when network latency is substantial, as, for
example when client and server are in different locations. This
document will propose extensions to NFSv4 that would allow client
reconnection to be dispensed with.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Zhang, et al. Expires 23 October 2023 [Page 1]
Internet-Draft Sequence ID calibration April 2023
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 23 October 2023.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Operations for Seqence ID calibration . . . . . . . . . . . . 4
2.1. Operation 76: SEQENCE_QUERY-Query sequence ID of designated
session and slot for calibration . . . . . . . . . . . . 4
2.1.1. ARGUMENTS . . . . . . . . . . . . . . . . . . . . . . 4
2.1.2. RESULTS . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.3. DESCRIPTION . . . . . . . . . . . . . . . . . . . . . 5
2.1.4. IMPLEMENTATION . . . . . . . . . . . . . . . . . . . 5
2.2. Operation 16:CB_SEQUENCE_QUERY- Query backchannel sequence
ID of designated session and slot for calibration . . . . 6
2.2.1. ARGUMENT . . . . . . . . . . . . . . . . . . . . . . 6
2.2.2. RESULT . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.3. DESCRIPTION . . . . . . . . . . . . . . . . . . . . . 6
2.2.4. IMPLEMENTATION . . . . . . . . . . . . . . . . . . . 7
3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
4. Extraction of XDR . . . . . . . . . . . . . . . . . . . . . . 8
5. Security Considerations . . . . . . . . . . . . . . . . . . . 8
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 8
7. Normative References . . . . . . . . . . . . . . . . . . . . 9
Appendix A. An Appendix . . . . . . . . . . . . . . . . . . . . 9
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9
Zhang, et al. Expires 23 October 2023 [Page 2]
Internet-Draft Sequence ID calibration April 2023
1. Introduction
Using the process detailed in [RFC8178], the revisions in this
document become an extension of NFSv4.2 [RFC7862]. They are built on
top of the external data representation (XDR) [RFC4506] generated
from [RFC7863].
In NFSv4 minor version 1, according to [RFC8881], Error code
NFS4ERR_SEQ_MISORDERED is returned by three operations.
* The first operation is CREATE_SESSION. csa_sequence is one
argument of this operation, which is used for serializing
CREATE_SESSION via a per-client ID sequence number by the client.
In CREATE_SESSION request, csa_sequence should be equal to ether
the sequence ID in the client ID's slot(a retry), or the slot's
sequence ID + 1(correct normal request). Otherwise,
NFS4ERR_SEQ_MISORDERED will be returned from the server.
* The second operation is SEQUENCE. sa_sequenceid is one argument of
this operation. In SEQUENCE request, If the difference between
sa_sequenceid and the server's cached sequence ID at the slot ID
is two (2) or more, or if sa_sequenceid is less than the cached
sequence ID (accounting for wraparound of the unsigned sequence ID
value), then the server MUST return NFS4ERR_SEQ_MISORDERED.
* The third operation is CB_ SEQUENCE, which is similar to SEQUENCE.
csa_sequenceid is one argument of this operation. In CB_ SEQUENCE
request, If the difference between csa_sequenceid and the client's
cached sequence ID at the slot ID is two (2) or more, or if
csa_sequenceid is less than the cached sequence ID (accounting for
wraparound of the unsigned sequence ID value), then the client
MUST return NFS4ERR_SEQ_MISORDERED.
Mis-order requests may happen as a result of network partition,
software bug, etc. For such request, the operations subsequent to
SEQUENCE, if any, are not processed, and so slots state (sequence ID,
cached reply) are not changed. That means, requests before this mis-
ordered one were processed correctly and the session state was
correct. In the current implementation, for most of the clients,
this error code will trigger the requester breaking the session and
creating a new session. This process unacceptably interferes with
ongoing IO operations, especially for the IOs on the normal slots.
For example of a persistent session, there are several slots on a
session. Requests on the slots are being processed correctly and
replies are being received normally. Suppose on one slot of them, a
mis-ordered request is received by the server and a response with
NFS4ERR_SEQ_MISORDERED error returns to the client. Then, the client
Zhang, et al. Expires 23 October 2023 [Page 3]
Internet-Draft Sequence ID calibration April 2023
is going to destroy the session and establish a new session. Before
the new session is ready, new requests will not be performed until
the pending operations finished. The effects on IOs of normal slots
will become dramatic especially in network latency is substantial,
as, for example when client and server are in different locations.
The client has to break the session because it does not know what
sequence is expected for that session and slot. This current cached
sequence information would be available to the client eliminating any
need to break the session.
Two operations, SEQENCE_QUERY and CB_SEQENCE_QUERY, are added to
query sequence ID cached when getting NFS4ERR_SEQ_MISORDERED error.
2. Operations for Seqence ID calibration
2.1. Operation 76: SEQENCE_QUERY-Query sequence ID of designated
session and slot for calibration
2.1.1. ARGUMENTS
<CODE BEGINS>
///
/// /*
/// * structure for sequenceid query
/// */
/// struct SEQUENCE_QUERY4args {
/// sessionid4 sqa_sessionid;
/// slotid4 sqa_slotid;
/// };
///
<CODE ENDS>
2.1.2. RESULTS
<CODE BEGINS>
///
/// struct SEQUENCE_QUERY4resok {
/// sessionid4 sqr_sessionid;
/// slotid4 sqr_slotid;
/// sequenceid4 sqr_sequenceid;
/// };
/// union SEQUENCE_QUERY4res switch (nfsstat4 sqr_status) {
/// case NFS4_OK: SEQUENCE_QUERY4resok sqr_resok4;
/// default: void;
/// };
///
<CODE ENDS>
Zhang, et al. Expires 23 October 2023 [Page 4]
Internet-Draft Sequence ID calibration April 2023
2.1.3. DESCRIPTION
The SEQUENCE_QUERY operation is used by the client to query sequence
ID cached for designated session and slot.
SEQUENCE_QUERY MUST appear as the sole operation type of any COMPOUND
in which it appears. Multiple SEQUENCE_QUERY operations can be used
in one COMPOUND to query multiple sequence IDs cached for multiple
slots. The error NFS4ERR_NOT_ONLY_OP will be returned when that
constraint is not met. Operations other than SEQUENCE,
SEQENCE_QUERY, BIND_CONN_TO_SESSION, EXCHANGE_ID, CREATE_SESSION, and
DESTROY_SESSION, MUST NOT appear as the first operation in a
COMPOUND.
If SEQUENCE_QUERY is received on a connection not associated with the
session via CREATE_SESSION or BIND_CONN_TO_SESSION, and connection
association enforcement is enabled (see Section 18.35), then the
server returns NFS4ERR_CONN_NOT_BOUND_TO_SESSION.
The sqa_sessionid argument identifies the session to which this
request applies. The sqr_sessionid result MUST equal sqa_sessionid.
The sqa_slotid argument is the index in the reply cache for the
request. The sqr_slotid result MUST equal sqa_slotid.
The sqr_sequenceid field is the cached sequence ID on the slot. The
client SHOULD use this value to calibrate sa_sequenceid in the next
SEQUENCE operation, that is, sqr_sequenceid+1 SHOULD be used as the
sequence ID of the next request on this slot.
2.1.4. IMPLEMENTATION
For CREATE_SESSION, SEQUENCE operations, if the sequence ID in the
request is mis-ordered(see [RFC8881] 18.46.3 Section), the replier
will fail the request by NFS4ERR_SEQ_MISORDERED and keep the reply
cache unchanged on the slot of this session. When getting
NFS4ERR_SEQ_MISORDERED error code in the response, the client SHOULD
query the cached sequence ID of the slot and session by
SEQUENCE_QUERY to calibrate its sequence ID for the subsequent
requests. That is, the sequence ID in next request on this slot
SHOULD be sqr_sequenceid+1.
SEQUENCE_QUERY will leave the state of the slot (sequence ID, cached
reply) unchanged and lease of state related to the client ID not
renewed.
If the client is querying an unknown session ID to the server, the
server SHOULD return NFS4ERR_BADSESSION in the response.
Zhang, et al. Expires 23 October 2023 [Page 5]
Internet-Draft Sequence ID calibration April 2023
If the client is attempting to access a slot the replier does not
have in its slot table (It is possible the slot may have been
retired), NFS4ERR_BADSLOT SHOULD be returned in the response.
2.2. Operation 16:CB_SEQUENCE_QUERY- Query backchannel sequence ID of
designated session and slot for calibration
2.2.1. ARGUMENT
<CODE BEGINS>
///
/// /*
/// * callback program structure for sequenceid query
/// */
/// struct CB_SEQUENCE_QUERY4args {
/// sessionid4 csqa_sessionid;
/// slotid4 csqa_slotid;
/// };
///
<CODE ENDS>
2.2.2. RESULT
<CODE BEGINS>
///
/// struct CB_SEQUENCE_QUERY4resok {
/// sessionid4 csqr_sessionid;
/// slotid4 csqr_slotid;
/// sequenceid4 csqr_sequenceid;
/// };
/// union CB_SEQUENCE_QUERY4res switch (nfsstat4 csqr_status){
/// case NFS4_OK: CB_SEQUENCE_QUERY4resok csqr_resok4;
/// default: void;
/// };
///
<CODE ENDS>
2.2.3. DESCRIPTION
CB_SEQUENCE_QUERY is used to calibrate sequence ID of the call back
request of the server for the backchannel of the session.
For each CB_COMPOUND request, the first operation MUST be CB_SEQUENCE
or CB_SEQUENCE_QUERY. If any other operation is in the first
position of CB_COMPOUND except CB_SEQUENCE_QUERY and CB_SEQUENCE,
NFS4ERR_OP_NOT_IN_SESSION MUST be returned.
Zhang, et al. Expires 23 October 2023 [Page 6]
Internet-Draft Sequence ID calibration April 2023
If the first operation is CB_SEQUENCE, CB_SEQUENCE MUST appear once.
The error NFS4ERR_SEQUENCE_POS MUST be returned when CB_SEQUENCE is
found in any position in a CB_COMPOUND beyond the first. If the
first operation of a CB_COMPOUND is CB_SEQUENCE_QUERY,
CB_SEQUENCE_QUERY MUST be the sole operation type. There can be
multiple CB_SEQUENCE_QUERY in this CB_COMPOUND to request multiple
cached sequence IDs of designated sessions and slots. If any other
operations are found in this CB_COMPOUND, NFS4ERR_NOT_ONLY_OP MUST be
returned.
The csqa_sessionid argument identifies the session to which this
request applies. The csqr_sessionid result MUST equal
csqa_sessionid.
The csqa_slotid argument is the index in the reply cache for the
request. The csqr_slotid result MUST equal sqa_slotid.
The csqr_sequenceid field is the cached sequence ID on the slot. The
server SHOULD use this value to calibrate csa_sequenceid in the next
SEQUENCE operation, that is, csqr_sequenceid+1 SHOULD be used as the
sequence ID of the next request on this slot.
2.2.4. IMPLEMENTATION
For CB_SEQUENCE operations, if the sequence ID in the call back
request is mis-ordered(see [RFC8881] 20.9 Section), the client will
fail this request by NFS4ERR_SEQ_MISORDERED and keep the reply cache
unchanged on the slot of this session. When getting
NFS4ERR_SEQ_MISORDERED error code in the response, the server SHOULD
query the cached sequence ID of the slot and session by
CB_SEQUENCE_QUERY to calibrate its sequence ID for the subsequent
requests. That is, the sequence ID in next call back request on this
slot should be csqr_sequenceid+1. CB_SEQUENCE_QUERY will leave the
state of the slot (sequence ID, cached reply) unchanged and the reply
of CB_SEQUENCE_QUERY will not renew the lease of state related to the
client ID on the server side.
If the server is querying an unknown session ID to the client, the
client SHOULD return NFS4ERR_BADSESSION in the response.
If the server is attempting to access a slot the client does not have
in its slot table (It is possible the slot may have been retired),
NFS4ERR_BADSLOT SHOULD be returned in the response.
3. IANA Considerations
This memo includes no request to IANA.
Zhang, et al. Expires 23 October 2023 [Page 7]
Internet-Draft Sequence ID calibration April 2023
Note to RFC Editor: this section may be removed on publication as an
RFC.
4. Extraction of XDR
This document contains the external data representation (XDR)
[RFC4506] description of the new open flags for delegating the file
to the client. The XDR description is embedded in this document in a
way that makes it simple for the reader to extract into a ready-to-
compile form. The reader can feed this document into the following
shell script to produce the machine readable XDR description of the
new flags:
<CODE BEGINS>
#!/bin/sh grep '^ *///' $* |
sed 's?^ */// ??' | sed 's?^ *///$??'
<CODE ENDS>
That is, if the above script is stored in a file called "extract.sh",
and this document is in a file called "spec.txt", then the reader can
do:
<CODE BEGINS>
sh extract.sh < spec.txt
> seqid_calibration.x
<CODE ENDS>
The effect of the script is to remove leading white space from each
line, plus a sentinel sequence of "///". XDR descriptions with the
sentinel sequence are embedded throughout the document.
Note that the XDR code contained in this document depends on types
from the NFSv4.2 nfs4_prot.x file (generated from [RFC7863]). This
includes both nfs types that end with a 4, such as offset4, length4,
etc., as well as more generic types such as uint32_t and uint64_t.
While the XDR can be appended to that from [RFC7863], the various
code snippets belong in their respective areas of the that XDR.
5. Security Considerations
There are no new security considerations beyond those in [RFC7862].
6. Acknowledgements
The authors would like to acknowledge David Noveck, Thomas Haynes,
Rick Macklem, Tom Talpey for reviews of the various versions of the
draft.
Zhang, et al. Expires 23 October 2023 [Page 8]
Internet-Draft Sequence ID calibration April 2023
7. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC4506] Eisler, M., Ed., "XDR: External Data Representation
Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May
2006, <https://www.rfc-editor.org/info/rfc4506>.
[RFC7862] Haynes, T., "Network File System (NFS) Version 4 Minor
Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862,
November 2016, <https://www.rfc-editor.org/info/rfc7862>.
[RFC7863] Haynes, T., "Network File System (NFS) Version 4 Minor
Version 2 External Data Representation Standard (XDR)
Description", RFC 7863, DOI 10.17487/RFC7863, November
2016, <https://www.rfc-editor.org/info/rfc7863>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8178] Noveck, D., "Rules for NFSv4 Extensions and Minor
Versions", RFC 8178, DOI 10.17487/RFC8178, July 2017,
<https://www.rfc-editor.org/info/rfc8178>.
[RFC8881] Noveck, D., Ed. and C. Lever, "Network File System (NFS)
Version 4 Minor Version 1 Protocol", RFC 8881,
DOI 10.17487/RFC8881, August 2020,
<https://www.rfc-editor.org/info/rfc8881>.
Appendix A. An Appendix
Authors' Addresses
Zhang Mingqian (editor)
Huawei Technologies Co.
China
Email: zhangmingqian.zhang@huawei.com
Yang Jing (editor)
Huawei Technologies Co.
China
Email: yangjing8@huawei.com
Zhang, et al. Expires 23 October 2023 [Page 9]
Internet-Draft Sequence ID calibration April 2023
Sai Chakravarthy Tangudu (editor)
Huawei Technologies Co.
India
Email: sai.chakravarthy.tangudu@huawei.com
Rijesh Kunhi Parambattu (editor)
Huawei Technologies Co.
India
Email: rijesh.kunhi.parambattu@huawei.com
Zhang, et al. Expires 23 October 2023 [Page 10]
