Internet DRAFT - draft-dnoveck-nfs-extension
draft-dnoveck-nfs-extension
NFSv4 D. Noveck
Internet-Draft HPE
Intended status: Informational March 10, 2016
Expires: September 11, 2016
NFS Protocol Extension: Retrospect and Prospect
draft-dnoveck-nfs-extension-04
Abstract
This document surveys the processes by which the NFS protocols have
been extended in the past, including all of the NFS major and minor
versions. It also looks forward to methods of protocol extension
that might be used by NFS in the future.
A particular focus is on how the minor versioning model of NFSv4 has
worked and what is being done to enhance version management within
NFSv4. The work already done in the new NFSv4 version management
document is summarized, and there is a discussion of further issues
the working group will need to address in moving that work forward.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 11, 2016.
Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
Noveck Expires September 11, 2016 [Page 1]
Internet-Draft nfs-extension March 2016
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Use of Key Words Defined in RFC2119 . . . . . . . . . . . 3
2. Protocol Extension . . . . . . . . . . . . . . . . . . . . . 5
3. Protocol Extension Mechanisms . . . . . . . . . . . . . . . . 5
3.1. Specific Protocol Mechanisms Designed for Extension . . . 6
3.2. Protocol Extension by XDR Replacement . . . . . . . . . . 6
3.3. Protocol Extension by XDR Extension . . . . . . . . . . . 7
3.4. Combination of Protocol Extension Mechanisms . . . . . . 8
3.5. Switching Extension Mechanisms . . . . . . . . . . . . . 8
4. Pre-IETF NFS Versioning . . . . . . . . . . . . . . . . . . . 9
4.1. The Pre-IETF NFS Environment . . . . . . . . . . . . . . 9
4.2. Transition from NFSv2 to NFSv3 . . . . . . . . . . . . . 9
5. Initial Work on NFSv4 Within IETF . . . . . . . . . . . . . . 10
5.1. Transition from NFSv3 to NFSv4.0 . . . . . . . . . . . . 10
5.2. NFSv4 and XDR Extensibility . . . . . . . . . . . . . . . 12
5.3. Initial Minor Versioning Model for NFSv4 . . . . . . . . 13
5.4. Goals of Minor Versioning for NFSv4 . . . . . . . . . . . 15
6. Use of Minor Versioning in NFSv4 . . . . . . . . . . . . . . 16
6.1. Transition from NFSv4.0 to NFSv4.1 . . . . . . . . . . . 16
6.2. Transition from NFSv4.1 to NFSv4.2 . . . . . . . . . . . 22
6.3. Evolution of Minor Versioning Model within NFSv4 . . . . 23
6.4. Review of NFSv4 Versioning through NFSv4.2 . . . . . . . 24
7. Inherited NFSv4 Versioning Approach . . . . . . . . . . . . . 25
7.1. Non-XDR-based Changes . . . . . . . . . . . . . . . . . . 25
7.2. The Role of Minor Version Number in NFSv4 . . . . . . . . 25
7.3. Inherited NFS Versioning Practices . . . . . . . . . . . 27
7.4. Problems with Inherited NFS Versioning Approach . . . . . 27
8. Formulating a New NFSv4 Extension Approach . . . . . . . . . 30
9. Current Versioning-related Work . . . . . . . . . . . . . . . 31
9.1. Creation of New NFSv4 Version Management Document . . . . 31
9.2. Related Changes to Other Documents . . . . . . . . . . . 31
9.3. Further work on New NFSv4 Versioning Document . . . . . . 32
9.4. Issues Needing further discussion . . . . . . . . . . . . 34
10. Looking Back . . . . . . . . . . . . . . . . . . . . . . . . 35
11. Looking Forward . . . . . . . . . . . . . . . . . . . . . . . 38
12. Security Considerations . . . . . . . . . . . . . . . . . . . 39
13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 39
14. References . . . . . . . . . . . . . . . . . . . . . . . . . 39
14.1. Normative References . . . . . . . . . . . . . . . . . . 39
14.2. Informative References . . . . . . . . . . . . . . . . . 39
Noveck Expires September 11, 2016 [Page 2]
Internet-Draft nfs-extension March 2016
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 41
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 41
1. Introduction
This document examines the subject of protocol extension within the
NFS family of protocols. In order to better understand the issues
that exist going forward with NFSv4, we examine the history of
protocol extension throughout the development of NFS including
o The pre-IETF period
o The development of successive NFSv4 minor versions, under minor
versioning rules originally defined in [RFC3530] and subsequently
modified based on the working group's then-current needs.
o The consolidation of version management rules in [NFSv4-vers]
along with the other changes made in that document to make NFSv4
version management more flexible.
With this history in mind, we discuss the issues that the working
group needs to address in order to define and adopt a consistent and
comprehensive approach to version management for the NFSv4 protocols.
1.1. Use of Key Words Defined in RFC2119
It is intended that these key words be used in this document in a way
that is consistent with their definition in [RFC2119].
However, because of the considerations noted below, there are some
issues that readers need to be aware of.
It is not altogether clear whether these keywords are to be limited
to requirements regarding protocol implementations, or whether they
can reasonably be used in specifying guidance directed at future
potential RFCs. In many cases, these terms are defined in ways that
strongly suggest that they are meant to apply to implementations and
much of the discussion seems to make sense only in that context. On
the other hand, there is no explicit statement to that effect.
As an example of the difficulties that can result in trying to
resolve this question, consider the statement in [RFC2119] saying
that these imperatives "MUST only be used where it is actually
required for interoperation or to limit behavior which has potential
for causing harm (e.g., limiting retransmissions)." On the one hand,
this is a case in which these terms are directed at a future
document, rather than implementations. However, it seems as if this
use is itself not consistent with what it requires of others. While
Noveck Expires September 11, 2016 [Page 3]
Internet-Draft nfs-extension March 2016
the misuse of these terms is undesirable and it is best to avoid such
misuse, it is hard to see exactly how that is "actually required for
interoperation or to limit behavior which has potential for causing
harm."
The nature of this document is such that it does not directly specify
implementation requirements. In some cases it discusses potential
requirements that an RFC derived from [NFSv4-vers]) might direct to
implementations. In other cases, requirements are one step farther
removed from protocol implementations. For example, the version
management RFC might specify the kinds of requirements that future
minor version definition documents and feature definition documents
might specify.
The documents that are discussed herein, have at times used these key
words in a way that seems to be at variance with the definitions in
[RFC2119]. For details, see Sections 6.1 and 9.2.
The reader should be aware of our practices in this document
regarding these terms. We only apply these upper-case key words to
directions regarding protocol implementations, rather than to
guidance intended for those writing RFCs. While it is possible that
there could be cases in which directions targeted at future RFC's are
necessary to assure interoperation etc., we know of no actual
instances.
In this document, these keywords will often appear as quotations from
existing documents, either direct or indirect. Readers should be
aware that it is possible that some such uses might not be in accord
with [RFC2119].
In other cases, these keywords will be in the context of proposals/
suggestions of what future RFCs could or might say regarding certain
issues.
The use of these terms as part of minor versioning rules poses some
troublesome issues in the context of this document. These rules have
appeared in [RFC3010], [RFC3530], [RFC5661], and [NFSv42]. They have
also appeared in various drafts of [NFSv4-vers], and in various
drafts leading up to [RFC7530]. In presenting these rules, there
have been multiple switches between the RFC2119 keywords and
corresponding lower-case terms.
Use of the terms "OPTIONAL" and "REQUIRED" as feature statuses would
conform to our general practice since a feature status is essentially
a way in which a minor version specification RFC might REQUIRE (or
not) implementations to support a given feature. Unfortunately,
"RECOMMENDED" would not fit that model very well. Also, it is
Noveck Expires September 11, 2016 [Page 4]
Internet-Draft nfs-extension March 2016
difficult to see how the same feature can be "OPTIONAL" in one minor
version and "REQUIRED" in another given the meanings that [RFC2119]
assigns to these terms.
In this document, in the interests of uniformity, we will use lower-
case terms for feature statuses, except when we are referring to what
is said in a particular document which used the upper-case keywords.
2. Protocol Extension
Protocols often require means by which they can be extended. Such
extensions may be needed to meet new requirements, to correct
protocol weaknesses exposed by experience, or even to correct
protocol bugs (as becomes more probable when protocols are published
as RFC's without fully fleshed-out implementations).
We need to distinguish here between protocol "extension" and
"versioning". Versioning is a form of protocol extension but not
every form of protocol extension can be accommodated within a
versioning paradigm.
When a versioning paradigm is in place, groups of extensions are
conceived of as ordered, allowing extensions in subsequent versions
to build upon those in previous versions. When multiple extensions
are combined into a single version, each of the extensions may be
built assuming that the others will be present as well. In such
cases, there can be the opportunity to make design changes in the
protocol, allowing elements of the protocol to be restructured,
sometimes in major ways.
When a versioning paradigm is in effect and extensions are optional,
extensions cannot build upon one another, since the presence of any
particular extension cannot be assumed. In such cases, the ability
to restructure the protocol is reduced, but smaller changes may be
introduced more easily.
In this latter case, it is not clear that the word "versioning" is
appropriate. Nevertheless, in this document, we will, as in the
phrase "NFSv4 minor versioning," use the existing terminology without
necessarily subscribing to the view that "versioning" is the
appropriate description.
3. Protocol Extension Mechanisms
Some factors that are often relevant in deciding on the means by
which a protocol will be extended:
Noveck Expires September 11, 2016 [Page 5]
Internet-Draft nfs-extension March 2016
o Whether extensions are to be individually selectable (i.e.
optional) or assumed to be always present, allowing one to build
upon another earlier one.
o The size and scope of extensions that will be made.
o Compatibility issues with existing implementations.
o Issues that relate to ensuring that when individual extensions,
separately arrived at, are each optionally allowed, the extensions
used are compatible and can be used effectively together.
o The overall implementation framework. For example, RPC-based
protocols may do extension by means of the RPC version mechanism.
Because NFS is layered on RPC and is described using an XDR
description, the presentation of extension mechanisms below reflects
those facts. Although XDR is mentioned in Sections 3.2 and 3.3 the
reader should not assume that particular aspects of XDR itself are
critical to the discussion. Similar extension approaches and
analogous considerations would apply to other similar methods of
protocol description.
3.1. Specific Protocol Mechanisms Designed for Extension
Often, protocols will be designed with specific mechanisms, designed
to allow protocol extension. An example is the provision for TCP
options (see [RFC0793] and [RFC2780].) Most often, such mechanisms
are designed to allow individual extensions to be designed and
implemented independently, with any dependency relations between
extensions specified separately and not enforced by the extension
mechanism itself.
3.2. Protocol Extension by XDR Replacement
RPC-based protocols may, and often do, provide for protocol extension
by replacing the XDR for one version with that for another. In this
case the new protocol version could be represented by a new RPC
protocol number but the more common pattern is to use the RPC
versioning mechanism to manage selection of the proper protocol
variant. The use of the RPC versioning mechanism enforces a
versioning paradigm of this sort on protocols using this extension
mechanism.
This extension mechanism allows very extensive protocol changes, up
to and including the replacement of one protocol by an entirely
different one. For some kinds of protocol extensions, this seems the
only way to effect the change.
Noveck Expires September 11, 2016 [Page 6]
Internet-Draft nfs-extension March 2016
This approach was used to effect a transition between NFSv2 and NFSv3
(See Section 4.2) and between NFSv3 and NFSv4.0 (See Section 5.1).
3.3. Protocol Extension by XDR Extension
It is possible to replace an XDR definition by one which is an
extension in the sense that,
o The set of messages described by the second definition is a
superset of that described by the first.
o Each message within the set of messages described by the first
definition is recognized as having exactly the same structure/
interpretation by the second definition.
o Each message within the set of messages described by the second
definition but not the first definition must be recognized as part
of an unsupported extension.
Within an XDR/RPC framework, extensions can be arrived at by:
o Addition of previously unspecified RPC operation codes.
o Addition of new, previously unused, values to existing enums.
o Addition of previously unassigned bit values to a flag word.
o Addition of new cases to existing switches, provided that the
existing switch did not contain a default case.
Such an extension relation between XDR descriptions is reflexive,
antisymmetric, and transitive and thus defines a partial order. In
practice, provisions have to be made to make sure that two extensions
of the same description are compatible (i.e. either one is an
extension of the other, or there is a another description that is a
valid extension of both).
To put things in concrete terms, such compatibility can be assured if
measures are taken to ensure:
o That the same operation code is not used for two different RPC
operations.
o That the same enum value is not assigned two different meanings.
o That the same bit flag value is not assigned two different
meanings.
Noveck Expires September 11, 2016 [Page 7]
Internet-Draft nfs-extension March 2016
o That whenever the same case value is added to the same switch in
two different extensions, the content assigned to the two matching
added cases is the same.
o That default cases are never added to existing switches.
In contrast to XDR replacement discussed in Section 3.2, use of XDR
extension remains atypical and is not supported by the XDR/RPC
infrastructure. It is important to keep this fact in mind, as we
discuss the use of XDR extension by NFSv4 in Section 5.2 and beyond.
3.4. Combination of Protocol Extension Mechanisms
It is possible to use multiple such means of protocol extension
simultaneously. When this is done, the result is a composite
extension mechanism. For example, if the XDR replacement or XDR
extension mechanism is adopted, a protocol-specific mechanism can be
added to it, if the protocol-specific mechanism is built on objects
whose XDR definition is sufficiently generic. (e.g. opaque arrays or
feature bitmasks).
It can be argued that the NFSv4 attribute model provides such an
embedded protocol-specific extension mechanism, since sets of
attribute values are specified as XDR opaque arrays and attribute
sets are specified as variable-length arrays of 32-bit integers
allowing new attribute bits to be defined outside of the XDR
definition framework.
Note that there exists specification text that suggests that
attributes are part of the XDR specification, making it hard to reach
a firm conclusion on the matter. However, the resolution of this
question does not affect the other matters discussed below, since, in
either case, we have an extension mechanism that allows optional
extensions.
3.5. Switching Extension Mechanisms
While it is possible to choose multiple extension mechanisms for
different sorts of extensions, based on their characteristics,
protocols typically do not take advantage of that flexibility.
On the other hand, protocols do, as NFS has done, change their
preferred extension mechanisms in response to long-term changes in
requirements. However, once having done so, they rarely switch back.
Changing extension mechanisms is a big step, both conceptually and in
implementation terms, and is not commonly repeated.
Noveck Expires September 11, 2016 [Page 8]
Internet-Draft nfs-extension March 2016
In the case of NFS, the possibility of returning to an XDR
replacement model (as a major version change) has always been
available, but there has never been any significant interest in
taking this step. It remains unlikely, although not impossible, that
this could happen in the future.
4. Pre-IETF NFS Versioning
4.1. The Pre-IETF NFS Environment
NFSv2 and NFSv3 were described by the informational RFC's [RFC1094]
and [RFC1813]. These documents each described existing
interoperating client and server implementations. Thus they started
with running code. If there was a rough consensus in effect, it was
that these were useful protocols to use and thus that someone
building a client or server had to interoperate with the existing
implementations.
The following characteristics of protocol development during that
period are noteworthy.
o Most client implementations were implemented on very similar
systems, in terms of the API's supported and many specifics of the
local filesystems exported by servers. In particular, clients
exposed filesystem functionality to applications via a standards-
compliant API (POSIX).
o Often, the important client and server implementations were done
by the same organization.
As a result of these commonalities, specifications tended to avoid a
lot of detail that would have been required in a more diverse
environment. New protocol features were thought of in terms of
generally understood client and server implementation frameworks and
it was generally clear which of those could be implemented without
markedly changing that framework.
During this period, there was little perceived need for optional
protocol features within NFS, in part because of the commonalities
noted above. In some cases in which additional functionality was
desired, the facility was added via an optional sideband protocol
(e.g. NLM).
4.2. Transition from NFSv2 to NFSv3
There were a number of significant changes involved, but only the
first two were of major importance.
Noveck Expires September 11, 2016 [Page 9]
Internet-Draft nfs-extension March 2016
o Converting file sizes and offsets from 32-bit to 64-bit unsigned
integers.
o Support for uncommitted WRITEs and the COMMIT request.
o Provision for WRITE to return atomic pre- and post-write file
attributes, in order to allow a client to determine whether
another client was writing the file.
Interestingly enough, this feature was not carried over into
NFSv4.
o The READDIRPLUS request.
o The addition of NFS3ERR_JUKEBOX (the precursor of NFS4ERR_DELAY).
Of these only the first actually needed something as drastic as the
XDR replacement model. The others could have been handled simply by
adding new RPC requests and an enum value to an existing NFSv2 XDR.
Since NFS's extension mechanism was then XDR replacement, such
choices were not available.
5. Initial Work on NFSv4 Within IETF
Control of the development of NFS was transferred to the IETF in
connection with the development of NFSv4. In what follows, we will
be distinguishing between "NFSv4" as a family of protocols and
"NFSv4.0", the first minor version of NFSv4, also sometimes referred
to as "NFSv4".
5.1. Transition from NFSv3 to NFSv4.0
NFSv4.0 was the first NFS version published as a Standards track
document. Although an initial [RFC3010], entitled "NFS version 4
protocol" was published as a Proposed Standard, it was never
implemented due to issues with the design of locking.
Subsequently, [RFC3530], entitled "Network File System (NFS) version
4 Protocol" was published as a Proposed Standard, obsoleting
[RFC3010]. Later, the bis document [RFC7530] along with the XDR
definition [RFC7531] were published as Proposed Standards.
The set of changes made to create NFSv4.0 was larger by far than that
for NFSv3. A partial list follows.
o Conversion to a stateful protocol, including support for locking.
Locking facilities included support for OPENs (with share
Noveck Expires September 11, 2016 [Page 10]
Internet-Draft nfs-extension March 2016
reservations), byte-range locking (optionally including mandatory
byte-range locks) and delegations.
o The COMPOUND operation. Although the main motivation for this
mechanism was latency reduction, its main role has been to provide
the basic framework for XDR extensibility (see Section 5.2).
o Conversion to a bi-directional protocol, by the addition of
(optional) callbacks.
o Internationalization.
o Support for filesystems doing case-insensitive name matching.
o A new, extensible file attribute model, including support for
acls, and conversion of user and group identifiers to a string
model, opening the way for multi-domain support.
o Support for named attributes.
o Merger of ancillary protocols (e.g. MOUNT, NSM, NLM) into the NFS
Protocol proper.
o Support for crossing of filesystem boundaries within a server's
file name space (originally done for the incorporation of MOUNT
functionality).
o Support for such multi-server operations as migration,
replication, and referral.
Referrals were not explicitly mentioned in [RFC3530] and are
explained in [RFC7530].
These features/extensions were implemented via the XDR replacement
model. This was the only realistic alternative, not only because of
the size of the list, but also because some of the changes undercut
some of the central design elements of the pre-IETF NFS protocol.
Note that minor versioning, an important element of NFSv4, is not
discussed above. This is partly because minor versioning was not
part of NFSv4.0, even though it was discussed in [RFC3530]. Minor
versioning only became an NFSv4 feature during the development of
NFSv4.1, as discussed in Section 6.1. We will discuss initial plans
for minor versioning in Sections 5.3 and 5.4.
Noveck Expires September 11, 2016 [Page 11]
Internet-Draft nfs-extension March 2016
5.2. NFSv4 and XDR Extensibility
Two of the elements within NFSv4.0 have had an important role in
allowing NFSv4 to be modified using an XDR extension approach, which
was initially used to implement minor versioning.
o Since COMPOUND was the only RPC procedure, that aspect of NFSv4
never had to change. The COMPOUND request is a variable-length
array, with each element being a switched union selected by an
operation code. The case of COMPOUND results is similar.
New request operations could be added by extending the two
switched unions, one for operations and one for results.
Because of the way XDR works in many implementations, it is often
not possible to obtain a specific error code in response to an
undefined operation code. Instead, the request containing this
operation might be reported as undecipherable. To deal with this,
clients would have to test for server awareness of particular
protocol features before attempting to use the feature element in
question.
o The new attributes model, unlike COMPOUND, was originally designed
to enable protocol extensibility. Sets of attributes are
specified as XDR variable-length arrays containing attribute bit
masks and sets of attributes by nominally opaque arrays.
New attributes can be added by defining the numeric constant
identifying the added attribute and specifying the XDR type of new
attribute.
Because attribute sets are defined as opaque arrays, this has no
effect on the code generated by XDR. Decoding of the collection
of attributes within the nominally opaque array specifying a set
of attributes is not part the task of the code corresponding to
the XDR specification of NFSv4 protocol.
A number of other elements of the protocol were subject to extension:
o Bits within flag words could be extended by defining the relevant
constants. This does not result in any change in the code
generated by XDR.
o Enumerations, such as error codes, may be extended by added the
new enumeration value to the existing XDR code. Unless the XDR
code actually refers to the new values, this does not result in
any change in the code generated by XDR.
Noveck Expires September 11, 2016 [Page 12]
Internet-Draft nfs-extension March 2016
o The issues for addition of new cases to existing switched unions
are similar to those for addition of operations to COMPOUND, as
described above. Just as in that case the requester (i.e. client)
needs to test to make sure the responder (i.e. server) can
properly interpret the extended union before attempting to send
it.
Issues regarding additional callback operations are similar to those
for request operations. CB_COMPOUND takes the place of COMPOUND and
the client and server switch roles. The server is the requester and
the client is the responder
In this document, we refer to the individual extensions listed above
as "feature elements". For some previous documents defining minor
versioning rules (e.g. [RFC5661] and [NFSv42]), it is unclear
whether the rules are referring to individual feature elements or a
set of such elements. Our use of the term "feature elements" is
desirable for clarity because of the uncertainty just mentioned and
because ([NFSv4-vers]) uses the word "feature" in a different sense
(i.e. to denote a set of feature elements which are documented
together).
5.3. Initial Minor Versioning Model for NFSv4
The minor versioning approach for NFSv4 uses an XDR extension model.
It was presented within a versioning paradigm but the fact that all
the added features were to be (at least initially) optional indicated
that features were intended to be built independently, and that
clients were expected to deal with their presence or absence. Note
that the term "features" is not explicitly defined. We assume that
the definition includes operations within COMPOUND or CB_COMPOUND,
attributes, added flag bits and enum values, and new cases of XDR
switch definitions. As noted above, we refer to such items as
"feature elements", even though the rules we are discussing may be
using the term "features", and there has been a lack of clarity as to
what precisely is meant (See Section 6.1.)
Now let's look at some specifics of the minor version rules
established for NFSv4 in [RFC3530]. Note that some of these were
significantly modified by [RFC5661] and [NFSv42], as discussed in
Section 6.4.
o No RPC requests may be added. Thus COMPOUND (and NULL) are to be
the only requests within all NFSv4 minor versions.
Similarly for callbacks, CB_COMPOUND and CB_NULL are the only
requests within the callback program.
Noveck Expires September 11, 2016 [Page 13]
Internet-Draft nfs-extension March 2016
o The arguments for COMPOUND and CB_COMPOUND contain a 32-bit
minorversion field.
Although this field is part of the minor versioning paradigm, it
is not clear how useful it is, as long as all extensions are
optional. For a more detailed discussion of this issue, see
Section 7.2.
o Features may be defined as optional, recommended, or required.
Later, these designations were converted to use the corresponding
upper-case terms defined in [RFC2119]. See Sections 6.1 and 9.2
for details.
These designations apply to implementation by the server. For
clients, no operations are defined as required, although it is
hard to imagine an NFSv4.0 client that does not use PUTFH or
SETCLIENTID, for example.
o Features may be upgraded or downgraded along the
optional/recommended/required scale.
o Features may be declared "mandatory to not implement". This
allows the deletion of a feature while retaining as reserved the
value previously assigned.
o Clients and servers that support a particular minor version must
support all previous minor versions as well.
o New features may not be designated as required in the minor
version in which they are introduced.
o Clients are not allowed to use stateids, filehandles, or similar
returned objects from the COMPOUND procedure with one minor within
another COMPOUND procedure with a different value of the
minorversion field.
This model was subsequently modified in [RFC5661] and in [NFSv42].
See Sections 6.1 and 6.2 for details.
Many of the events anticipated in the model presented above have
never been realized and it is likely that they never will be
realized. See Section 6.3 for some details. Examples are:
o There have never been optional attributes.
o Features have never been upgraded or downgraded in the transition
between minor versions.
Noveck Expires September 11, 2016 [Page 14]
Internet-Draft nfs-extension March 2016
5.4. Goals of Minor Versioning for NFSv4
If we try to understand why the minor versioning model was adopted we
can first look at [RFC3530], which has a number of relevant
statements.
Protocol extensibility (as opposed to versioning) is mentioned early
in the RFC. In a short bulleted list of protocol goals, "Designed
for protocol extensions" appears together with the following text:
The protocol is designed to accept standard extensions that do not
compromise backward compatibility.
Later, the following text appears at the start of the section "Minor
Versioning".
To address the requirement of an NFS protocol that can evolve as
the need arises, the NFS version 4 protocol contains the rules and
framework to allow for future minor changes or versioning.
The document does not explain why the goal of extensibility was
constrained by the subsequent establishment of a strict versioning
model. It is probably the case that the implications of this
constraint were not really examined, and that the shift from major to
minor versioning seemed to the working group like a sufficiently
radical change to address foreseeable change management issues. It
may also be the case that the fact that NFSv4 was moving outside the
existing framework for RRC/XDR versioning made additional conceptual
change difficult to consider.
Together with the rules that follow, which define an XDR extension
model, we can draw the following conclusions.
o That the use of XDR replacement (presumably to implement "major"
versioning) was felt to be not helpful in the current
circumstances and that a lighter-weight alternative was desirable.
o That clients were, as a general rule, expected to deal with the
presence or absence of particular extensions
If we look at how the various feature statuses are used, we have a
basis to understand how the model was intended to work
o Rule #12, in saying that features could not be mandatory at
initial introduction, states that this rule "forces the use of
implementation experience before designating a feature as
mandatory." One can conclude from this that it was anticipated
that many features might be introduced without implementation
Noveck Expires September 11, 2016 [Page 15]
Internet-Draft nfs-extension March 2016
experience and that features designated "optional" might be better
thought of as experimental.
o If it was expected that features might be introduced without
implementation experience, there would need to be some way to
correct those flaws that were found after introduction. As the
restriction on changes to existing XDR structures would limit the
ability of new minor versions to correct those flaws, it appears
that the provision for declaring features mandatory to not
implement was necessary not only to delete obsolete features, but
also as a way to correct flawed features by replacing an existing
feature by a similar one, with appropriate corrections.
Given the above it is hard to escape the conclusion that the ability
to fix protocol bugs, in addition to the adding of entirely new
features was intended, at least implicitly, to be part of the NFSv4
minor versioning model. Given that the lines between fixing bugs,
correcting feature deficiencies, enhancing features, and providing
new features are hard to draw in a precise way, there would have been
no way to proceed on any other basis.
Despite the above, as things developed, issues explicitly involving
protocol bug correction were not discussed during the period in which
minor versions were being constructed. The issue of using NFSv4's
extension model to correct protocol bugs was next addressed by
[NFSv4-vers] as discussed in Section 9.
6. Use of Minor Versioning in NFSv4
6.1. Transition from NFSv4.0 to NFSv4.1
NFSv4.1 was described in [RFC5661] and [RFC5662], each of which was
published as a Proposed Standard.
NFSv4.1 made a major change to NFSv4.0. It was able to do so
primarily using an XDR extension model although it did not follow the
rules laid out in Section 5.3. Instead, [RFC5661] presented its own
set of minor versioning rules.
The following major changes in the rules were made.
o Uses of the terms "recommended", "required", "should", etc. were
switched to use the corresponding upper-case words defined in
[RFC2119]. This included conversion of "recommended" attributes
to be "RECOMMENDED". The reasons for this change remain unclear.
Noveck Expires September 11, 2016 [Page 16]
Internet-Draft nfs-extension March 2016
Unlike the changes noted below, this change was incorporated in
[RFC7530]. For a discussion of how this situation was dealt with
during the IESG review of that document, see Section 9.2
o The rule requiring that features be introduced initially as
optional was modified to enable features described as
"infrastructural" to be required upon introduction.
o Also, the requirement that clients and servers support previous
minor versions changed from a "must" to a "SHOULD". Presumably,
this change reflects the fact that a minor version with
substantial infrastructural changes is essentially a new protocol,
making the "must" seem dubious. Whether the "SHOULD" here is in
line with [RFC2119] needs to be explored.
NFSv4.1 also made a change that affected the interpretation of the
minor versioning rules, apart from any text changes to those rules.
In [RFC3530], the term "feature" is not defined, leading one to
conclude that it denotes what we have called "feature elements." By
publishing a table showing the status of various operations and
callbacks and their relationship to various specific features,
[RFC5661] opened the way to a different interpretation. However,
this shift was incomplete, leaving room for continued ambiguity
since:
o Only a small set of features are listed, leaving such operations
as OPENATTR as orphan feature elements. They are listed as
"OPTIONAL", in which case their status of optional features
implies that (some) feature elements are features as well.
o Many operations are listed as "REQUIRED" with no assigned feature
named. This poses no immediate problem since most of these are
sessions-related. However, given the provision that features can
be downgraded, it is not clear what is being referred to here.
o Feature elements such as attributes are not listed, although they
clearly have a status as "OPTIONAL" or "RECOMMENDED", and in some
cases a relationship to a broader feature.
NFSv4.1 also bypassed the versioning rules by making non-XDR-based
protocol changes. See Section 7.1 for a discussion of the logic
behind such changes.
A large set of changes was associated with the following two
structural modifications in the protocol. Each of these involved
multiple XDR-based feature elements and some non-XDR-based semantic
change.
Noveck Expires September 11, 2016 [Page 17]
Internet-Draft nfs-extension March 2016
o Support for a sessions model including support for EOS (exactly-
once semantics).
o A new set of operations were added which enable the client and
server to identify themselves to one another. Although these were
implemented to support the sessions model and are often thought of
as part of the sessions model, they are best thought of as
logically distinct.
The session model involved the following set of protocol changes:
o The new operation SEQUENCE and CB_SEQUENCE were defined to allow
per-request session-related information to be specified while
avoiding changes to existing XDR structures.
SEQUENCE is specified as required in [RFC5661], while CB_SEQUENCE
is specified as optional.
o The new callback operation CB_RECALL_SLOT was introduced as
required, although servers are allowed to escape the requirement
by not creating a callback channel.
Since each use of CB_RECALL_SLOT requires an initial CB_SEQUENCE
in the CB_CPMPOUND, it seems that it is an error to designate
CB_RECALL_SLOT as required while CB_SEQUENCE is optional.
o Many semantic changes were made to existing operations to support
this arrangement. Since SEQUENCE was supposed to be the first
operation of each request, the semantics of each request was
modified to make it invalid to specify that operation if SEQUENCE
had not appeared first.
o The semantics of each of the operations which previously had a
clientid in their arguments was modified since the associated
clientid was now inferable from the session on which the
associated request was issued.
o The semantics of each operation which used owner-based seqids was
modified since these seqids were no longer required to support
replay detection.
o Because of the deletion of owner-based replay detection,
OPEN_CONFIRM was not needed and it became mandatory to not
implement.
o Also, related to the deletion of owner-based replay detection was
the fact that RELEASE_LOCKOWNER became mandatory to not implement.
Because lockowners no longer held replay detection state, servers
Noveck Expires September 11, 2016 [Page 18]
Internet-Draft nfs-extension March 2016
were able to delete them at will, eliminating the need for the
client to be involved in deciding when it was valid to release
them.
o Stateids became tied to the specific session on which they were
created, and from the session to the particular client with which
they were associated.
o RENEW became mandatory to not implement, since every request made
on a specific session had the effect of renewing the lease of the
associated client.
As a result of these changes, no COMPOUND request that contains any
operation which is valid in NFSv4.0 is also valid in NFSv4.1. Either
it contains an operation which has become mandatory to not implement,
or it contains an operation which requires a preceding SEQUENCE
operation, which did not exist in NFSv4.0. The only COMPOUND valid
in both protocols is one which consists a zero-length operation
array.
As mentioned previously, the sequence of operations by which clients
and servers connected to one another was expanded and reorganized.
This had a major role in enabling a number of functional enhancements
to the protocol.
o Providing for the creation and management of sessions, in
connection with implementing exactly-once semantics, as discussed
above.
o Proving a way for the server to identify himself to the client,
allowing the client to determine and use appropriate forms of
trunking in clustered-server situations.
o Restructuring the methods by which connections are associated to
clients, allowing callbacks to a assigned to connection separate
from that used in the forward direction.
The above set of changes involved multiple operations.
o EXCHANGE_ID was introduced as required, replacing SETCLIENTID
which became mandatory to not implement.
o CREATE_SESSION was introduced as required. Since it had the role
of confirming the original EXCGANGE_ID, it effectively replaced
SETCLIENTID_CONFIRM which became mandatory to not implement.
o DESTROY_CLIENTID and DSTROY_SESSION were introducing as required,
providing cleanup facilities missing from NFSv4.0
Noveck Expires September 11, 2016 [Page 19]
Internet-Draft nfs-extension March 2016
o BIND_CONN_TO_SESSION and BACKCHANNEL_CTL were introduced as
required, in order to regularize the management of bindings
between connections and sessions.
The following additional feature elements were added as required at
initial introduction:
o TEST_STATEID and FREE_STATEID were added to allow better
management of lock revocation and lost-lease situations.
o RECLAIM_COMPLETE was added to allow better server sequencing of
lock reclaim operations.
o suppattr_exclcreat was added as a REQUIIRED attribute to give
clients information about attributes that might be validly
specified as part of an exclusive create.
Given the above list, one is forced to the conclusion that these
feature elements were not included as required at initial
introduction because there was anything particularly
"infrastructural" about them. Rather, there were various feature
elements within NFSv4.1 that it was convenient to make required at
initial introduction. As a result, their presumed infrastructural
character arises from their inclusion as required protocol elements.
See Section 9.4 for a discussion of how such feature elements might
relate to the proposed reformulation of NFSv4 version management.
The addition of such feature elements did not give rise to any
difficulties despite the fact that the rules makes their inclusion
problematic. Rather, as discussed below, the problems that NFSv4.1
created for the NFSv4 versioning model arose from the features that
were most clearly of an infrastructural character.
In addition to these required feature elements, there were a number
of non-XDR-based changes made in NFSv4.1. Although not thought of as
"features" at the time, they are described in [RFC5661], which gives
no indication that support is optional. These include:
o Addition of a new special stateid to represent the current
stateid.
o New rules for the interpretation of a stateid seqid of zero.
o New rules regarding the interaction of the MODE and acl-related
attributes.
There also a number of non-required feature elements. While such
feature elements are optional in the sense that a server may or may
Noveck Expires September 11, 2016 [Page 20]
Internet-Draft nfs-extension March 2016
not support them, it is not clear that all are optional in terms of
the existing minor versioning rules. Given that all attributes are
specified as REQUIRED OR RECOMMENDED, it may be that attributes that
may or may not be supported are considered recommended from the
viewpoint of the minor versioning rules. Here and in the future we
will use "non-required" instead of "optional or recommended".
The following non-required feature elements were added.
o Feature elements to support Parallel NFS
o WANT_DELEGATION to allow delegations to be obtained apart from
opens.
o SECINFO_NO_NAME was introduced as "RECOMMENDED", although it is
"REQUIRED" if the pNFS file layout type is supported.
o Feature elements to support directory delegations and
notifications.
o The MODE_SET_MASKED, SACL, DACL attributes.
o The FS_LOCATIONS_INFO and FS_STATUS attributes.
Note that there has been little implementation work so far on the
last three of these items.
Parallel NFS created an alternate protocol extension mechanism for
NFS. New pNFS mapping types could be added, without any change to
the existing XDR or change in the minor version number. Existing
mapping types might have their own extension mechanisms. There also
exists the possibility that features might be added within the NFSv4
protocol proper, designed to, or capable of, interacting with
particular mapping types. This document will not these issues but
they will have to be addressed by the NFSv4 Versioning RFC.
The set of changes introduced by NFSv4.1 was very large, and
essentially made NFSv4.1 a different protocol from NFSV4.0, albeit
one accomplished using the XDR extension model, and following the
modified minor versioning rules in [RFC5661]
As a result of this bifurcation, the incremental enhancement provided
by minor versioning became unavailable to NFSv4.0, while it still was
available to NFSv4.1. While there was some discussion in the working
group regarding the use of "micro-versioning" to address this gap, it
was not followed up on and work to reform NFSv4 version management
followed a different path.
Noveck Expires September 11, 2016 [Page 21]
Internet-Draft nfs-extension March 2016
While there was a general sense within the working group that
versions as large as NFSv4.1 should be avoided in the future, there
was no effort to modify the versioning rules to make this less
likely.
6.2. Transition from NFSv4.1 to NFSv4.2
While NFSv4.2 has not been defined in an RFC, the associated
specifications have been approved by the IESG and are being prepared
for publication. It is thus highly unlikely to experience additions
to or deletions from its feature set before publication as a Proposed
Standard. [NFSv42] and [NFSv42-dotx] can serve as useful references
for this analysis.
Because of the lengthy process involved in producing NFSv4.1, the
working group decided that NFSv4.2 was to be a relatively small minor
version consisting only of optional features which would be
documented by specifying changes from NFSv4.1.
The following features (all optional) are provided for in NFSv4.2:
o Support for labeled NFS.
o Server-side copy features, including intra-server copy, inter-
server copy, and clone (a variant of intra-server copy).
o An operation fence option on EXCHANGE_ID.
o Application data holes (formerly application data blocks).
o Disk-space reservation (nominally "recommended" since it is
implemented by an attribute and attributes have never been
declared "optional").
o Hole-punching operations.
o READ_PLUS.
Note that there are two pieces of infrastructure that are used by
multiple features above. These are not "infrastructural" in the
sense mentioned in Section 6.1 (i.e. they are not required), but they
do serve an infrastructural role in that are required to be present
if one of the optional features that use them are supported.
o WRITE_PLUS used to implement (ordinary) hole punching and
application data holes.
Noveck Expires September 11, 2016 [Page 22]
Internet-Draft nfs-extension March 2016
o OFFLOAD operations/callbacks used to support WRITE_PLUS and
server-side copy.
6.3. Evolution of Minor Versioning Model within NFSv4
As noted above, there have been changes made by [RFC5661] and
[NFSv42] in the NFSV4 minor versioning model.
o NFSv4.1 (in [RFC5661]) introduced the concept of "infrastructural"
feature elements (i.e. those allowed to be required at initial
introduction).
o NFSV4.1 made additional non-XDR-based changes, which, although not
explicitly defined as such, were effectively required.
Taking note of these changes, we can classify potential minor
versions, starting with those that currently exist, based on how they
affect inter-version compatibility.
o Minor version zero which introduced a new (major) version of the
NFS protocol. All of the operations within it are new and a
subset are effectively required.
o Minor versions which make required changes in the protocol that
affect all implementations of the previous minor version.
Such changes may include addition of required/infrastructural
feature elements, incorporation of an effectively required non-
XDR-based change, or the deletion of a required feature element by
making it mandatory to not implement.
Currently, the only such version is minor version one, although it
is possible that there could be others in the future.
o Minor versions that only add optional/recommended feature
elements, each present or absent on the server with clients
needing to be able to deal individually with their presence or
absence.
Currently, the only such version is minor version two. It is
likely there will be others in the future and it is possible that
all future minor versions will be of this character.
o Minor versions which make required changes in the protocol that
will affect some implementations of the previous minor version.
These can result from feature element status changes. Changing a
non-required feature element to required affects only
Noveck Expires September 11, 2016 [Page 23]
Internet-Draft nfs-extension March 2016
implementations that do not support the feature element.
Conversely, deleting a non-required feature element by making it
mandatory to not implement affects only implementations that do
support the feature element.
No such minor versions currently exist.
6.4. Review of NFSv4 Versioning through NFSv4.2
To summarize protocol extension as it has been applied to the NFSv4
protocols:
o NFSV4.0 was implemented using the XDR replacement approach
inherited from NFSv2 and NFSv3. As was to be expected given the
nature and scope of the changes, its development took considerable
time.
It defined a protocol extension approach based on the XDR
extension mechanism which specified in framework built around the
concept of minor versions. However, this mechanism was not used
as part of the implementation of NFSv4.0.
o NFSV4.1 was primarily implemented using the XDR extension
mechanism. To implement sessions, it was forced to modify the
extension approach in the only way that seemed viable in the
circumstances. As a result, the specification process took a long
time, since it made significant structural changes to the protocol
and also because it specified the entire protocol in a new RFC,
rather than documenting a set of extensions.
NFSv4.1 also made a number of modifications to the NFSv4 protocol
which did not involve any XDR-based changes at all. These are
discussed in Section 7.1. While not considered infrastructural,
or even thought of as "features" at the time, these changes are
effectively required and the possibility of such changes needs to
be considered as the working group formulates its approach to the
problems of NFSv4 version management.
o NFSV4.2 returned to the original XDR extension mechanism and was
intended to be a small incremental update with a one-hundred page
(or less) specification. The fact that this turned out to be a
multi-year effort has occasioned concern and we will discuss how
the process can be streamlined and otherwise made more effective.
The history of NFSv4.2 development serves to illustrate some of
the inherent problems in the then-current approach to minor
versioning. Since similar problems can be expected to recur
Noveck Expires September 11, 2016 [Page 24]
Internet-Draft nfs-extension March 2016
unless that approach is changed, attention needs to be paid to
understanding why such difficulties were experienced.
It is important to note that NFSv4.0 (in [RFC3530] and [RFC7530]),
both about 300 pages and NFSV4.1 (in [RFC5661]), about 600 pages
documented the entire minor version. On the other hand, the NFSv4.2
specification document (in [NFSv42]) simply specified the changes
from the previous minor version, in about 100 pages.
Given the difficulties of writing very large specifications, it has
to be considered extremely unlikely that any future minor versions
will be documented other than as a set of changes from the previous
minor version.
7. Inherited NFSv4 Versioning Approach
7.1. Non-XDR-based Changes
Although not recognized at the time, the existence of non-XDR-based
protocol changes is part of the existing NFSv4 versioning approach
and must be addressed in any revision of NFSv4 version management.
Such changes have been of two types:
o Changes in field interpretation and use. Although the intention
has always been that all such matters were to be defined in XDR,
there are areas where this is not true. The interpretation of
certain opaque fields and of strings are cases in which the field
interpretation and use is defined in protocol specification text.
o Changes in operation semantics. These may apply to only a few
operations or to most or all operations.
All such changes have been implemented as required with
implementations using the minor version field of the COMPOUND to
determine whether the change applies to a particular request.
7.2. The Role of Minor Version Number in NFSv4
Minor versions which may affect inter-version compatibility form an
ascending sequence in which we also have a versioning paradigm,
implemented principally using XDR extension.
Optional-feature-only minor versions are fundamentally different.
Each NFSv4.2 server implements the same protocol as NFSv4.1 with a
particular set of optional or recommended feature elements beyond
those that are required. This set may range from the null set all
the way to all of the non-required feature elements. Here, it
Noveck Expires September 11, 2016 [Page 25]
Internet-Draft nfs-extension March 2016
appears that the versioning paradigm is not appropriate to the
reality of the extension mechanism.
As a way of illustrating the basic point , let us consider two
servers each of which only supports operations within NFSv4.1:
o The first server "supports" NFSv4.2 but none of the non-required
feature elements added in [NFSv42]. In this case, any attempt by
a client to use one of those features will result in an
NFS4ERR_OPNOTSUPP being returned.
o Let us say that the second server does not support NFSv4.2 and
supports precisely the same set of feature elements. In this
case, a request will be rejected (with error
NFS4RR_MINOR_VERS_MISMATCH) if its COMPOUND minorversion field is
two and if the field is one, any unsupported NFSv4.2 operation
will be rejected with NFS4ERR_OP_ILLEGAL.
Although this obeys the rules as they stand, there is no obvious
value for the client, the server, or the protocol in making these
artificial distinctions. Optional-feature-only minor versions such
as NFSv4.2 are not minor versions in the same sense that NFSv4.0 and
NFSv4.1 are. In this case the minorversion field is not providing
much useful information, while the set of operations supported is the
important thing that the server implementer chooses and that the
client needs to know.
The role of the minorversion field needs to be better understood.
This is particularly so in light of the fact that it appears to be
that the original intention was that all or most minor versions would
be optional-feature-only minor versions, where, as we have seen, it
has no useful role.
While this field is useful in distinguishing NFsv4.0 from NFSv4.1,
such transitions were never anticipated when the NFSv4 minor
versioning model was first created. A plausible hypothesis is that
the switch from "extension" to "versioning" led to a perceived
requirement for a version number without much thought about whether
it was needed and for what purpose.
Nevertheless this field has proved useful despite the weaknesses
noted above. It appears that we have the following ironic situation:
o The most likely original motivation, to get agreement on a common
XDR, appears to be unnecessary because the XDR extension approach
implies that a common XDR can be determined if different XDRs are
extensions of a common base XDR.
Noveck Expires September 11, 2016 [Page 26]
Internet-Draft nfs-extension March 2016
o The actual uses for this field involve changes in feature
statuses, which have only happened in ways that were strongly
discouraged by the original definition of minor versioning, and
non-XDR-based changes (see Section 7.1), which were never
seriously considered as being related to versioning.
7.3. Inherited NFS Versioning Practices
The following pattern was followed for NFSv4.2, and, unless a new
version management approach is adopted, seems likely to persist.
o Various features are sketched out in individual drafts.
o The working group reaches by rough consensus as to the extensions/
features to be included in the minor version. At the time
consensus is reached, the features may vary as to their maturity.
Some have individual drafts which sketch them out while others do
not.
o Any existing individual drafts are combined into a draft of a
working group document intended to eventually evolve into an RFC
describing the new minor version. These are then supplemented
with descriptions of the other features chosen for inclusion.
o This document goes though further refinement and cycles of working
group document review. At some point a companion -dot-x document
is prepared and reviewed by the working group as well.
o The two documents go through working group last call, IESG review,
and RFC publication.
This pattern of development is not a good fit for the kind of minor
version that NFSv4.2 is and many future such minor versions will be.
Such versions consist of a set of mostly unrelated features, each
individually selectable or not by implementers, artificially yoked
together. In essence, we have a "feature batch" rather than a minor
version.
7.4. Problems with Inherited NFS Versioning Approach
A number of issues have been noted with the existing process for
NFSv4.2, leading to the conclusion that the process needs to be
revised in some way, for future minor versions, of the same sort.
o It takes too long to get a minor version drafted and through
working group and IESG review. Despite the fact that NFSv4.2 was
intended to be a fairly minimal minor version, describable in a
one-hundred-page spec, it looks like the pace of development is
Noveck Expires September 11, 2016 [Page 27]
Internet-Draft nfs-extension March 2016
such that there will be nearly a five-year delay from the time
that the first NFSv4.2 draft was submitted to the time at which
the corresponding RFCs are published.
o The timeline for some of the features within NFSv4.2 is even
longer. It will have taken nearly seven years for the inter-
server copy feature to go from initial draft to RFC publication of
the minor version of which it is a part.
o Only quite late in the development of the version have there been
significant active implementations in which proposed last-minute
protocol changes could be tested for validity. As an example of
the problem, consider the decision to pass source stateids to the
COPY op. If there had been prototype implementations of inter-
server copy, the problems that this created (since stateids are
tied to clientids in NFSv4.1 and beyond) would have quickly become
manifest.
o Many features within NFSv4.2 had not received the kind of
searching review appropriate to later stages of specification
development, until very late in the process.
Some instances of problems/issues ascribable to a lack of searching
document review at an early stage are described below. Rather than
requiring the necessary review prior to feature acceptance, a common
pattern has been that important issues are only discovered on those
occasions in which it appears that work on the minor version is
coming to a close and that there are issues that have to be addressed
before they create difficulties for prospective implementers.
o The state of the IO hints feature is most unsatisfactory. It is
not clear how, or even if, it is possible to specify this in a way
that interoperable clients and servers can be written which
respond appropriately to such hints so that useful performance
improvements can be demonstrated.
o It was the general understanding within the group that labeled NFS
required use of RPCSEC_GSSv3, when in fact, only one case of
labelled NFS, the server-enforced one, required it. When it
became clear that RPCSEC_GSSV3 had not been worked on for a long
time, the working group had to address that issue seriously.
o The security for inter-server copy was specified to be dependent
on RPCSEC_GSSv3, yet, when it was found that RPCSEC_GSSv3 seemed
not to be on the horizon, it turned out that a simple alternative
was available. After a great deal of uncertainty about whether
the functionality needed for inter-server copy would really be
Noveck Expires September 11, 2016 [Page 28]
Internet-Draft nfs-extension March 2016
doable in RPCSEC_GSSv3, the working group had a range of choices
for inter-server copy security.
Later, after much work was done on RPCSEC_GSSv3, the working group
had the option of going back to an approach dependent on
RPCSEC_GSSv3.
o Application data blocks and related features have had a
complicated history within NFSv4.2. Application data blocks were
superseded by application data holes. There was little interest
in implementing the feature since, as specified, a server
implementation would require extensive filesystem extensions.
Also, client-side API's were lacking, meaning that the only
possible client-side implementations would be in special-purpose
clients tightly bound to a particular implementation.
Nevertheless, the feature continued in this unsatisfactory state
for a long while.
Eventually, it became clear that, as the feature was defined, it
imposed significant implementation constraints even on NFSv4.2
clients not implementing the feature. At this point, it became
clear that significant changes had to be made.
This issue was resolved by limiting the scope of the proposed
feature so that servers were not required to remember the
parameters relevant to application data holes, but only the data
that they represent.
If we look at the problems above, we can understand better how such
problems can arise. In short, the decision as to what features to
include within a minor version was not a good use of the rough
consensus model. In proceeding on that basis, the group created a
set of perverse incentives that undercut the process. Also, as the
process goes on for a long time, as is likely, these perverse
incentives are intensified. Consider the following points:
o It is not clear exactly what the consensus as to proposed minor
version contents actually means. Working group members might
interpret it as meaning "These features are worth pursuing and
they should be pursued". However, if they thought the definition
was more like, "each of these features is so important that, if it
is not ready, any other feature, including the one I'm interested
in, should be delayed also", then it is hard to imagine any such
rough consensus existing. Note that, given the minor versioning
implementation laid out in Section 7.3, the latter definition is,
for functional purposes, the effective meaning, of the minor
version content consensus.
Noveck Expires September 11, 2016 [Page 29]
Internet-Draft nfs-extension March 2016
o Given that many features are linked together, any delay in one
feature, once it is accepted as part of the feature batch, delays
all of the features, making it hard for people to comment
forthrightly on any significant specification inadequacies. Not
only will it delay your preferred feature, but if the problems are
not fixed, the only recourse is an extreme penalty. As a result,
it often seems not worth pursuing these sorts of issues.
o As the version turnaround cycle is so long, it is very difficult
to remove a feature from a minor version feature batch. Given
that these are all features that have enough interest to be in the
minor version, it is hard to transfer the feature into the next
minor version, given that doing so will certainly result in a
multi-year delay, at a minimum.
As a result, features, once accepted, have an implicit guarantee
of inclusion in the minor version, undercutting the motivation of
the proposer and others to work to move the feature forward.
On the other hand, uncertainty about the time of specification
completion often makes it hard to plan for and allocate resources
to development of client and server prototypes and
implementations.
o Given that responsibility for a minor version is transferred to
the editor of the minor version definition document at an early
stage, the process is such that is not clear who has
responsibility to follow up on the work necessary to make the
feature happen. Although formally this is the responsibility of
the minor version editor, he may not have the required time and
expertise in all cases. The lack of designated feature owner
makes it hard to follow up on things that require follow-up to
progress at an appropriate pace.
8. Formulating a New NFSv4 Extension Approach
The following issues led the working group to consider formulation of
a new approach to NFSv4 versioning:
o The experience of NFSv4.2 showed that, although there was a need
for shorter documents, addressing that need without other changes
in how things were done left important issues unaddressed.
o The lack of implementation experience for many features led to a
general feeling that the working group needed to do more to
encourage/require development of feature implementations before
they were published as Proposed Standards.
Noveck Expires September 11, 2016 [Page 30]
Internet-Draft nfs-extension March 2016
o The publication of multiple minor versioning rules came to seem at
variance with the idea of a rule-based approach. Eventually, it
was decided that a single version management document was needed
for NFSv4 as a whole.
9. Current Versioning-related Work
9.1. Creation of New NFSv4 Version Management Document
The working group is now working on a standards-track document
([NFSv4-vers]) defining an approach to version management that is
intended to apply to NFSv4 as a whole. The major advances that
formed the basis for that new document include
o Creation of a set of minor versioning rules to form a basis for a
unified set of versioning rules. Such a set of rules would
supersede the per-minor-version rules that had previously been
present.
o Creation of the concept of extensible minor version with an
associated workflow that avoids many of the problems noted above
with regard to the batching of features.
o Explicitly allowing XDR changes to correct protocol errors. This
addresses a situation in which there was sufficient uncertainty
about such changes to prevent them from being considered, even
though they were not explicitly disallowed.
9.2. Related Changes to Other Documents
During this period, changes were made to some related documents, as
they moved toward publication.
Some of these changes were made to simplify handling of the
transition in versioning models that would occur upon publication of
[NFSv4-vers] as an RFC.
o During the IESG review process, drafts of [RFC7530] were modified
to eliminate specific minor versioning rules. As a result these
rules did not appear in [RFC7530]
This made sense because the rules in question did not apply to
NFSv4.0, and were, with regard to NFSv4.1 and beyond, superseded
by those in [RFC5661].
o The restatement of minor versioning rules in [NFSv42] was
eliminated. In its place was left a short statement of
v4.2-relevant exceptions from existing rules. The text is
Noveck Expires September 11, 2016 [Page 31]
Internet-Draft nfs-extension March 2016
compatible with the base rules being taken from either [RFC5661]
or [NFSv4-vers].
As a result of these changes, upon publication of an NFSv4 version
management RFC, it will only need to be marked as updating [RFC5661].
Another relevant change concerned the use of the term "RECOMMENDED"
regarding attributes which are not REQUIRED. As it appeared that
this usage is inconsistent with [RFC2119], the draft of [RFC7530] was
modified and [RFC7530] was published including the following
statement:
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
in this document are to be interpreted as described in RFC 2119
except where "REQUIRED" and "RECOMMENDED" are used as qualifiers
to distinguish classes of attributes as described in
Section 1.3.3.2 and Section 5.
A reasonable conclusion to draw from this is that while certain
attributes are indeed REQUIRED, the other ones cannot be designated
as "RECOMMENDED" as that term is defined in [RFC2119].
Work still needs to be done to deal with the analogous issue in
[RFC5661].
This change relates to the NFSv4 feature status model. Although
there has been discussion of feature elements moving within a status
hierarchy including OPTIONAL, RECOMMENDED, and REQUIRED statuses,
with the exception of attributes, "RECOMMENDED" has never been used.
Given that all feature elements are, in RFC2219 terms, either
OPTIONAL or REQUIRED opens the way to a simpler feature status model.
9.3. Further work on New NFSv4 Versioning Document
Previously the versioning document incorporated two conceptual models
which needed to be better integrated to give us a more coherent
treatment of version management within NFSv4.
o The versioning rules inherited from [RFC5661] are focused on minor
versions and individual XDR changes (in our terms these are
"feature elements" although the existing rules refer to them as
"features"). Even though the working group expects the addition
of incremental features to be at the core of our prospective
versioning practice, the rules relate to that approach only by
implication.
Noveck Expires September 11, 2016 [Page 32]
Internet-Draft nfs-extension March 2016
o The version extensibility and protocol fix work had focused on
(non-required) features, but had no real place for minor versions
other than as a passive recipient of those features. It was
decided to focus on this aspect of version management since it is
likely to be the primary means by which changes will be introduced
into NFSv4 in the future. Nevertheless, there was a need to
examine the possibility of minor versions that might refactor
things or otherwise allow other sorts of minor-version-level
change, that are outside the scope of things that can be done via
the simple extension model explored so far.
Recent work has restructured the treatment of NFSv4 version
management into a layered structure.
o Changes, including XDR changes and non-XDR changes.
o Features, which are groups of protocol changes specified together.
o Minor versions which combine a set of features which may by
optional or required.
One way of dealing with the issues surrounding the minor versioning
rules is to re-organize them so as to move away from a single set of
minor versioning rules, while adapting them to the evolving version
management model in [NFSv4-vers].
Although the result may not have the format of a numbered list of
rules, there will be sets of guidelines that serve the needed
functions. One possible organization is the following:
o Rules defining the types of protocol changes that may be made.
These would include the XDR extensions mentioned in the existing
rules, but there would also have to be provision for at least some
of the non-XDR-based changes that have been useful in the past. A
large part of the original rules wound up in this group.
o Rules governing the construction and organization of features.
This is a new area.
o Rules governing the incorporation of features in the protocol,
whether this as part of a published minor version, an extension to
a published minor version, or as a correction to a published minor
version. This is also a new area.
o Rules governing the changes of feature statuses in a minor
version. These are the only rules regarding minor versions that
are still directed at minor versions. In the same class are rules
Noveck Expires September 11, 2016 [Page 33]
Internet-Draft nfs-extension March 2016
which would deal with possibility of post facto changes of inter-
feature compatibility constraints.
o Rules that deal with the interaction of multiple minor versions,
on the model of rules 11 and 13 in [RFC5661]. These are the only
rules directed to implementers, as opposed to those defining new
features and minor versions.
9.4. Issues Needing further discussion
While there has been general acceptance of the idea that version
management needs to become more flexible and incremental, the details
need to be more fully discussed.
One particular area of discussion is to get more clarity on the role
of minor versions within the new version management approach. It
appears that there is a range of opinions about how often minor
versions will be needed. Some people are of the opinion that all
required protocol change can be done using the extension model,
making an provisions for additional minor versions redundant. Others
feel that there may well be situations in which changes which cannot
be performed using the existing model will be required.
Fortunately, it is not necessary to get agreement on this issue to
proceed with this document. As long as people are in agreement as to
the situations that would require a new minor version, the fact that
they have different expectations as to whether and when those will
occur is not relevant at this point. Those are matters best left for
discussion when the relevant situations might arise.
Currently [NFSv4-vers] defines the following events as requiring a
minor version to effect. These need to be discussed to make sure the
group has a general understanding of our versioning options going
forward.
o Addition of required features.
o Changes of features from optional to required or the reverse.
o Conversion of features to mandatory to not implement.
o Any non-XDR-based semantic changes.
o Any change to the status of feature elements within existing
features.
Noveck Expires September 11, 2016 [Page 34]
Internet-Draft nfs-extension March 2016
o Any change that affect constraints regarding what existing
features may be present together, including feature dependence and
compatibility rules.
It might also be useful for the working group to have a discussion of
situations in which it might choose to create a new minor version,
even if not obliged to do so.
o To simplify the task of clients in determining what features
servers might be aware of.
o To logically isolate a previous epoch of protocol extension
preparatory to moving the protocol in a new direction.
Another issue that the working group might profitably discuss is the
question of versions that make larger sets of protocol changes, on
the order of NFSv4.0 or NFSv4.1. While most people feel that such
changes are quite unlikely in the foreseeable future, there is likely
to be a range of opinions about how a version management RFC should
deal with the matter.
o The text might make it clear that when introducing a feature as
required, the scope of related changes that follow from it (i.e.
what [RFC5661] presumably means by its "infrastructural"
character) is a reason to make it optional at initial
introduction, rather than the reverse.
o The text might simply ban introducing a feature as required,
although some of these would not pose problems if they are small
enough, as was the case with some of those introduced in NFSv4.1.
(See Section 6.1).
o Stating that such changes be made using the XDR replacement model,
implicitly indicating that such versions should be part of
NFSv5.x, and thus out of scope for an NFSv4 document.
10. Looking Back
We can summarize the history of protocol extension within the NFS
family as protocols as follows:
o At first, NFS used the XDR replacement paradigm in order to effect
protocol extension, following standard practice for XDR-based
protocols.
o With the development of NFSv4.0, the basic mechanisms were in
place to adopt an XDR extension paradigm as a means of protocol
extension. Despite this, for reasons that are not very clear, a
Noveck Expires September 11, 2016 [Page 35]
Internet-Draft nfs-extension March 2016
minor versioning approach was established, even though the basic
goal, to enable optional features, was at variance with this.
o A number of efforts were made to implement this minor versioning
approach. While protocol improvements were made, the results were
troubling and various ad hoc adjustments were made. For example,
when NFSv4.1 broke with the optional-feature-only idea and
produced a 900-page spec, it was decided that NFSv4.2 was to be a
small minor version with a spec shorter than 100 pages.
o When it turned out NFSv4.2, despite being so much smaller, took
over four years to go from -00 to IESG consideration (as opposed
to three years for NFSv4.1) did it become clear that serious
reform was in order.
A question that has been asked is why NFSv4 has had such difficulty
arriving at a workable approach to protocol extension, compared to
other protocols, that have embraced incremental extension without
difficulty. This is a question for which a definitive answer is not
possible. Nevertheless, it is a reasonable question, and the author
has attempted to answer it.
In part, the problem stems from an early confusion of extensibility
and versioning, as we have seen in Section 5.4. While this sheds
light on the origin of the problem, in a way it compounds our
difficulties. Given that this same confusion first appeared in
[RFC3010], it appears that it has taken around fifteen years to
provide a version management framework appropriate to the initial
goals for NFSv4.
In part, this extended hiatus derives from the history cited above.
o It is difficult to focus properly on change management in the
absence of actual change.
o Since NFSv4.1 did not really implement the version management
approach initially specified for NFSv4, there was no experience to
see how well that worked until NFSv4.2 was fairly far along.
o The unexpected complexity and difficulty of NFSv4.2 may have
forced attention on completing that specification, as opposed to
investigating how/why those difficulties arose.
Despite the plausibility of the above, the author feels that it
doesn't fully explain the length of the gap and thinks that looking
for a more systemic explanation is worthwhile. The following
suggestions are worth considering as an explanation for NFSv4's
difficulties compared to other protocols:
Noveck Expires September 11, 2016 [Page 36]
Internet-Draft nfs-extension March 2016
o When other protocols implement extension mechanisms, they
generally do so in a specialized way. This is opposed to the case
of NFSv4 in which a very large part of the protocol is, at least
theoretically, subject to change. It would not be surprising if
this situation occasioned concern about the possibility of
uncontrolled change causing a reduction of effective
interoperability.
o The history of NFS created expectations that numbered versions
would be produced. The initial definition of the minor versioning
model reinforced those expectations and created institutional
barriers that strengthened them. For example, the working group
charter would at times mention numbered minor versions and
discussed the items that they were expected to contain.
o Because NFSv4 was an XDR-based protocol, there were further
conceptual barriers to change that developed. This was in part
because RPC version negotiation is built around the concept of
numbered versions but a more fundamental issue derives from the
fundamentals of XDR, which, by seeking a full definition of the
protocol to compile, seems to foreclose the idea of extensibility
by focusing on the need for equivalence/identity between two
protocol definitions. As a result partial order defined by the
XDR extension relation may have been outside the natural
conceptual framework for an XDR-based protocol.
In part, these are problems which derive from the identification of a
protocol with the contents of an XDR file. This has had two
unfortunate consequences.
o If the XDR file had not changed, it was assumed that the protocol
had not changed. As a result, there has been no awareness of the
version management implications of non-XDR-based semantic changes
made in NFSv4.1, simply because there was no corresponding change
to the XDR.
o It is often the case that it is felt that a new error cannot be
added simply because the error code would be added to an existing
enum and thus change the XDR code, which is felt to be inherently
dangerous, even though the code generated by XDR would not change
at all.
The fact that some of the necessary extensions did affect the code
generated by XDR led to further difficulties. While it might seem
straightforward to design a mechanism to allow extensions, and NFSv4
in [RFC3530] defined such a mechanism, the information hiding that is
part of the XDR interface could have caused the details of the
compatibility issues to be obscured. Instead, the natural tendency
Noveck Expires September 11, 2016 [Page 37]
Internet-Draft nfs-extension March 2016
was only to see that the XDR's were different, and thus presumably
incompatible. As a result, the virtues of the NFSv4 extensibility
infrastructure were difficult to see, and so we were unable to take
full advantage of them.
11. Looking Forward
Once this document gets through working group last call, there are a
number of events that must occur before implementation of a new
version management approach can begin.
o The new version management document [NFSv4-vers] needs to be
approved for publication as a Proposed Standard.
o The NFSv4.2 documents ([NFSv42] and [NFSv42-dotx]) need to be
approved for publication as a Proposed Standards. Since
[NFSv4-vers] states that each minor version beyond NFSv4.1 will be
extensible by default, these approvals and that of [NFSv4-vers]
may happen in any order.
o Some extension needs to be accepted by the working group as a
feature specification document, as described in [NFSv4-vers]. A
likely candidate for this role is [xattr] which introduces a
feature that allows user-specified extended attributes. Anther
possible candidate feature is discussed below. As work proceeds
on that possible feature specifications and on [NFSv4-vers],
reviewers would verify that they meet the requirements for a
feature specification document.
Another possible extension involves the mode_umask attribute
described in [mode-umask], which is currently an individual draft.
This attribute allows ACL inheritance to avoid problems that have
arisen in adapting to the POSIX-based umask concept. This
extension's purpose is to address an oversight in the design of
NFSv4's access control attributes, present in all existing minor
versions, including NFSv4.0. Because of that fact, it is likely to
be turned into a working group item and it might be considered as a
protocol correction, possibly updating [RFC7530], after [NFSv4-vers]
is published as a Proposed Standard.
Once [xattr] and [NFSv4-vers] are published, the extended attributes
feature would become a valid optional extension to NFSv4.2 and
clients and server would be able to support it. If [mode-umask] and
[NFSv4-vers] are published the same thing would happen with the
mode_umask attribute.
Given that a dramatic change will be involved, it seems that a
discussion of risk mitigation is in order. The most important
Noveck Expires September 11, 2016 [Page 38]
Internet-Draft nfs-extension March 2016
component of addressing the risks associated with a new process is
active concern about problems that arise. This needs to be combined
with a willingness to make appropriate adjustments if problems arise.
Clearly, we need to avoid a situation in which an extension model is
not working and the problems go on for years without being addressed.
The principal means by which the risk of change may be mitigated
would involve care in approving extensions. Both the working group
and the IESG should exercise the requisite care, to make sure that
the extensions are clearly documented, have an appropriate scope, and
address real problems, Further, having a clear requirement for
interoperable prototype implementations should have the effect of
limiting excessive feature creation activity.
12. Security Considerations
Since no substantive protocol changes are proposed here, no security
considerations apply.
As features and minor versions designed and specified in standards-
track documents, their security issues will be addressed and each RFC
candidate will receive the appropriate security review from the NFSv4
working group and IESG.
13. IANA Considerations
The current document does not require any actions by IANA.
Depending on decisions that the working group makes about how to
address the issues raised in this document, future documents may
require actions by IANA.
14. References
14.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
14.2. Informative References
[mode-umask]
Fields, J. and A. Gruenbacher, "Allowing inheritable NFSv4
ACLs to override the umask", February 2016,
<http://www.ietf.org/id/draft-bfields-nfsv4-umask-01.txt>.
Noveck Expires September 11, 2016 [Page 39]
Internet-Draft nfs-extension March 2016
Work in progress.
[NFSv4-vers]
Noveck, D., "NFSv4 Version Management", January 2016,
<http://www.ietf.org/id/
draft-ietf-nfsv4-versioning-03.txt>.
Work in progress.
[NFSv42] Haynes, T., Ed., "NFS Version 4 Minor Version 2", January
2016, <http://www.ietf.org/id/
draft-ietf-nfsv4-minorversion2-41.txt>.
Work in progress.
[NFSv42-dotx]
Haynes, T., Ed., "NFS Version 4 Minor Version 2 Protocol
External Data Representation Standard (XDR) Description",
January 2016, <http://www.ietf.org/id/
draft-ietf-nfsv4-minorversion2-dot-x-41.txt>.
Work in progress.
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7,
RFC 793, DOI 10.17487/RFC0793, September 1981,
<http://www.rfc-editor.org/info/rfc793>.
[RFC1094] Nowicki, B., "NFS: Network File System Protocol
specification", RFC 1094, DOI 10.17487/RFC1094, March
1989, <http://www.rfc-editor.org/info/rfc1094>.
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
Version 3 Protocol Specification", RFC 1813,
DOI 10.17487/RFC1813, June 1995,
<http://www.rfc-editor.org/info/rfc1813>.
[RFC2780] Bradner, S. and V. Paxson, "IANA Allocation Guidelines For
Values In the Internet Protocol and Related Headers",
BCP 37, RFC 2780, DOI 10.17487/RFC2780, March 2000,
<http://www.rfc-editor.org/info/rfc2780>.
[RFC3010] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "NFS version 4
Protocol", RFC 3010, DOI 10.17487/RFC3010, December 2000,
<http://www.rfc-editor.org/info/rfc3010>.
Noveck Expires September 11, 2016 [Page 40]
Internet-Draft nfs-extension March 2016
[RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "Network File System
(NFS) version 4 Protocol", RFC 3530, DOI 10.17487/RFC3530,
April 2003, <http://www.rfc-editor.org/info/rfc3530>.
[RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1
Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010,
<http://www.rfc-editor.org/info/rfc5661>.
[RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed.,
"Network File System (NFS) Version 4 Minor Version 1
External Data Representation Standard (XDR) Description",
RFC 5662, DOI 10.17487/RFC5662, January 2010,
<http://www.rfc-editor.org/info/rfc5662>.
[RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System
(NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530,
March 2015, <http://www.rfc-editor.org/info/rfc7530>.
[RFC7531] Haynes, T., Ed. and D. Noveck, Ed., "Network File System
(NFS) Version 4 External Data Representation Standard
(XDR) Description", RFC 7531, DOI 10.17487/RFC7531, March
2015, <http://www.rfc-editor.org/info/rfc7531>.
[xattr] Naik, M., "File System Extended Attributes in NFSv4",
March 2016,
<http://www.ietf.org/id/draft-ietf-nfsv4-xattrs-02.txt>.
Work in progress.
Appendix A. Acknowledgements
The author wishes to thank Chuck Lever of Oracle for his
comprehensive document review and his many important suggestions,
which helped to significantly improve this document.
Author's Address
David Noveck
Hewlett Packard Enterprise
165 Dascomb Road
Andover, MA 01810
US
Phone: +1 978 474 2011
Email: davenoveck@gmail.com
Noveck Expires September 11, 2016 [Page 41]
