pNFS Metadata Striping
draft-mbenjamin-nfsv4-pnfs-metastripe-01

Abstract

This Internet-Draft describes a means to add metadata striping to pNFS. The text of this draft is substantially based on prior drafts by Eisler, M., with some departures. The current draft attempts to define a somewhat lighter-weight protocol, in particular, seeks to permit striping for "filehandle only" operations such as LOCK and OPEN + CLAIM_FH, without clients having to obtain metadata layouts on regular files. We gratefully acknowledge the primary contributions of Mike Eisler, Pranoop Ersani, and others.

Internet Draft Comments

Comments regarding this draft are solicited.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on January 16, 2014.

Copyright Notice

Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction and Motivation
• 2. Short List of Protocol Changes from Previous Drafts
• 2.1. File-system wide Striping for Filehandle-Only Operations
• 2.2. Uniform Filehandles
• 2.3. Simplified Multipath Device Model
• 2.4. Cookie Model
• 2.5. LAYOUTCOMMIT
• 2.6. Recommended Attributes
• 2.6.1. meta_stripe_deviceid (deviceid4)
• 2.6.2. meta_stripe_count (uint32_t)
• 2.7. PREADDIR (Operation)
• 3. Terminology
• 4. Scope of Metadata Layouts
• 4.1. Inode Striping
• 4.2. Dentry Striping
• 4.2.1. Name-Based Operations
• 4.2.2. Directory Enumeration
• 5. The Metadata Striping Layout
• 5.1. Name
• 5.2. Value
• 5.3. Data Type Definitions
• 5.3.1. Layout Hint
• 5.3.2. Devices
• 5.3.3. Metadata Layout
• 5.3.4. Layoutupdate4 lou_body
• 5.4. Metadata Layout Semantics
• 5.4.1. LAYOUTGET Argument Conventions
• 5.4.2. Inode Striping Layouts
• 5.4.2.1. Inode Stripe Hints
• 5.4.3. Dentry Striping Layouts
• 5.4.3.1. L-MDS Selection for Name-based Operations
• 5.4.3.2. Directory Enumeration
• 5.5. LAYOUTCOMMIT
• 6. Further Considerations
• 6.1. Storage Access Protocols
• 6.2. Revocation of Layouts
• 6.3. Stateids
• 6.4. Lease Terms
• 6.5. Layout Operations Sent to an L-MDS
• 6.6. Filehandles in Metadata Layouts
• 6.7. Restriping
• 6.7.1. Layout Recall Cases
• 6.7.2. Hint Invalidation
• 6.8. Recovery
• 6.9. Failure and Restart of Client
• 6.10. Failure and Restart of Server
• 7. Negotiation
• 8. Usage Examples
• 8.1. open-lock-write-close
• 8.2. parallel create-layoutcommit
• 8.3. parallel directory listing
• 9. Operational Recommendation for Deployment
• 10. Acknowledgements
• 11. Security Considerations
• 12. IANA Considerations
• 13. References
• 13.1. Normative References
• 13.2. Informative References
• Authors' Addresses

1. Introduction and Motivation

The NFSv4.1 specification describes pNFS [NFSv4.1]. pNFS distributes (stripes) file data across multiple storage devices. In NFSv4.1, parallel access is limited to the data contents of regular files. Metadata is not distributed or striped: the model presented in the NFSv4.1 specification is that of a single metadata server. This document describes a means to add metadata striping to pNFS, which includes the notion of multiple metadata servers.

Two methods are described. The first, called inode striping, directs metadata operations associated with a file handle to a preferred metadata server. The second, called dentry striping, distributes directory operations across a collection of metadata servers. With metadata striping, multiple metadata servers may work together to provide a higher parallel performance.

2. Short List of Protocol Changes from Previous Drafts

2.1. File-system wide Striping for Filehandle-Only Operations

Stripe hints redirect clients to a preferred metadata server for filehandle-only operations (below), but are backed by a single layout per-file system, rather than per-file, as in [METASTRIPE]. The new model is lighter weight, but since it remains layout-based retains the advantages of pNFS device indirection and garbage collection.

2.2. Uniform Filehandles

[METASTRIPE] offers implementations the option to propagate layout filehandles for all metadata layout types. Since it would be impossible to reasonably support this under the new proposed model for filehandle-only operations, we propose instead that L-MDS filehandles always be equivalent to I-MDS filehandles.

2.3. Simplified Multipath Device Model

[METASTRIPE] defines two different methods for encoding metadata server locations, only the "simple" model uses the pNFS device mechanism. In this draft, we propose a single model based on pNFS devices, in which there is a one-to-one mapping between devices and L-MDS servers. This approach facilitates sharing device addresses across layouts which have servers in common and also minimizes the difficulty of reclaiming devices no longer in use by any metadata layout.

2.4. Cookie Model

NFSv4 associates with each entry in a directory a unique value of type cookie4, a 64-bit integer. [METASTRIPE] involves cookies in stripe selection, and imposes specific requirements on cookie values. In the current proposal we treat cookies as opaque values except as specified in ordinary NFSv4.1. We concur with [METASTRIPE] that cookies MUST be unique within any logical directory regardless of the striping pattern. As in ordinary NFSv4.1, the behavior of READDIR (or PREADDIR, below) when cookie has a value previously returned to a client by the same server, but no longer associated with any directory entry, is not defined.

2.5. LAYOUTCOMMIT

In this draft, we introduce layout-subtype specific data for the LAYOUTCOMMIT operation.

2.6. Recommended Attributes

We propose two new recommended attributes.

meta_stripe_deviceid (deviceid4)

meta_stripe_count (uint32_t)

2.6.1. meta_stripe_deviceid (deviceid4)

An attribute of type meta_stripe_deviceid represents an inode stripe hint. This attribute MUST NOT be offered to clients unless they hold a valid inode striping layout on the containing file system.

2.6.2. meta_stripe_count (uint32_t)

The meta_stripe_count attribute represents, for directory objects, the directory's current stripe count, which may help the client decide if it will request a dentry striping layout on the directory. This attribute MAY be offered only to clients which hold an inode striping layout on the containing file system.

2.7. PREADDIR (Operation)

The NFSv4.1 READDIR operation has insufficient information to perform all possible enumerations required in our proposed dentry striping model. We propose a new PREADDIR operation which takes, in addition to all the current READDIR operations, also a controlling metadata layout stateid and stripe number.

3. Terminology

Initial Metadata Server (I-MDS). The I-MDS is the metadata server from which the client obtains a filehandle prior to acquiring any layout on the file.

Layout Metadata Server (L-MDS). The L-MDS is the metadata server from which the client obtains a filehandle from after redirection from a layout.

Regular file: An object of file type NF4REG or NF4NAMEDATTR.

Inode striping. Hint-based indirection to a preferred MDS for filehandle-based operations, backed by a filesystem-wide metadata layout.

Dentry striping. Fine-grained, layout-based indirection for parallel operations on directories, using a striping pattern.

4. Scope of Metadata Layouts

This proposal assumes a model where there are two or more servers capable of supporting NFSv4.1 operations. At least one server is an I-MDS, and the I-MDS should be thought of as a normal NFSv4.1 server, with the additional capability of granting metadata layouts on demand. The I-MDS might also be capable of granting non-metadata layouts, but this is orthogonal to the scope of metadata striping.

The model also requires at least one additional server, an L-MDS, that is capable of supporting NFSv4.1 operations that are directed to the server by the I-MDS. It is permissible for an I-MDS to also be an L-MDS, and an L-MDS to also be an I-MDS. Indeed, a simple submodel is for every NFSv4.1 server in a set to be both an I-MDS and L-MDS.

For convenience, we divide NFSv4.1 metadata operations into three classes:

• Filehandle-only. These are operations that take only filehandles as arguments, i.e. the current filehandle, or both the current filehandle and the saved filehandle, and no component names of files (e.g., LOCK, LAYOUTGET).
• Name-based. These are operations that take one or two filehandles (i.e. the current filehandle, or both the current file handle and the saved filehandle) and one or two component names of files (e.g., LINK, RENAME).
• Directory-enumeration. These are operations that take one filehandle and return the contents of a directory. Currently, NFSv4 has only one such operation, READDIR. This draft adds a section, PREADDIR.

Metadata striping applies to all of the foregoing NFSv4.x operations, and is of two types:

• inode striping uses hints (attribute-based indications) backed by a filesystem-wide layout to direct clients to a preferred MDS on which to perform filehandle-only operations
• dentry striping uses fine-grained metadata layouts on directories to support execution of name-based operations (directory enumeration, creates) on a set of MDS servers in parallel

4.1. Inode Striping

To avoid an explosion of new client state, a coarse-grained hinting mechanism is used to direct filehandle-only operations to a preferred metadata server.

As specified in 5.12.1 of [NFSv4.1], when a client encounters file system which supports LAYOUT4_METADATA, it can obtain a metadata layout of subtype LAYOUTMETA4_INODE, whose scope is the entire file system, using the LAYOUTGET operation on any filehandle object in the file system which it is permitted to access.

Then using ordinary READDIR and GETATTR requests, the client can obtain for any object in the file system a meta_stripe_deviceid attribute that indicates the preferred device to send filehandle-only or name-based operations for that object.

For example, suppose that after obtaining an ordinary filehandle via OPEN, a LAYOUTMETA4_INODE layout on the containing file system, and a meta_stripe_deviceid hint from a previous GETATTR, READDIR, or PREADDIR,, the client wants to get a byte range lock on the file. The client sends the LOCK request to the network address (pNFS device, L-MDS) indicated by the meta_stripe_deviceid attribute.

4.2. Dentry Striping

For name-based and directory enumeration operations, a more fine-grained, layout-based redirection mechanism is used.

When a client obtains a filehandle for an object that is of type directory and wishes to take advantage of metadata striping, the client first obtains a metadata layout of subtype LAYOUTMETA4_DENTRY on the directory. The client is provided with a directory-specific list of network addresses (devices) to which to send requests specific to objects in that directory.

4.2.1. Name-Based Operations

For name-based operations, the dentry striping layout indicates the preferred destinations in the network to send name-based operations for that directory (e.g., CREATE). The preferred destinations MUST apply to the current filehandle that the operation uses. In other words, for LINK and RENAME, which take both the saved filehandle and the current filehandle as parameters, the pNFS client would use the stripe hint of the target directory (indicated in the current filehandle) for guidance where to send the operation. Note that if an L-MDS accepts a LINK or RENAME operation, the L-MDS MUST perform the operation atomically. If it cannot, then the L-MDS MUST return the error NFS4ERR_XDEV, and the client MUST send the operation to the I-MDS.

The choice of destination is a function of the name the client is requesting. For example, after the client obtains the filehandle of a directory via LOOKUP and the metadata layout via LAYOUTGET, the client wants to open a regular file within the directory. As with the LAYOUT4_NFSV4_1_FILES layout type, the client has a list network addresses to which to send requests. With the LAYOUT4_NFSV4_1_FILES layout, the choice of the index in the list of network addresses was computed from the offset of the read or write request. With the metadata layout, the choice of the index is derived from the name (or some other method, such as the name and one or more attributes of the directory, such as the filehandle, fileid, as below.) passed to OPEN.

4.2.2. Directory Enumeration

For directory-enumeration operations, the dentry striping layout indicates the preferred destination in the network to send (P)READDIR operations for that directory. For example, after the client obtains the filehandle of a directory via LOOKUP and the metadata layout via LAYOUTGET, the client wants to read the directory. As with the LAYOUT4_NFSV4_1_FILES layout type, the client has a list network addresses to which to send requests. With the LAYOUT4_NFSV4_1_FILES layout, the choice of the index in list of network addresses was computed from the offset of the read or write request. For dentry striping layouts, the index counts from 0 to the dentry stripe count, less 1.

5. The Metadata Striping Layout

5.1. Name

The name of the metadata striping layout type is LAYOUT4_METADATA.

5.2. Value

The value of the metadata striping layout type is TBD1.

5.3. Data Type Definitions

5.3.1. Layout Hint

  ///  %
  ///  %/* Encoded in the loh_body field of type layouthint4: */
  ///  %
  ///  struct md_dirsize_layouthint4 {
  ///         uint64_t *mdlh_min_est;
  ///         uint64_t *mdlh_avg_est;
  ///         uint64_t *mdlh_max_est;
  ///         uint32_t *mdlh_stripe_count;
  ///         uint32_t *mdlh_stripe_modulus;
  ///  };
  

///  % /*
///  %  * Encoded in the da_addr_body field of data type
///  %  * device_addr4:
///  %  */
///  struct md_layout_addr4 {
///      multipath_list4    mdla_multipath_list<>;
///  };
  

///  enum md_layout_subtype4 {
///     LAYOUTMETA4_INODE = 0,
///     LAYOUTMETA4_DENTRY
///  };

///
///  enum md_namebased_alg4 {
///     MDN_ALG_CITYHASH64 = 0,
///     /* XXX TBD2 */
///  };
///

///  struct md_layout_dentry {
///     switch(uint32_t mdln_namebased_alg) {
///             case MDN_ALG_CITYHASH64:
///                     uint32_t seed;
///     };
///
///     deviceid4 mdln_devicelist<>;
///     uint32_t    mdln_stripe_pattern<>;
///  };

///  struct md_layout4 {
///     union md_layout_type
///             switch (enum md_layout_subtype4 subtype) {
///             case LAYOUTMETA4_INODE:
///                     void;
///             case LAYOUTMETA4_DENTRY:
///                     md_layout_dentry mdl_layout;
///     };
///  };

///
///  struct md_dentry_layoutupdate4 {
///      int32_t mdlu_entries_added;
///      int32_t mdlu_entries_removed;
///      nfstime4 mdlu_last_update;
///  };
///
///  % /*
///  %  * Encoded in the lou_body field of data type
///  %  * layoutupdate4:
///  %  */
///  struct md_layout_update4 {
///      union md_layout_type switch (enum md_layout_subtype4 subtype) {
///             case LAYOUTMETA4_INODE:
///                     void;
///             case LAYOUTMETA4_DENTRY:
///                     md_layoutupdate4 mlu_dentry;
///     };
///  };

Let F be the function specified in mdln_namebased_alg;

Let X = (x1, x2, x3, ...) some set of inputs for  function F, such
that x1 SHOULD be the component name of the file, and x2, x3, ... any
additional parameters required for the chosen F, their arguments
asserted to be values available to the client.

Let stripe_unit_number = F(X);
Let stripe_count = number of elements in mdl_layout.mdln_stripe_pattern;
Let idx =
    mdl_layout.mdln_stripe_pattern(stripe_unit_number % stripe_count);
Let deviceid = mdl_layout.mdln_devicelist[idx];

For stripe_number in 0 .. length(mdl_layout.mdln_stripe_pattern) -1
        do 
                Let stripe =
                    mdl_layout.mdln_stripe_pattern[stripe_number];
                Let device = mdl_layout.mdln_devicelist[stripe];
<PREADDIR at device, layout_stateid, stripe_number>

	  I-MDS: LAYOUTGET for inode layout -> ino_stateid
	  I-MDS: OPEN('foo') -> open_stateid
	  I-MDS: GETATTR(meta_stripe_deviceid) -> in_deviceid
	  I-MDS: GETDEVICEINFO(in_deviceid) -> [L-MDS]

	  L-MDS: LOCK(open_stateid) -> lock_stateid
	  L-MDS: WRITE(lock_stateid)
	  I-MDS: CLOSE(open_stateid)
	  I-MDS: LAYOUTRETURN(dn_stateid)
	  I-MDS: LAYOUTRETURN(ino_stateid)
	  

	  I-MDS: LAYOUTGET for inode layout -> ino_stateid
	  I-MDS: LAYOUTGET(dir) for dentry layout 
	    -> {dn_stateid, dn_deviceid, dn_placement}
	  I-MDS: GETDEVICEINFO(dn_deviceid)
	    -> [L-MDS1, L-MDS2, L-MDS3]

	  dn_placement('foo') -> L-MDS1
	  L-MDS1: CREATE(dir, 'foo')

	  dn_placement('bar') -> L-MDS2
	  L-MDS2: CREATE(dir, 'bar')

	  dn_placement('baz') -> L-MDS3
	  L-MDS3: CREATE(dir, 'baz')

	  I-MDS: LAYOUTCOMMIT(dn_stateid, +3)
	  I-MDS: LAYOUTRETURN(dn_stateid)
	  I-MDS: LAYOUTRETURN(ino_stateid)
	  

	  I-MDS: LAYOUTGET for inode layout -> ino_stateid
	  I-MDS: LAYOUTGET(dir) for dentry layout
	    -> dn_stateid, dn_deviceid
	  I-MDS: GETDEVICEINFO(dn_deviceid)
	    -> [L-MDS1, L-MDS2, L-MDS3]

	  L-MDS1: PREADDIR(dn_stateid, stripe=0, cookie=0)
	    -> [a, b, c]
	  L-MDS2: PREADDIR(dn_stateid, stripe=1, cookie=0)
	    -> [d, e, f]
	  L-MDS3: PREADDIR(dn_stateid, stripe=2, cookie=0)
	    -> [g, h, i]

	  I-MDS: LAYOUTRETURN(dn_stateid)
	  I-MDS: LAYOUTRETURN(ino_stateid)