Internet DRAFT - draft-mzhang-nfsv4-recursively-setting

draft-mzhang-nfsv4-recursively-setting







Network File System Version 4                                   M. Zhang
Internet-Draft                                       R. Kunhi Parambattu
Intended status: Standards Track                                 J. Yang
Expires: 27 June 2024                                            D. Geng
                                                                Z. Jiang
                                                                   Y. Du
                                                     Huawei Technologies
                                                        25 December 2023


              Recursively Setting Directories and Subitems
               draft-mzhang-nfsv4-recursively-setting-00

Abstract

   In recent years, the concept of near-data computing has been widely
   recognized in storage architectures.  The core idea is to process
   data nearby, reduce the overhead of network transmission, and utilize
   the computing capability of smart devices (such as intelligent NICs,
   smart SSDs, and DPUs).  This reduces CPU and memory usage of clients
   (computing nodes) and improves data processing efficiency.  This
   design idea is applied in NFSv4.2 or future NFS verions, such as
   Server-Side Copy, in which client sends the control command and the
   storage server copies data without passing through the data between
   the client and storage server.  Compared with traditional copy
   operations, data is read from the source storage server and then
   written to the target storage server after two network transmissions.
   Data transmission on the network is reduced, and bandwidth resources
   are greatly released.  In addition, the client changes from an
   original data copy executor to a data copy controller, and a specific
   execution action is executed by the storage server.  Therefore, a
   large amount of computing resources and memory resources are saved on
   the client side.

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.







Zhang, et al.             Expires 27 June 2024                  [Page 1]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 27 June 2024.

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Problem Statement . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Protocol Overview . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Implementation Considerations . . . . . . . . . . . . . . . .   4
   4.  Recursive Set Operations  . . . . . . . . . . . . . . . . . .   7
   5.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  12
     5.1.  Normative References  . . . . . . . . . . . . . . . . . .  12
     5.2.  Informative References  . . . . . . . . . . . . . . . . .  12
   Appendix A.  An Appendix  . . . . . . . . . . . . . . . . . . . .  12
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  12

1.  Problem Statement

   In actual storage applications, users often recursively set the
   attributes of directories and subitems(their subfiles and
   subdirectories).  Message interaction between client and server is
   complex, and the client consumes a lot of resources, which does not
   match the concept of near-data computing.  FIG. 1 shows the existing
   flowchart of recursively setting the attributes of all files under
   dir1.




Zhang, et al.             Expires 27 June 2024                  [Page 2]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   Step 1: The client sends the readdir command to obtain the list of
   all files in dir1.

   Step 2: The storage server responds to the readdir operation.  If the
   directory contains many subitems, the client needs to run the readdir
   operation for multiple times.

   Step 3: The client obtains the file list based on the response
   information, executes the getattr request for each file, and obtains
   the file attributes.

   Step 4: The storage server responds to the getattr request.

   Step 5: The client sends a setattr request.

   Step 6: The storage server responds to the setattr request.

   Repeat step 1 to step 6 to traverse all files.  If the dir1 directory
   contains 100,000 files, repeat step 1 to step 6 for 100,000 times to
   recursively set the dir1 attribute.  It can be learned that the
   process consumes CPU resources and memory resources of the client,
   and a large number of messages are exchanged between the client and
   the storage server.  As a result, an end-to-end time for the
   attribute setting operation is relatively long.

   preamble to the figure.


                    Client                                Server
                    +                                       +
                    |                                       |
                    |------ Readdir ----------------------->|
                    |<--------------------------------------|
                    |------ Getattr ----------------------->|
                    |<--------------------------------------|
                    |------ Setattr ----------------------->|
                    |<--------------------------------------|
                    |         ....                          |
                    |                                       |

           Figure 1: Existing flowchart for recursive set operation

   As you can see, this figure doodled and dawdled.

2.  Protocol Overview

   After adopting the concept of near data calculation, the above
   scenario can be optimized.



Zhang, et al.             Expires 27 June 2024                  [Page 3]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   Step 1: The client identifies that the operation object of the
   attribute setting is a directory and the attribute setting is
   recursive, and invokes a new operation (for example, RECURSIVE_SET).

   Step 2: The storage server recursively queries all files in the
   directory, sets attributes for each file, and sends a RECURSIVE_SET
   response.

   Compared with the original process, this process not only saves the
   CPU and memory usage of the client, but also significantly reduces
   the number of messages exchanged between the client and server.  This
   greatly improves the efficiency in recursive attribute setting
   scenarios.

   Similar to the design of Server-Side Copy, the new operation word
   RECURSIVE_SET is used to recursively set the attributes of a
   directory and its subitems.  This operation can be in synchronous or
   asynchronous mode, which is determined by the input parameter:

   o If no back channel is created when the client and server establish
   a connection, the client can only use the synchronous mode in the
   RECURSIVE_SET request.  If the client uses the asynchronous mode, the
   server returns the error code NFS4ERR_CB_PATH_DOWN.

   o If a back channel is created when the client and server establish a
   connection, the RECURSIVE_SET request can be in synchronous or
   asynchronous mode.

   Note to RFC Editor: this section may be removed on publication as an
   RFC.

3.  Implementation Considerations

   A recommended Recursive Set operation in an synchronous mode is shown
   in Figure 2.

   Step 1: The client sends a RECURSIVE_SET request.  In the request,
   select the synchronization setting mode and set rsa_sync to true.

   Step 2: If the storage server completes the setting within the
   timeout period, the storage server returns the request result to the
   client.  If the setting is not complete within the timeout period,
   generate rsr_callback_id and rsr_recursiveverf.  In addition, the
   rsr_callback_id, rsr_recursiveverf, and error code are sent to the
   client.






Zhang, et al.             Expires 27 June 2024                  [Page 4]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   Step 3: The client sends a RECURSIVE_SET_STATUS query request.  The
   request contains the information of rss_stateid.  The value of
   rss_stateid is obtained from the response packet of RECURSIVE_SET
   which value is the value rsr_callback_id.  If the value of
   rss_stateid is the same as the value of rsr_callback_id cached on the
   storage server, the storage server returns the setting result.
   Storage server return NFS4_OK if the server successfully finish the
   recursive setting or NFS4_Pending if the setting is till on going, or
   else set failure error code.  If the value of rss_stateid in the
   request is different from the value cached on the server, the storage
   server returns the error code NFS4ERR_BAD_STATEID.

   Step 4: After receiving the RECURSIVE_SET_STATUS response, the client
   delays sending the RECURSIVE_SET_STATUS request if the error code is
   NFS4_PENDING.  The delay setting request must contain rss_stateid.
   If the error code returned by the server is NFS4_OK, the recursive
   attribute setting is successful.  If the error code is the error code
   of setattr defined in NFSv4.2, the recursive attribute setting fails.
   In this case, the client returns a response to the application.

   preamble to the figure.


                 Client                                                 Server
                 +                                                         +
                 |                                                         |
                 |------ RECURSIVE_SET(rsa_sync = 1) --------------------->|
                 |                                                         |
                 |<-----Response(rsr_callback_id=0, rsr_recursiveverf=0)---|  within the timeout period
                 |                                                         |
                 |                                                         |
                 |<----Response(rsr_callback_id=1, rsr_recursiveverf=1)----|  beyond the timeout period
                 |                                                         |
                 |                                                         |
                 |                                                         |
                 |-------RECURSIVE_SET_STATUS(rss_stateid=1)-------------->|
                 |                                                         |
                 |<------Response------------------------------------------|
                 |                                                         |
                 |                                                         |

                   Figure 2:  A synchronous Recursive Set

   An alternative Recursive Set operation in an synchronous mode is also
   given in Figure 3.

   As you can see, this figure doodled and dawdled.




Zhang, et al.             Expires 27 June 2024                  [Page 5]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   Step 1: The client sends a RECURSIVE_SET request.  In the request,
   select the synchronization setting mode and set rsa_sync to true.

   Step 2: If the storage server completes the setting within the
   timeout period, the storage server returns the request result to the
   client.  If the setting is not complete within the timeout period,
   generate rsr_callback_id and rsr_recursiveverf, and set the error
   code to NFS4_PENDING.  In addition, the rsr_callback_id,
   rsr_recursiveverf, and error code are sent to the client.

   Step 3: The client sends a RECURSIVE_SET_STATUS query request.  The
   request contains the information of rss_stateid.  The value of
   rss_stateid is obtained from the response packet of RECURSIVE_SET
   which value is the value rsr_callback_id.  If the value of
   rss_stateid is the same as the value of rsr_callback_id cached on the
   storage server, the storage server returns the setting result. (if
   Success, set NFS4_Pending, or else set failure error code).  If the
   value of rss_stateid in the request is different from the value
   cached on the server, the storage server returns the error code
   NFS4ERR_BAD_STATEID.

   Step 4: After receiving the RECURSIVE_SET response, the client delays
   sending the RECURSIVE_SET_STATUS request if the error code is
   NFS4_PENDING.  The delay setting request must contain rss_stateid.
   If the error code returned by the server is NFS4_OK, the recursive
   attribute setting is successful.  If the error code is the error code
   of setattr defined in NFSv4.2, the recursive attribute setting fails.
   In this case, the client returns a response to the application.

   A recommended Recursive Set operation in an asynchronous mode is
   shown in Figure 3.

   Step 1: The client sends a RECURSIVE_SET request.  In the request,
   the asynchronous setting mode is rsa_sync=false.

   Step 2: The storage server needs to generate rsr_callback_id and
   rsr_recursiveverf, and set the error code to NFS4_OK.  In addition,
   the rsr_callback_id, rsr_recursiveverf, and error code are sent to
   the client.  The storage server continues the recursive setting
   operation.

   Step 3: After receiving the response, the client checks the error
   code and starts an asynchronous task to receive the callback message
   from the server.







Zhang, et al.             Expires 27 June 2024                  [Page 6]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   Step 4: The client create an asynchronous listening task and matches
   rsr_callback_id and rsr_recursiveverf.  If the matching succeeds, the
   task is successfully executed.  If rsr_callback_id can be matched but
   rsr_recursiveverf cannot be matched, client skip the message.

   Step 5: After recursively setting attributes, the storage server
   sends a message to the client through the backchannel of the NFS4 and
   notifies the client of the setting result.

   Step 6: If the client does not receive the asynchronous message, the
   started task is forcibly terminated when the session is destroyed.
   If an error occurs when the storage server recursively sets subitem
   attributes, the storage server terminates the task and returns the
   error code to the client.  All possible errors are subject to the
   error codes defined by setattr.

   preamble to the figure.


                 Client                                                 Server
                 +                                                         +
                 |                                                         |
                 |------ RECURSIVE_SET(rsa_sync = 0) --------------------->|
                 |                                                         |
                 |<------Response(rsr_callback_id=1, rsr_recursiveverf=1)--|
                 |                                                         |
                 |                                                         |
                 |<------CB_RECURSIVE_SET----------------------------------|
                 |                                                         |
                 |                                                         |
                 |                                                         |

                   Figure 3: An asynchronous Recursive Set

   As you can see, this figure doodled and dawdled.


4.  Recursive Set Operations

   4.1 Operation TBD1: RECURSIVE_SET – Recursively sets the attributes
   of a directory and its subitems

   ARGUMENT

   <CODE BEGINS>

   Struct RECURSIVE_SET4args {




Zhang, et al.             Expires 27 June 2024                  [Page 7]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   bool rsa_sync;

   };

   <CODE ENDS>

   RESULT

   <CODE BEGINS>

   struct recursive_set_response4 {

   stateid4 rsr_callback_id;

   verifier4 rsr_recursiversr_recursiveverf;

   };

   union RECURSIVE_SET4res (nfsstat4 rsr_status) {

   case NFS4_OK:

   recursive_set_response4 rsr_resok4;

   default:

   void;

   };

   <CODE ENDS>

   DESCRIPTION

   The RECURSIVE_SET operation is used by the client to recursively set
   the attributes of a directory and all its subitems.  The operation
   should be placed after setattr in the existing setattr operation.
   After the storage server receives the setattr combination operation,
   if the setattr operation does not followed by RECURSIVE_SET, the
   original process remains unchanged.  If the setattr operator is
   followed by the RECURSIVE_SET operation, the storage server considers
   the attributes of the directory and its subitems to initiate
   recursive set mode.

   If the storage is successfully executed, the values of
   rsr_callback_id and rsr_recursiversr_recursiveverf are 0.





Zhang, et al.             Expires 27 June 2024                  [Page 8]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   If the storage is not complete, the values of rsr_callback_id and
   rsr_recursiversr_recursiveverf are generated.

   If rsa_sync is set to true, there are two processing policies:

   o According to the NFSv4 protocol, the client must wait for the
   response from the server.  Therefore, the client can wait for the
   processing result from the server.  A problem in this mode is that
   the current request occupies a slot in a session, resulting in a
   decrease in the number of available slots.  If multiple tasks of the
   same type are being executed, no slot is available on the client in
   severe cases.

   o The storage server determines the implementation duration.  If the
   implementation duration is too long, the storage server may return
   non-zero values of rsr_callback_id and
   rsr_recursiversr_recursiveverf.

   After the client receives the request, the client waits for a period
   of time and executes RECURSIVE_SET_STATUS to query the execution
   progress of the current task.  If the server does not finish the
   execution , NFS4ERR_PENDING is returned.  After receiving the error
   code, the client retryes the query after a period of time.  If the
   execution is complete, NFS4_OK is returned.


   4.2 Operation TBD2: RECURSIVE_SET_STATUS – Query the result of the
   recursively setting the attributes of directories and their subitems

   ARGUMENT

   <CODE BEGINS>

   struct RECURSIVE_SET_STATUS4args {

   stateid4 rssa_stateid;

   };

   <CODE ENDS>

   RESULT

   <CODE BEGINS>

   #define NFS4ERR_PENDING 10090

   struct RECURSIVE_SET_STATUS4res {



Zhang, et al.             Expires 27 June 2024                  [Page 9]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   nfsstat4 rssr_status;

   };

   <CODE ENDS>

   DESCRIPTION

   The RECURSIVE_SET_STATUS operation is used by the client to query the
   status of a recursively set task (attributes of a directory and its
   subitems).  If the task on the storage server is complete, NFS4_OK is
   returned.  If any error occurs during task execution, a response
   error code is returned and the error code is not extended or modified
   in this case so the error code is the same as the error code that may
   occur during the setattr operation.  If the current setting task is
   not complete, NFS4_PENDING is returned.

   4.3 Operation TBD3: RECURSIVE_SET_CANCEL – Canceling a Running Task
   on the Client

   ARGUMENT

   <CODE BEGINS>

   //The following operation is used to cancel the recursive setting
   task that is being executed.

   struct RECURSIVE_SET_CANCEL4args {

   stateid4 rsca_stateid;

   };

   <CODE ENDS>

   RESULT

   <CODE BEGINS>

   struct RECURSIVE_SET_CANCEL4res {

   nfsstat4 rscr_status;

   };

   <CODE ENDS>

   DESCRIPTION



Zhang, et al.             Expires 27 June 2024                 [Page 10]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   RECURSIVE_SET_CANCEL is used to cancel the task that is being
   executed.  The request packet contains rsca_stateid.  The value of
   rsca_stateid is obtained from the response of RECURSIVE_SET.  If the
   storage server fails to cancel the task, NFS4ERR_DELAY is returned.
   When receiving the message, the client delays the retry.  If the
   current task is complete, NFS4_OK is returned.

   4.4 Operation TBD4: CB_RECURSIVE_SET_NOTIFY – Notify the recursively
   setting result to client

   ARGUMENT

   <CODE BEGINS>

   struct CB_RECURSIVE_SET_NOTIFY4args {

   nfs_fh4 crsna_fh;

   stateid4 crsna_stateid;

   verifier4 crsna_recursiveverf;

   nfsstat4 crsna_status;

   };

   <CODE ENDS>

   RESULT

   <CODE BEGINS>

   struct CB_RECURSIVE_SET_NOTIFY4res {

   nfsstat4 crsnr_status;

   };

   <CODE ENDS>

   DESCRIPTION










Zhang, et al.             Expires 27 June 2024                 [Page 11]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   CB_RECURSIVE_SET_NOTIFY is used to send the server callback to client
   to notify the client of the result of the task of recursively setting
   the attributes of directories and their subitems.  Client check the
   crsna_stateid and crsna_recursiveverf and client will finish the wait
   task if they are matched the value received from previous
   RECURSIVE_SET response or will skip the notification in case of not
   match and return NFS4ERR_BAD_STATEID to server.

5.  References

5.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC7862]  Haynes, T., "Network File System (NFS) Version 4 Minor
              Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862,
              November 2016, <https://www.rfc-editor.org/info/rfc7862>.

5.2.  Informative References

   [InfRef]   "", 2004.

Appendix A.  An Appendix


Authors' Addresses

   Minqian Zhang
   Huawei Technologies
   1899 Xiyuan
   Chengdu
   High-tech West District, 611731
   China
   Phone: +86-13547833949
   Email: zhangmingqian.zhang@huawei.com


   Rijesh Kunhi Parambattu
   Huawei Technologies
   Email: rijesh.kunhi.parambattu1@huawei.com


   Jing Yang
   Huawei Technologies
   Email: yangjing8@huawei.com



Zhang, et al.             Expires 27 June 2024                 [Page 12]

Internet-Draft     Recursively Setting Directoriestle      December 2023


   Dongyu Geng
   Huawei Technologies
   Email: gengdongyu@huawei.com


   Zhongbing Jiang
   Huawei Technologies
   Email: jiangzhongbing@huawei.com


   Yunfei Du
   Huawei Technologies
   Email: duyunfei1@huawei.com






































Zhang, et al.             Expires 27 June 2024                 [Page 13]