AI Preferences                                                L. Peiyuan
Internet-Draft                                                Condé Nast
Intended status: Experimental                               8 April 2025
Expires: 10 October 2025


                 Protocol for Basic Automation Control
                   draft-liao-aipref-autoctl-core-00

Abstract

   This document specifies the core automation-preferences.txt protocol,
   a machine-readable document that defines server-side automation
   permissions with a focus on essential controls for AI and automation
   use cases.  Unlike the traditional robots.txt, which governs
   crawling, this protocol addresses a broader range of automation
   activities while maintaining simplicity for initial implementation.
   It defines the fundamental file format, policy declarations, HTTP
   method restrictions, and purpose requirements.  Advanced features are
   addressed in a separate extension specification.

About This Document

   This note is to be removed before publishing as an RFC.

   The latest revision of this draft can be found at
   https://datatracker.ietf.org/doc/draft-liao-aipref-autoctl-core/.
   Status information for this document may be found at
   https://datatracker.ietf.org/doc/draft-liao-aipref-autoctl-core/.

   Discussion of this document takes place on the AI Preferences Working
   Group mailing list (mailto:ai-control@ietf.org), which is archived at
   https://mailarchive.ietf.org/arch/browse/ai-control/.  Subscribe at
   https://www.ietf.org/mailman/listinfo/ai-control/.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.







Peiyuan                  Expires 10 October 2025                [Page 1]

Internet-Draft               aipref-autoctl                   April 2025


   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 10 October 2025.

Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Applicability . . . . . . . . . . . . . . . . . . . . . .   3
     1.2.  Relationship to Extension Specification . . . . . . . . .   3
   2.  Conventions and Definitions . . . . . . . . . . . . . . . . .   3
   3.  Protocol Specification  . . . . . . . . . . . . . . . . . . .   4
     3.1.  File Location and Format  . . . . . . . . . . . . . . . .   4
     3.2.  Automation Policy Declaration . . . . . . . . . . . . . .   4
     3.3.  HTTP Method Restrictions  . . . . . . . . . . . . . . . .   5
     3.4.  Purpose Declaration . . . . . . . . . . . . . . . . . . .   5
     3.5.  Scope and Applicability . . . . . . . . . . . . . . . . .   6
   4.  Extension Mechanism . . . . . . . . . . . . . . . . . . . . .   7
   5.  Implementation and Enforcement  . . . . . . . . . . . . . . .   7
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   8
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .   8
     8.2.  Informative References  . . . . . . . . . . . . . . . . .   9
   Sample automation-preferences.txt File  . . . . . . . . . . . . .   9
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .   9









Peiyuan                  Expires 10 October 2025                [Page 2]

Internet-Draft               aipref-autoctl                   April 2025


1.  Introduction

   The evolution of web automation has outpaced the capabilities of
   existing standards such as robots.txt, which only provides for
   crawler permissions.  This document introduces the core automation-
   preferences.txt protocol that enables server operators to explicitly
   define basic policies governing automated interactions.  These core
   policies cover fundamental aspects of automation control, including
   HTTP method restrictions and automation purpose declarations.

   The automation-preferences.txt protocol is designed with
   extensibility in mind, allowing future enhancements while maintaining
   backward compatibility.  This specification focuses on essential
   controls to facilitate initial adoption and implementation, with more
   advanced features defined in a separate extension document.

1.1.  Applicability

   The automation-preferences.txt protocol applies to automated systems
   interacting with web servers, especially those driven by foundation
   models or other types of advanced AI models.  It is designed to
   benefit both content owners-by allowing them to specify acceptable
   automation behaviors-and developers of automated systems, who can use
   these directives to ensure compliance.

1.2.  Relationship to Extension Specification

   This document defines the core functionality of the automation-
   preferences.txt protocol.  A separate document, "Protocol Extension
   for Advanced AI Automation Control," extends this core specification
   with additional directives and capabilities.  The extension
   specification builds upon this core document without modifying its
   requirements, providing a path for progressive implementation.

   Implementations conforming to only this core specification are
   considered compliant with the automation-preferences.txt protocol.
   The extension specification defines optional enhancements that may be
   implemented once the core functionality is established.

2.  Conventions and Definitions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   The following terms are used in this document:



Peiyuan                  Expires 10 October 2025                [Page 3]

Internet-Draft               aipref-autoctl                   April 2025


   *  Automation: Programmatic interactions with a web server.

   *  State-changing requests: HTTP methods that alter server state
      (e.g., POST, PUT, DELETE, PATCH).

   *  Automation policy: A set of rules defining permitted and
      prohibited automated behaviors for a specific scope.

   *  Automation purpose: The declared intent or use case for which
      automation is being performed.

   All terminology defined in this core specification applies to the
   extension specification without redefinition.  The extension
   specification may introduce additional terms for concepts not covered
   in this document.

3.  Protocol Specification

3.1.  File Location and Format

   The automation-preferences.txt file MUST be hosted at the root of the
   domain, in the same manner as robots.txt.  The file is structured as
   a series of key-value pairs that specify automation permissions.

   The file MUST be served with the text/plain MIME type.  Lines
   beginning with the hash symbol (#) are considered comments and MUST
   be ignored by parsers.  Each directive consists of a field name,
   followed by a colon, followed by a value.  Multiple values MAY be
   separated by commas.

   Parsers MUST silently ignore any directives they do not recognize.
   This enables future extensions to add new capabilities without
   breaking compatibility with existing implementations.

3.2.  Automation Policy Declaration

   A top-level directive, AutomationPolicy, indicates the overall stance
   of the server regarding automated interactions and state-changing
   requests.  The following values are defined:

   *  *open*: Automation is generally permitted with few restrictions.

   *  *limited*: Automation is permitted with specific restrictions.

   *  *strict*: Automation is heavily restricted or prohibited except in
      specific circumstances.





Peiyuan                  Expires 10 October 2025                [Page 4]

Internet-Draft               aipref-autoctl                   April 2025


   If the AutomationPolicy directive is not present, clients SHOULD
   assume a default value of "limited".

   Example:

   AutomationPolicy: limited

                                  Figure 1

3.3.  HTTP Method Restrictions

   The protocol MUST explicitly list allowed and disallowed HTTP methods
   using the AllowedMethods and DisallowedMethods directives.
   Typically, GET and HEAD are permitted while methods such as POST,
   PUT, DELETE, and PATCH are disallowed for automated processing.

   If no HTTP method directives are specified, clients SHOULD assume
   that only GET and HEAD methods are permitted.

   Example:

   AllowedMethods: GET, HEAD
   DisallowedMethods: POST, PUT, DELETE, PATCH

                                  Figure 2

3.4.  Purpose Declaration

   The protocol enables automation clients to declare their intended
   usage purposes.  This is managed using the following directives:

   *  RequireAutomationPurpose: Boolean value indicating whether clients
      must declare a purpose for automation.

   *  AllowedPurposes: Comma-separated list of permitted purposes using
      standardized vocabulary terms.

   *  DisallowedPurposes: Comma-separated list of prohibited purposes.

   The specific vocabulary terms for automation purposes are
   intentionally not defined in this protocol specification.  Instead,
   this protocol provides a mechanism for expressing allowed and
   disallowed purposes, which can be populated with terms from any
   widely accepted vocabulary standard.  This approach ensures that the
   protocol remains flexible and can adapt to evolving vocabulary
   standards while maintaining the essential structure for purpose
   declarations.




Peiyuan                  Expires 10 October 2025                [Page 5]

Internet-Draft               aipref-autoctl                   April 2025


   Example:

   RequireAutomationPurpose: true
   AllowedPurposes: [PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2]
   DisallowedPurposes: [PLACEHOLDER_PURPOSE3]

                                  Figure 3

3.5.  Scope and Applicability

   In a manner similar to robots.txt, the automation-preferences.txt
   file is divided into groups, each of which applies to a specific
   subset of content.  Each group begins with one or more scope
   directives that define the target of the preferences.  The following
   directives MAY be used within a group:

   *  Scope: Specifies the URL pattern (e.g., /admin/) to which the
      group applies.  Wildcards MAY be used to indicate variable
      components of the URL.

   *  Host: Specifies a subdomain or host.  If present, the group
      applies only to the indicated subdomain; if omitted, the group is
      assumed to apply to the entire host.

   Groups are processed in order of specificity.  When multiple groups
   match a given request, the group with the longest matching Scope
   directive SHALL take precedence.

   Example:

   # Group 1: Applies to the entire site
   Host: example.com
   Scope: /
   AutomationPolicy: limited
   AllowedMethods: GET, HEAD
   DisallowedMethods: POST, PUT, DELETE, PATCH

   # Group 2: Specific preferences for the /admin/ path
   Host: example.com
   Scope: /admin/
   AutomationPolicy: strict
   AllowedMethods: GET
   DisallowedMethods: POST, PUT, DELETE, PATCH

                                  Figure 4






Peiyuan                  Expires 10 October 2025                [Page 6]

Internet-Draft               aipref-autoctl                   April 2025


4.  Extension Mechanism

   The automation-preferences.txt protocol is designed for
   extensibility.  The protocol defines a forward-compatible approach
   where implementations:

   *  MUST process all recognized directives according to this
      specification

   *  MUST silently ignore any unrecognized directives

   *  MUST NOT fail or produce errors when encountering extended
      directives

   This enables future extensions to add new capabilities while
   maintaining compatibility with implementations of this core
   specification.  Extended directives may include:

   *  Rate limiting controls

   *  Automation technology restrictions

   *  API and XHR permissions

   *  Session requirements

   *  Asset-level annotation methods

5.  Implementation and Enforcement

   Servers implementing this protocol SHOULD:

   *  Verify incoming requests against the rules specified in
      automation-preferences.txt.

   *  Respond with appropriate HTTP status codes (e.g., 403 Forbidden)
      for non-compliant requests.

   *  Include HTTP headers that reference the applicable automation
      policy for transparency.

   Clients consuming this protocol SHOULD:

   *  Fetch and parse the automation-preferences.txt file before
      performing automated operations.

   *  Honor the HTTP method restrictions specified in the file.




Peiyuan                  Expires 10 October 2025                [Page 7]

Internet-Draft               aipref-autoctl                   April 2025


   *  Declare their automation purpose when required.

   *  Respect the scope directives when performing operations on
      different paths.

   Implementations SHOULD cache the automation-preferences.txt file to
   reduce server load, but SHOULD NOT cache it for longer than 24 hours
   to ensure timely policy updates.

6.  Security Considerations

   The use of machine-readable automation policies introduces security
   considerations that must be addressed by implementations:

   *  Parsing of automation-preferences.txt MUST be performed securely
      to prevent vulnerabilities such as buffer overruns and denial-of-
      service attacks.

   *  Care SHOULD be taken to avoid exposing sensitive policy details
      that could be exploited by adversaries.

   *  The protocol does not provide authentication or cryptographic
      verification mechanisms for the file content.  Servers SHOULD
      ensure the file is served via secure connections to prevent
      tampering.

   *  The protocol does not enforce client compliance; it relies on
      good-faith adherence by automation providers.  Servers SHOULD
      implement additional detection and enforcement mechanisms as
      needed.

7.  IANA Considerations

   This document has no IANA actions.

8.  References

8.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.




Peiyuan                  Expires 10 October 2025                [Page 8]

Internet-Draft               aipref-autoctl                   April 2025


8.2.  Informative References

   [RFC9309]  Koster, M., Illyes, G., Zeller, H., and L. Sassman,
              "Robots Exclusion Protocol", RFC 9309,
              DOI 10.17487/RFC9309, September 2022,
              <https://www.rfc-editor.org/rfc/rfc9309>.

Sample automation-preferences.txt File

   The following is an example of a automation-preferences.txt file that
   adheres to this specification:

   # Automation preferences for example.com
   # Version: 1.0
   # Last updated: 2025-04-08

   # Group 1: Applies to the entire site
   Host: example.com
   Scope: /
   AutomationPolicy: limited
   AllowedMethods: GET, HEAD
   DisallowedMethods: POST, PUT, DELETE, PATCH
   RequireAutomationPurpose: true
   AllowedPurposes: [PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2]
   DisallowedPurposes: [PLACEHOLDER_PURPOSE3]
   ContactEmail: automation-policy@example.com

   # Group 2: Specific preferences for the /admin/ path
   Host: example.com
   Scope: /admin/
   AutomationPolicy: strict
   AllowedMethods: GET
   DisallowedMethods: POST, PUT, DELETE, PATCH
   AllowedPurposes: [PLACEHOLDER_PURPOSE1]
   DisallowedPurposes: [PLACEHOLDER_PURPOSE2], [PLACEHOLDER_PURPOSE3]

                                  Figure 5

Author's Address

   Liao Peiyuan
   Condé Nast
   United States of America
   Email: peiyuan_liao@condenast.com







Peiyuan                  Expires 10 October 2025                [Page 9]