Internet DRAFT - draft-williams-i18n-boundary-analysis
draft-williams-i18n-boundary-analysis
Network Working Group N. Williams
Internet-Draft Cryptonector
Intended status: Informational September 20, 2013
Expires: March 24, 2014
Boundary Analysis for Internationalization and Localization
draft-williams-i18n-boundary-analysis-00
Abstract
Internationalization of protocols and programs often requires
determining where to use one or another character repertoire,
codeset, encoding, where to perform localization, and so on. This
document aims to serve as a guide to Internet protocol designers in
determining what they may or should recommend or require of protocol
implementors. Of particular interest in this document are filesystem
protocols.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on March 24, 2014.
Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
Williams Expires March 24, 2014 [Page 1]
Internet-Draft I18N Boundary Analysis September 2013
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction and Motivation . . . . . . . . . . . . . . . 3
1.1. Conventions used in this document . . . . . . . . . . . . 3
2. Internationalization . . . . . . . . . . . . . . . . . . . 4
2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4
3. Filesystems and Remote/Distributed Filesystem Protocols . 6
3.1. On Filesystem Client and Server Implementation
Architectures and their Relevance . . . . . . . . . . . . 6
3.2. Obvious Boundaries . . . . . . . . . . . . . . . . . . . . 6
3.3. Legacy . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3.1. Legacy Problem #1: Loss of Metadata at the System Call
Boundary . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3.2. Legacy Problem #2: Unknown Character String Metadata
in Existing Filesystem Content . . . . . . . . . . . . . . 7
3.3.3. Legacy Problem #3: Poor Handling of Unicode
Equivalence (Normalization) . . . . . . . . . . . . . . . 8
3.3.4. Legacy Problem #4: Ignored Requirements . . . . . . . . . 8
3.3.5. Legacy Problem #5: Constraints Imposed by Non-Internet
Standards . . . . . . . . . . . . . . . . . . . . . . . . 8
3.4. A World Without Legacy . . . . . . . . . . . . . . . . . . 8
3.5. Coping with / Accepting Legacy . . . . . . . . . . . . . . 9
3.5.1. Implications . . . . . . . . . . . . . . . . . . . . . . . 9
3.6. Recommendations for Filesystem Protocols, Filesystems,
and Operating Systems . . . . . . . . . . . . . . . . . . 10
3.7. Interoperability Considerations for Filesystem
Protocols . . . . . . . . . . . . . . . . . . . . . . . . 11
4. Security Considerations . . . . . . . . . . . . . . . . . 12
5. IANA Considerations . . . . . . . . . . . . . . . . . . . 13
6. References . . . . . . . . . . . . . . . . . . . . . . . . 14
6.1. Normative References . . . . . . . . . . . . . . . . . . . 14
6.2. Informative References . . . . . . . . . . . . . . . . . . 14
Author's Address . . . . . . . . . . . . . . . . . . . . . 15
Williams Expires March 24, 2014 [Page 2]
Internet-Draft I18N Boundary Analysis September 2013
1. Introduction and Motivation
As the IETF has attempted to internationalize Internet protocols we
have learned some valuable lessons. It is time to write these down.
This document focuses on internationalization problems in the
filesystem and remote / distributed filesystem protocols space.
This document is INFORMATIVE.
1.1. Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
Where RFC2119 key words are used herein for stating requirements or
recommendations, they are used to as part of suggested normative
language to be used by normative Internet protocol specifications
that accept the internationalization advice given in this document.
Williams Expires March 24, 2014 [Page 3]
Internet-Draft I18N Boundary Analysis September 2013
2. Internationalization
Internationalizing a protocol roughly requires the following tasks:
1. decide where to use Unicode [XXX add reference] and what encoding
of Unicode
2. decide where any conversions to other codesets should be done, if
any
3. decide what Unicode characters (and non-characters) to permit or
forbid
4. decide what Unicode character mappings are appropriate
5. decide how to handle string equality, including case-sensitive
and case-insensitive behavior, and whether and how to handle
Unicode equivalence (normalization)
In practice, because historically most protocols and data formats do
not tag strings with any language nor codeset information, and
because codesets and their encodings often overlap, and other legacy
problems, there's no simple way to decide where to perform any
conversions, mappings, or checks.
We describe here our experience with NFSv4 in particular and
filesystems in general.
2.1. Terminology
[...]
Some terms used in this document:
just-use-8 Where a program or protocol component accepts character
strings, treating them as arbitrary octet strings, often assuming
that byte values less than 0x80 are US-ASCII, or that specific
byte values are specific US-ASCII characters (e.g., filesystem
path component separators).
just-send-8 Where a program or protocol component sends character
strings without regard as to whether the string's codeset/encoding
are the expected on-the-wire codeset/encoding.
just-use-UTF-8 Where a program or protocol component accepts
character strings that are valid UTF-8 strings withour regard to
normalization.
Williams Expires March 24, 2014 [Page 4]
Internet-Draft I18N Boundary Analysis September 2013
just-send-UTF-8 Where a program or protocol component sends UTF-8
character strings without attempting to normalize or perform any
similar steps (e.g., applying character mappings and/or
prohibitions).
... Define lots more and reference other RFCs...
Williams Expires March 24, 2014 [Page 5]
Internet-Draft I18N Boundary Analysis September 2013
3. Filesystems and Remote/Distributed Filesystem Protocols
Filesystems and filesystem protocols may be the most difficult
application to internationalize that we in the IETF have seen to
date. Initially, for NFSv4 [RFC3530] we believed that we could
simply mandate the use of UTF-8 [RFC3629], forbid some characters,
require a choice of normalization forms, and we'd be done. In
practice it was not so simple.
3.1. On Filesystem Client and Server Implementation Architectures and
their Relevance
To understand the difficulties faced in internationalizing NFSv4 we
need to understand the typical architecture of NFSv4 clients and
servers. We say "typical", but it really is typical: the vast
majority, if not all of the major general-purpose operating systems
in use at this time and over the entire history of NFSv4 share the
architecture that we describe here, differing only in minor details.
Normally the architecture and design of clients and servers would be
of no interest to the IETF: we want neiter to dictate nor be unduly
constrained by such things. In this case architecture and legacy
combine to create unusual problems for filesystem protocols. In this
case we must take implementation architecture into account!
Both, clients and servers, typically have a "kernel" that executes
privileged mode object code and which has a pluggable "virtual
filesystem switch" (VFS) -- an interface that abstracts filesystems
so as to permit support for many different types of filesystems.
Clients, and usually servers, also run user-mode object code -less
privileged than kernel-mode object code- that interfaces with
filesystems by invoking privileged kernel-mode code through well-
defined interfaces ("system calls") that allow the kernel to maintain
privilege separation and isolation. These system calls too present a
common, standard, abstract interface to all filesystems that can be
plugged into the kernel's VFS. Some servers run no user-mode object
code to speak of, running all fileserver protocol implementations in
kernel mode, nonetheless, the architecture is roughly the same for
servers as for clients.
3.2. Obvious Boundaries
Some boundaries are immediately evident:
o the system call layer, between user-mode and kernel mode
o the VFS boundary, between generic kernel object code and specific
filesystem implementations
Williams Expires March 24, 2014 [Page 6]
Internet-Draft I18N Boundary Analysis September 2013
o the network, between the client implementation and the server
implementation
o the VFS again, between the server and the filesystems beneath it
o persistent storage network, between specific filesystem
implementations and persistent storage
Their relevance to I18N will be discussed further below.
3.3. Legacy
Many, perhaps all commonly used general-purpose operating systems,
predate modern internationalization efforts. (Some operating
systems, such as those for mobile devices, are new enough that they
might well pose no legacy I18N issues for filesystems.)
Most such operating systems simply treated character strings as
mostly opaque at many if not all of the boundaries described in
Section 3.2, at most interpreting path component separator
characters, in the process assuming US-ASCII [XXX add reference] as
the lowest common denominator for the purpose of finding path
component separators.
Because these operating systems, filesystem on-disk formats, and
actual on-disk filesystems, predate modern internationalization
efforts, there exist many filesystems with object name strings of
unknown or mixed codesets. Strings, such as object names, in
filesystems are never tagged with codeset information because the
codeset information was and still is usually lost at the system call
boundary. The actual codesets (and encodings) used typically varies
along with user (and system administrator) locale preferences.
3.3.1. Legacy Problem #1: Loss of Metadata at the System Call Boundary
The first and foremost problem, then, is the loss of locale metadata
at the system call boundary. Without fixing this we cannot move to
an all-Unicode world in filesystems protocols.all
3.3.2. Legacy Problem #2: Unknown Character String Metadata in Existing
Filesystem Content
The second most important problem in filesystem internationalization
is the lack of locale (codeset, encoding) metadata for existing
(legacy) filesystem content, specifically file and directory names.
Williams Expires March 24, 2014 [Page 7]
Internet-Draft I18N Boundary Analysis September 2013
3.3.3. Legacy Problem #3: Poor Handling of Unicode Equivalence
(Normalization)
Historically Unicode input methods tend to produce pre-composed
codepoints -- something close to Normalization Form Composed (NFC).
But this is not always so.
Historically most filesystems treat file (and directory) names as
opaque, but at least one filesystem (Apple's HFS+ [XXX add
reference]) assumes UTF-8 and normalizes to Normalization Form
Decomposed (NFD) at object-create and object-lookup time.
This can result in subtle interoperability problems, as two objects
with equivalent names may exist in namespaces (directories) where
names are expected to be unique, or users may fail to input names
that match those that exist in a filesystem.
3.3.4. Legacy Problem #4: Ignored Requirements
The original NFSv4 specification [RFC3530] requires some character
mappings and prohibitions. Most implementations have ignored this
requirement.
3.3.5. Legacy Problem #5: Constraints Imposed by Non-Internet Standards
POSIX [XXX add reference] is one common standard for system call
interfaces to filesystems. Arguably it requires that:
1. applications observe the same file/directory names -when listing
a directory- as they created;
2. no aliases may exist for files/directories that are not
"symlinks" or "hardlinks".
This makes it very difficult to deploy Unicode normalization anywhere
other than the application. But it is not possible to fix every
POSIX application to normalize on create or lookup either!
3.4. A World Without Legacy
If we didn't have the legacy problems described above we could simply
mandate the use of Unicode in one specific encoding (e.g., UTF-8) "in
the middle", with the middle being: from the system call boundary, to
the VFS boundary, as well "on the wire". Any codeset conversions and
Unicode normalization would be performed at the system call boundary
(i.e., on the client) and at the VFS boundary (if, for example, a
filesystem on-disk format requires different codeset/encoding than
the protocol does on the wire).
Williams Expires March 24, 2014 [Page 8]
Internet-Draft I18N Boundary Analysis September 2013
Or perhaps in an ideal world all user applications may run only in
Unicode locales, and must perform explicit codeset conversions when
handling legacy (non-Unicode) data. This ideal is one we will likely
obtain in time, as legacy non-Unicode locales are abandoned, legacy
filesystems cleaned up, and new operating systems (or new versions of
them) take over older ones.
In an ideal world there would be no Unicode normalization problems
because either there would be just one normal form for Unicode or
because all implementations of filesystem clients, servers,
filesystems, and filesystem-using applications, would use a single,
common normal form. In practice this is almost certainly an
impossible ideal.
3.5. Coping with / Accepting Legacy
Legacy abounds. We must cope with it.
First, the IETF can't cause the system call boundary metadata loss
problem to be fixed. The architectures of the relevant operating
systems is such that the simplest fix for that problem is to convert
between the user-mode locale's codeset/encoding and the codeset/
encoding expected by the kernel. But getting such a fix to be
implemented and deployed is difficult for a number of reasons, not
the least of which is its impact on performance (for users using
locales that require conversions), but also complexity: the user-mode
side of system calls can sometimes be in a bootstrapping state during
which I18N object code may not have been loaded yet. The simplest
fix for this problem is to recommend that users use only locales that
use Unicode as the charater repertoire and codeset, preferably with
the encoding expected on the kernel-side of the system call boundary.
The second legacy problem -legacy filesystem content- can be
addressed by requiring manual inspection and repair of legacy
content, but there exist such vast amounts of legacy contents that
this is not a realistic option. There is no fix for the legacy
filesystem content problem.
3.5.1. Implications
Some implications of accepting legacy:
o we may want Unicode in the middle, but sometimes we'll have non-
Unicode content
o we can stop the creation of new non-Unicode content on disk, but
we can't really preclude access to it
Williams Expires March 24, 2014 [Page 9]
Internet-Draft I18N Boundary Analysis September 2013
o normalization-on-create is problematic
o normalization-on-lookup is problematic
o normalization-insensitive lookups are problematic
o ignoring normalization is problematic
With respect to normalization there's no one solution appropriate to
all use cases.
3.6. Recommendations for Filesystem Protocols, Filesystems, and
Operating Systems
o Filesystems SHOULD be configurable to reject object names which
are not valid in the filesystem's chosen Unicode encoding.
This allows filesystems (and their servers) to put a stop to the
rot, except, of course, for non-Unicode strings that happen to
appear as valid Unicode strings due to codeset/encoding aliasing.
o Remote / distributed filesystem protocols _should_ specify the use
of Unicode on the wire, but they should also allow the use of non-
Unicode names, leaving it to the filesystem to decide whether to
accept or reject such names.
* For example, this means that NFSv4 _servers_ SHOULD accept
object names -from clients- which are not valid UTF-8, contrary
to the original NFSv4 specification [RFC3530].
o Remote / distributed filesystem protocols should permit servers to
return non-Unicode object names to clients. This allows servers
to serve legacy non-Unicode content.
* For example, this means that NFSv4 clients SHOULD be prepared
to accept non-UTF-8 names from NFSv4 servers, contrary to the
original NFSv4 specification [RFC3530].
o Filesystem servers should accept object names -from filesystems-
which are not valid in the host operating system's chosen codeset
and encoding for use above the VFS.
o Filesystems SHOULD be configurable as to Unicode normalization,
allowing at least the following two options:
* Normalization-insensitive lookups.
Williams Expires March 24, 2014 [Page 10]
Internet-Draft I18N Boundary Analysis September 2013
* No normalization at all.
o Filesystems MAY be configurable as to Unicode normalization,
allowing these additional options:
* Normalize on create and lookup
o Operating systems SHOULD be configurable as to codeset/encoding
conversions at the system call boundary, allowing these options:
* convert to/from non-Unicode locales' codesets
* no conversion
o Operating systems that do not support codeset/encoding conversions
at the system call boundary SHOULD at least encourage users to use
or switch to using Unicode locales.
3.7. Interoperability Considerations for Filesystem Protocols
[[anchor1: Intent: describe interoperability problems that arise
given current NFSv4 deployments and legacy filesystem contents.]]
Williams Expires March 24, 2014 [Page 11]
Internet-Draft I18N Boundary Analysis September 2013
4. Security Considerations
[[anchor2: Lots to talk about here. For example, aliasing issues
w.r.t. multiple equivalent Unicode forms, and the resulting potential
for confusion.]]
Williams Expires March 24, 2014 [Page 12]
Internet-Draft I18N Boundary Analysis September 2013
5. IANA Considerations
There are no IANA considerations in this document.
Williams Expires March 24, 2014 [Page 13]
Internet-Draft I18N Boundary Analysis September 2013
6. References
6.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
6.2. Informative References
[RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "Network File System
(NFS) version 4 Protocol", RFC 3530, April 2003.
[RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
10646", STD 63, RFC 3629, November 2003.
Williams Expires March 24, 2014 [Page 14]
Internet-Draft I18N Boundary Analysis September 2013
Author's Address
Nicolas Williams
Cryptonector, LLC
Email: nico@cryptonector.com
Williams Expires March 24, 2014 [Page 15]
