| Label Generation Rules (lager) | K. Davies |
| Internet-Draft | ICANN |
| Intended status: Standards Track | A. Freytag |
| Expires: June 11, 2016 | ASMUS Inc. |
| December 9, 2015 |
Representing Label Generation Rulesets using XML
draft-ietf-lager-specification-05
This document describes a method of representing rules for validating identifier labels and alternate representations of those labels using Extensible Markup Language (XML). These policies, known as "Label Generation Rulesets" (LGRs), are used for the implementation of Internationalized Domain Names (IDNs), for example. The rulesets are used to implement and share that aspect of policy defining which labels and specific Unicode code points are permitted for registrations, which alternative code points are considered variants, and what actions may be performed on labels containing those variants.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 11, 2016.
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
This memo describes a method of using Extensible Markup Language (XML) to describe the algorithm used to determine whether a given identifier label is permitted, and under which conditions, based on the code points it contains and their context. These algorithms are comprised of a list of permissible code points, variant code point mappings, and a set of rules acting on them. These algorithms form part of an administrator's policies, and can be referred to as Label Generation Rulesets (LGRs). In deploying internationalized domain names (IDNs), they have also been known as IDN tables or variant tables.
There are other kinds of policies relating to labels which are not normally covered by Label Generation Rulesets and are therefore not representable by the XML format described here. These include, but are not limited to policies around trademarks, or prohibition of fraudulent or objectionable words.
Administrators of the zones for top-level domain registries have historically published their LGRs using ASCII text or HTML. The formatting of these documents has been loosely based on the format used for the Language Variant Table described in [RFC3743]. [RFC4290] also provides a "model table format" that describes a similar set of functionality. Common to these formats is that the algorithms used to evaluate the data therein are implicit or specified elsewhere.
Through the first decade of IDN deployment, experience has shown that LGRs derived from these formats are difficult to consistently implement and compare due to their differing formats. A universal format, such as one using a structured XML format, will assist by improving machine-readability, consistency, reusability and maintainability of LGRs.
When used to represent simple list of permitted code points, the format is quite straightforward. At the cost of some complexity in the resulting file, it also allows for an implementation of more sophisticated handling of conditional variants that reflects the known requirements of current zone administrator policies.
Another feature of this format is that it allows many of the algorithms to be made explicit and machine implementable. A remaining small set of implicit algorithms is described in this document to allow commonality in implementation.
While the predominant usage of this specification is to represent IDN label policy, the format is not limited to IDN usage and may also be used for describing ASCII domain name label rulesets, or other types of identifier labels beyond those used for domain names.
The following goals informed the design of this format:
These requirements are partly derived from reviewing the existing corpus of published IDN tables, plus the requirements of ICANN's work to implement an LGR for the DNS Root Zone [LGR-PROCEDURE]. In particular, Section B of that document identifies five specific requirements for an LGR methodology.
The syntax and rules in [RFC5892] and [RFC3743] were also reviewed.
It is explicitly not the goal of this format to stipulate what code points should be listed in an LGR by a zone administrator. Which registration policies are used for a particular zone is outside the scope of this memo.
An LGR is expressed as a well-formed XML Document [XML].
The XML Namespace URI is "urn:ietf:params:xml:ns:lgr-1.0". [Note: the examples and schemas for any non-final versions of this specification use a namespace that is not guaranteed. Early implementors should consider the need to revise the namespace in subsequent revisions.]
See Section 10.2 for more information.
The basic XML framework of the document is as follows:
<?xml version="1.0"?>
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
...
</lgr>
The "lgr" element contains up to three sub-elements. First is an optional "meta" element that contains all meta-data associated with the LGR, such as its authorship, what it is used for, implementation notes and references. This is followed by a "data" element that contains the substantive code point data. Finally, an optional "rules" element contains information on contextual and whole-label evaluation rules, if any, along with any specific "action" elements providing for the disposition of labels and computed variant labels.
<?xml version="1.0"?>
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
<meta>
...
</meta>
<data>
...
</data>
<rules>
...
</rules>
</lgr>
A document MUST contain exactly one "lgr" element. Each "lgr" element MUST contain exactly one "data" element, optionally preceded by one "meta" element and optionally followed by one "rules" element.
In the following descriptions, required, non-repeating elements or attributes are generally not called out explicitly, in contrast to optional ones or those that may be repeated. For attributes that take lists as values the elements are space-delimited.
The optional "meta" element is used to express meta-data associated within the LGR. It can be used to identify the author or relevant contact person, explain the intended usage of the LGR, and provide implementation notes as well as references. With the exception of "unicode-version" element, the data contained within is not required by software consuming the LGR in order to calculate valid labels, or to calculate variants. The "unicode-version" element MUST be used by a consumer of the table to identify that it has the correct Unicode property data to perform operations on the table.
The "version" element is optional. It is used to uniquely identify each version of the LGR. No specific format is required, but it is RECOMMENDED that it be the decimal representation of a single positive integer, which is incremented with each revision of the file.
An example of a typical first edition of a document:
<version>1</version>
The "version" element may have an optional "comment" attribute.
<version comment="draft">1</version>
The optional "date" element is used to identify the date the LGR was posted. The contents of this element MUST be a valid ISO 8601 "full-date" string as described in [RFC3339].
Example of a date:
<date>2009-11-01</date>
The optional "language" element signals that the LGR is associated with a specific language or script. The value of the "language" element MUST be a valid language tag as described in [RFC5646]. The tag may refer to a script plus undefined language if the LGR is not referring to a specific language.
Example of an English language LGR:
<language>en</language>
If the LGR applies to a specific script, rather than a language, the "und" language tag should be used followed by the relevant [RFC5646] script subtag. For example, for a Cyrillic script LGR:
<language>und-Cyrl</language>
If the LGR covers a specific set of multiple languages or scripts, the "language" element can be repeated. However, for cases of a script-specific LGR exhibiting insignificant admixture of code points from other scripts, it is RECOMMENDED to use a single "language" element identifying the predominant script. In the exceptional case of a multi-script LGR where no script is predominant, use Zyyy (Common):
<language>und-Zyyy</language>
Note that that for the particular case of Japanese, a script tag "Jpan" exists that matches the mixture of scripts used in writing that language. The preferred "language" element would be:
<language>und-Jpan</language>
This optional element refers to a scope, such as a domain, to which this policy is applied. The "type" attribute specifies the type of scope being defined. A type of "domain" means that the scope is a domain that represents the apex of the DNS zone to which the LGR is applied. For that type, the content of the "scope" element MUST be a valid absolute domain name, without a trailing dot. However, the DNS root zone is represented as ".".
<scope type="domain">example.com</scope>
There may be multiple "scope" tags used, for example to reflect a list of domains to which the LGR is applied. Types of scope other than "domain" are application-defined. Such application-defined type values must be in a namespace "apptype" previously defined in the document. An explanation of the application-defined type in the "description" element is RECOMMENDED.
The "description" element is an optional free-form element that contains any additional relevant description that is useful for the user in its interpretation. Typically, this field contains authorship information, as well as additional context on how the LGR was formulated and how it applies, such as citations and references that apply to the LGR as a whole.
This field should not be relied upon for providing instructions on how to parse or utilize the data contained elsewhere in the specification. Authors of tables should expect that software applications that parse and use LGRs will not use the description field to condition the application of the LGR's data and rules.
The element has an optional "type" attribute, which refers to the internet media type of the enclosed data. Typical types would be "text/plain" or "text/html". The attribute SHOULD be a valid MIME type. If supplied, it will be assumed that the contents are of that media type. If the description lacks a type field, it will be assumed to be plain text ("text/plain").
The "validity-start" and "validity-end" elements are optional elements that describe the time period from which the contents of the LGR become valid (i.e. are used in registry policy), and the contents of the LGR cease to be used.
The dates MUST confirm to the "full-date" format described in section 5.6 of [RFC3339].
<validity-start>2014-03-12</validity-start>
Whenever an LGR depends on character properties from a given version of the Unicode standard, the version number used in creating the LGR MUST be listed in the form x.y.z, where x, y, and z are positive, decimal integers (see [Unicode-Versions]). If any software processing the table does not have access to character property data of the requisite version, it MUST NOT perform any operations relating to whole-label evaluation relying on Unicode properties (Section 5.2.3).
The value of a given Unicode property in [UAX42] may change between versions, unless such change has been explicitly disallowed in [Unicode-Stability]. It is RECOMMENDED to only reference properties defined as stable or immutable. As an alternative to referencing the property, the information can be presented explicitly in the LGR.
<unicode-version>6.2.0</unicode-version>
It is not necessary to include a "unicode-version" element for LGRs that do not make use of Unicode properties, however, it is RECOMMENDED.
A Label Generation Ruleset may define a list of references which are used to associate various individual elements in the LGR to one or more normative references. A common use for references is to annotate that code points belong to an externally defined collection or standard, or to give normative references for rules.
References are specified in an optional "references" element contains any number of "reference" elements, each with a unique "id" attribute. It is RECOMMENDED that the "id" attribute be a zero-based integer, however, in addition to digits 0-9, it MAY contain uppercase letters A-Z, as well as period, hyphen, colon or underscore. The value of each "reference" element SHOULD be the citation of a standard, dictionary or other specification in any suitable format. In addition to an "id" attribute, a "reference" element may have a "comment" attribute for an optional free-form annotation.
<references>
<reference id="0">The Unicode Standard, Version 7.0</reference>
<reference id="1">Big-5: Computer Chinese Glyph and Character
Code Mapping Table, Technical Report C-26, 1984</reference>
<reference id="2" comment="synchronized with Unicode 6.1">
ISO/IEC
10646:2012 3rd edition</reference>
...
</references>
...
<data>
<char cp="0620" ref="0 2" />
...
</data>
A reference is associated with an element by using its id as part of an optional "ref" attribute (see Section 4.4.1). The "ref" attribute may be used with many kinds of elements in the "data" or "rules" sections of the LGR, most notably those defining code points, variants and rules. However, a "ref" attribute may not occur on certain kinds of elements, including references to nameed character classes or rules. See description of these elements below.
The bulk of a label generation ruleset is a description of which set of code points are eligible for a given label. For rulesets that perform operations that result in potential variants, the code point-level relationships between variants need to also be described.
The code point data is collected within the "data" element. Within this element, a series of "char" and "range" elements describe eligible code points, or ranges of code points, respectively.
Discrete permissible code points or code point sequences are declared with a "char" element, e.g.
<char cp="002D"/>
Ranges of permissible code points may be stipulated with a "range" element, e.g.
<range first-cp="0030" last-cp="0039"/>
The range is inclusive of the first and last code points. All attributes defined for a "range" element act as if applied to each code point within. A "range" element has no child elements.
It is always possible to substitute a list of individually specified code points for a range element. The reverse is not necessarily the case. Whenever such a substitution is possible, it makes no difference in processing the data. Tools reading or writing the LGR format are free to aggregate sequences of consecutive code points of the same properties into range elements.
Code points must be expressed in uppercase, hexadecimal, and zero padded to a minimum of 4 digits. In other words, they are represented according to the standard Unicode convention but without the prefix "U+". The rationale for not allowing other encoding formats, including native Unicode encoding in XML, is explored in [UAX42]. The XML conventions used in this format, including the element and attribute names, mirror this document where practical and reasonable to do so. It is RECOMMENDED to list all "char" elements in ascending order of the "cp" attribute.
All "char" elements in the data section MUST have distinct "cp" attributes. The "range" elements MUST NOT specify code point ranges that overlap either another range or any single code point "char" elements.
A sequence of two or more code points may be specified in an LGR, for example, when defining the source for n:m variant mappings. Another use of sequences would be in cases when the exact sequence of code points is required to occur in order for the constituent elements to be eligible, such as when a specific code point is only eligible when preceded or followed by another code point. The following would define the eligibility of the MIDDLE DOT (U+00B7) only when both preceded and followed by the LATIN SMALL LETTER L (U+006C):
<char cp="006C 00B7 006C" comment="Catalan middle dot"/>
All sequences defined this way must be distinct, but sub-sequences may be defined. Thus, the sequence defined here may coexist with single code point definitions such as:
<char cp="006C" />
As an alternative to using sequences to define a required context, a "char" or "range" element may specify conditional context using an optional "when" attribute as described below in Section 4.2. The latter method is more flexible in that such conditional context is not limited to specific code point in addition to allowing both prohibited as well as required context to be specified.
As described below, the "char" element, whether or not it is used for a single code point, or for a sequence, may have optional child elements defining variants. Both the "char" and "range" elements can take a number of optional attributes for conditional inclusion, commenting, cross referencing and character tagging, as described below.
A conditional context is specified by a rule that must be satisfied (or alternatively, must not be satisfied) for a code point in a given label, often at a particular location in a label.
To specify a conditional context a "when" or "not-when" attribute may be used. The value of each "when" or "not-when" attributes is a whole label or parameterized context rule as described below in Section 5.3. The content condition is met when the rule specified in the "when" attribute is matched. Alternatively, a "not-when" attribute may be used for a rule that must not be matched. It is an error to reference a rule that is not actually defined in the "rules" element.
A parameterized context rule (see Section 5.4) defines the context immediately surrounding a given code point; unlike a sequence, the context is not limited to a specific fixed code point, but for example may be a code point that is a member of a certain subset or has a certain Unicode property.
Given a suitable definition of a parameterized context rule named "follows-virama" this example specifies that a ZERO-WIDTH JOINER (U+200D) is restricted to immediately follow any of several code points classified as virama:
<char cp="200D" when="follows-virama" />
For a complete example, see Appendix A.
In contrast, a whole label rule (see Section 5.3) specifies a condition to be met by the entire label, for example that it must contain at least one code point from a given script anywhere in the label. In the following example no digit from either range must occur in a label that mixes digits from both ranges:
<data>
<range first-cp="0660" last-cp="0669" not-when="mixed-digits"
tag="arabic-indic-digits" />
<range first-cp="06F0" last-cp="06F9" not-when="mixed-digits"
tag="extended-arabic-indic-digits" />
</data>
(See Section 5.3.9 for an example of the "mixed-digits" rule.)
The "when" or "not-when" attributes are mutually exclusive. They may be applied to both "char" and "range" elements in the "data" element, as well as to "var" elements (see Section 4.3.5).
If a contextual condition is not satisfied for any code point in a label, the label is invalid, see Section 6.5. For variants, the conditional context restricts the definition of the variant to the case where the condition is met. Outside the specified context, a variant is not defined .
Most LGRs typically only determine simple code point eligibility, and for them, the elements described so far would be the only ones required for their "data" section. Others additionally specify a mapping of code points to other code points, known as "variants". What constitutes a variant code point is a matter of policy, and varies for each implementation. The following examples are intended to demonstrate the syntax; they are not necessarily typical.
Variant code points are specified using one of more "var" elements as children of a "char" element. The target mapping is specified using the "cp" attribute. Other, optional attributes for the "var" element are described below.
For example, to map LATIN SMALL LETTER V (U+0076) as a variant of LATIN SMALL LETTER U (U+0075):
<char cp="0075">
<var cp="0076"/>
</char>
A sequence of multiple code points can be specified as a variant of a single code point. For example, the sequence of LATIN SMALL LETTER O (U+006F) then LATIN SMALL LETTER E (U+0065) might hypothetically be specified as a variant for an LATIN SMALL LETTER O WITH DIAERESIS (U+00F6) as follows:
<char cp="00F6">
<var cp="006F 0065"/>
</char>
The source and target of a variant mapping may both be sequences, but not ranges.
If the source of one mapping is a prefix sequence of the source for another, both variant mappings will be considered at the same location in the input label when generating permuted variant labels. If poorly designed, an LGR containing such an instance of a prefix relation could generate multiple instances of the same variant label for the same original label, but with potentially different dispositions. Any duplicate variant labels encountered MUST be treated as an error (see Section 7.4).
The "var" element specifies variant mappings in only one direction, even though the variant relation is usually considered symmetric, that is, if A is a variant of B then B should also be a variant of A. The format requires that the inverse of the variant be given explicitly to fully specify symmetric variant relations in the LGR. This has the beneficial side effect of making the symmetry explicit:
<char cp="006F 0065">
<var cp="00F6"/>
</char>
Variant relations are normally not only symmetric, but also transitive. If A is a variant of B and B is a variant of C, then A is also a variant of C. As with symmetry, these transitive relations are spelled out explicitly in the LGR.
All variant mappings are unique. For a given "char" element all "var" elements MUST have a unique combination of "cp", "when" and "not-when" attributes. It is RECOMMENDED to list the "var" elements in ascending order of their target code point sequence. (For "when" and "not-when" attributes, see Section 4.3.5).
Variants may be tagged with an optional "type" attribute. The value of the "type" attribute may be any non-empty value not starting with an underscore and not containing spaces. This value is used to resolve the disposition of any variant labels created using a given variant. (See Section 6.2.)
By default, the values of the "type" attribute directly describe the target policy status (disposition) for a variant label that was generated using a particular variant, with any variant label being assigned a disposition corresponding to the most restrictive variant type. Several conventional disposition values are predefined below in Section 6. Whenever these values can represent the desired policy, they SHOULD be used.
<char cp="767C">
<var cp="53D1" type="allocatable"/>
<var cp="5F42" type="blocked"/>
<var cp="9AEA" type="blocked"/>
<var cp="9AEE" type="blocked"/>
</char>
By default, if a variant label contains any instance of one of the variants of type "blocked" the label would be blocked, but if it contained only instances of variants to be allocated it could be allocated. See the discussion about implied actions in Section 6.6.
The XML format for the LGR makes the relation between the values of the "type" attribute on variants and the resulting disposition of variant labels fully explicit. See the discussion in Section 6.2. Making this relation explicit allows a generalization of the "type" attribute from directly reflecting dispositions to a more differentiated intermediate value that used in the resolution of label disposition. Instead of the default action of applying the most restrictive disposition to the entire label, such a generalized resolution can be used to achieve additional goals, such as limiting the set of allocated variant labels, or to implement other policies found in existing LGRs (see for example Appendix B).
Because variant mappings MUST be unique, it is not possible to define the same variant for the same "char" element with different type attributes (see however Section 4.3.5).
A null variant is a variant string that maps to no code point. This is used when a particular code point sequence is considered discretionary in the context of a whole label. To specify a null variant, use an empty cp attribute. For example, to mark a string with a ZERO WIDTH NON-JOINER (U+200C) to the same string without the ZERO WIDTH NON-JOINER:
<char cp="200C">
<var cp=""/>
</char>
This is useful in expressing the intent that some code points in a label are to be mapped away when generating a canonical variant of the label. However, in tables that are designed to have symmetric variant mappings, this could lead to combinatorial explosion, if not handled carefully.
The symmetric form of a null variant is expressed as follows:
<char cp="">
<var cp="200C" type="invalid" />
</char>
A "char" element with an empty "cp" attribute MUST specify at least one variant mapping. It is strongly RECOMMENDED to use a type of "invalid" or equivalent when defining variant mappings from null sequences, so that variant mapping from null sequences are removed in variant label generation (see Section 4.3.2).
At first sight there seems to be no call for adding variant mappings for which source and target code points are the same, that is for which the mapping is reflexive, or, in other words, an identity mapping. Yet such reflexive mappings occur frequently in LGRs that follow [RFC3743].
Adding a "var" element allows both a type and a reference id to be specified for it. While the reference id is not used in processing, the type of the variant can be used to trigger actions. In permuting the label to generate all possible variants, the type associated with a reflexive variant mapping is applied to any of the permuted labels containing the original code point.
In the following example, the code point U+3473 exists both as a variant of U+3447 and as a variant of itself (reflexive mapping). Assuming an original label of "U+3473 U+3447", the permuted variant "U+3473 U+3473" would consist of the reflexive variant of U+3473 followed by a variant of U+3447. Accordingly, the types for both of the variant mappings used to generate that particular permutation would have the value "preferred" given the following definitions of variant mappings:
<char cp="3447" ref="0">
<var cp="3473" type="preferred" ref="1 3" />
</char>
<char cp="3473" ref="0">
<var cp="3447" type="blocked" ref="1 3" />
<var cp="3473" type="preferred" ref="0" />
</char>
Having established the variant types in this way, a set of actions could be defined that return a disposition of "allocatable" or "activated" for a label consisting exclusively of variants with type "preferred" for example. (For details on how to define actions based on variant types see Section 6.2.1.)
In general, using reflexive variant mappings in this manner makes it possible to calculate disposition values using a uniform approach for all labels, whether they consist of mapped variant code points, original code points, or a mixture of both. In particular, the dispositions for two otherwise identical labels may differ based on which variant mappings were executed in order to generate each of them. (For details on how to generate variants and evaluate dispositions, see Section 7.)
Another useful convention that uses reflexive variants is described below in Section 6.2.1.
Fundamentally, variants are mappings between two sequences of code points. However, in some instances for a variant relationship to exist, some context external to the code point sequence must also be considered. For example, a positional context may determine whether two code point sequences are variants of each other.
An example of that are Arabic code points which can have different forms based on position, with some code points sharing forms, thus making them variants in the positions corresponding to those forms. Such positional context cannot be solely derived from the code point by itself, as the code point would be the same for the various forms.
As described in Section 4.2 a "when" or "not-when" attribute may be given for any "var" element to specify required or prohibited contextual conditions under which the variant defined.
Assuming the "rules" element contains suitably defined rules for "arabic-isolated" and "arabic-final", the following example shows how to mark ARABIC LETTER ALEF WITH WAVY HAMZA BELOW (U+0673) as a variant of ARABIC LETTER ALEF WITH HAMZA BELOW (U+0625), but only when it appears in its isolated or final forms:
<char cp="0625">
<var cp="0673" when="arabic-isolated"/>
<var cp="0673" when="arabic-final"/>
</char>
While only a single "when" or "not-when" attribute can be applied to any "var" element, multiple "var" elements using the same mapping, but different "when" or "not-when" attributes may be specified. The combination of mapping and conditional context defines a unique variant.
Care must be taken to ensure that for each variant label at most one of the contextual conditions is met for variants with the same mapping; otherwise duplicate variant labels would be created for the same input label. Any such duplicate variant labels MUST be treated as an error, see Section 7.4.
Two contexts may be complementary, as in the following example, which shows ARABIC LETTER TEH MARBUTA (U+0629) as a variant of ARABIC LETTER ALEF MAKSURA (U+0649), but with two different types.
<char cp="0647" >
<var cp="0629" not-when="arabic-final" type="blocked" />
<var cp="0629" when="arabic-final" type="allocatable" />
</char>
The intent is that in final position a label that uses U+0629 instead of U+0647 should be considered essentially the same label and therefore allocatable to the same entity, while the same substitution in non-final context leads to labels that are different, but considered confusable so that either one, but not both should be delegatable.
For symmetry, the reverse mappings must exist, and must agree in their "when" or "not-when" attributes. However, symmetry does not apply to the other attributes. For example, these are potential reverse mappings for the above:
<char cp="0629" >
<var cp="0647" not-when="arabic-final" type="allocatable" />
<var cp="0647" when="arabic-final" type="allocatable" />
</char>
Here, both variants have the same "type" attribute. While it is tempting to recognize that in this instance the "when" and "not-when" attributes are complementary and therefore between them cover every single possible context, it is STRONGLY RECOMMENDED to use the format shown in the example that makes the symmetry easily verifiable by parsers and tools. (The same applies to entries created for transitivity.)
Arabic is an example of a script for which such conditional variants have been implemented based on the joining contexts for Arabic code points. The mechanism defined here supports other forms of conditional variants that may required by other scripts.
Two attributes, the "ref" and "comment" attributes, can be used to annotate individual elements in the LGR. They are ignored in machine-processing or the LGR. The "ref" attribute is intended for formal annotations and the "comment" attribute for free form annotations. The latter can be applied more widely.
Reference information may optionally be specified by a "ref" attribute, consisting of a space delimited sequence of reference identifiers (see Section 3.3.8).
<char cp="522A" ref="0">
<var cp="5220" ref="2 3"/>
<var cp="5220" ref="2 3"/>
</char>
This facility is typically used to give source information for code points or variant relations. This information is ignored when machine-processing an LGR. If applied to a range the "ref" attribute applies to every code point in the range. All reference identifiers MUST be from the set declared in the "references" element (see Section 3.3.8). It is an error to repeat a reference identifier in the same "ref" attribute. It is RECOMMENDED that identifiers be listed in ascending order.
In addition to "char", "range" and "var" elements in the data section, a "ref" attribute may be present for a number of elements types contained in the "rules" element as described below: actions, literals ("char" inside a rule), as well as for definitions of rules and classes, but not for references to named character classes or rules using the "by-ref" attribute defined below. (The use of the "by-ref" and "ref" attributes is mutually exclusive.) None of the elements in the metadata take a "ref" attribute; to provide additional information use the "description" element instead.
Any "char", "range" or "variant" element in the data section may contain an optional "comment" attribute. The contents of a "comment" attribute are free-form plain text. Comments are ignored in machine processing of the table. Comment attributes may also be placed on all elements in the "rules" section of the document, such as actions and match operators, such as literals ("char"), as well as definitions of classes and rules, but not on child elements of the "class" element. Finally, in the metadata, only the "version" and "reference" elements may have "comment" attributes (to match the syntax in [RFC3743]).
Typically, LGRs are used to explicitly designate allowable code points, where any label that contains a code point not explicitly listed in the LGR is considered an ineligible label according to the ruleset.
For more complex registry rules, there may be a need to discern one or more subsets of code points. This can be accomplished by applying an optional "tag" attribute to "char" or "range" elements that are child elements of the "data" element. By collecting code points that share the same tag value, character classes may be defined (see Section 5.2.2) which can then be used in whole label evaluation rules (see Section 5.3.2).
Each "tag" attribute may contain multiple values separated by white space. A tag value is an identifier, which may also include certain punctuation marks, such as colon. Formally, it MUST correspond to the XML 1.0 Nmtoken (Name token) production. It is an error to duplicate a value within the same "tag" attribute. A "tag" attribute for a "range" element applies to all code points in the range. Because code point sequences are not proper members of a set of code points, a "tag" attribute MUST NOT be present in a "char" element defining a code point sequence.
The code points in a label sometimes need to satisfy context-based rules, for example for the label to be considered valid, or to satisfy the context for a variant mapping (see the description of the "when" attribute in Section 5.4).
A Whole Label Evaluation rule (WLE) is applied to the whole label. It is used to validate both original labels and variant labels computed from them using a permutation over all applicable variant mappings. A conditional context rules is a specialized form of WLE specific to the context around a single code point or code point sequence. For example, if a rule is referenced in the "when" attribute of a variant mapping it is used to describe the conditional context under which the particular variant mapping is defined to exist.
Each rule is defined in a "rule" element. A rule may contain the following as child elements:
Collectively, these are called match operators and are listed in Section 5.3.2.
Character classes are sets of characters that often share a particular property. While they function like sets in every way, even supporting the usual set operators, they are called character classes here in a nod to the use of that term in regular expression syntax. (This also avoids confusion with the term "character set" in the sense of character encoding.)
Character classes (or sets) can be specified in several ways:
A character class has an optional "name" attribute, consisting of a single, identifier not containing spaces. All names for classes must be unique. If the "name" attribute is omitted, the class is anonymous and exists only inside the rule or combined class where it is defined. A named character class is defined independently and can be referenced by name from within any rules or as part of other character class definitions.
<class name="example" comment="an example class definition">
<char cp="0061" />
<char cp="4E00" />
</class>
...
<rule>
<class by-ref="example" />
</rule>
An empty "class" element with a "by-ref" attribute is a reference to an existing named class. The "by-ref" attribute cannot be used in the same "class" element with any of these attributes: "name", "from-tag", "property" or "ref". The "name" attribute MUST be present, if and only if the class is a direct child element of the "rules" element. It is an error to reference a named class for which the definition has not been seen.
The "char" or "range" elements that are child elements of the "data" element may contain a "tag" attribute that consists of one or more space separated tag values, for example:
<char cp="0061" tag="letter lower"/>
<char cp="4E00" tag="letter"/>
This defines two tags for use with code point U+0061, the tag "letter" and the tag "lower". Use
<class name="letter" from-tag="letter" />
<class name="lower" from-tag="lower" />
to define two named character classes, "letter" and "lower", containing all code points with the respective tags, the first with 0061 and 4E00 as elements and the latter with 0061, but not 4E00 as an element. The "name" attribute may be omitted for an anonymous in-place definition of a nested, tag-based class.
Tag values are typically identifiers, with the addition of a few punctuation symbols, such as colon. Formally they MUST correspond to the XML 1.0 Nmtoken (Name token) production. While a "tag" attribute may contain a list of tag values, the "from-tag" attribute always contains a single tag value.
If the document contains no "char" or "range" elements with a corresponding tag, the character class represents the empty set. This is valid, to allow a common "rules" element to be shared across files. However, it is RECOMMENDED that implementations allow for a warning to ensure that referring to an undefined tag in this way is intentional.
A class is defined in terms of Unicode properties by giving the Unicode property alias and the property value or property value alias, separated by a colon.
<class name="virama" property="ccc:9" />
The example above selects all code points for which the Unicode canonical combining class (ccc) value is 9. This value of the ccc is assigned to all code points that encode viramas. The string "ccc" is the short-alias for the canonical combining class, as defined in the Unicode Character Database [UAX42].
Unicode properties may, in principle, change between versions of the Unicode Standard. However, the values assigned for a given version are fixed. If Unicode Properties are used, a Unicode version MUST be declared in the "unicode-version" element in the header. (Note: some Unicode properties are by definition stable across versions and do not change once assigned (see [Unicode-Stability].)
It is RECOMMENDED that all implementations processing LGR files provide support for the following minimal set of Unicode properties:
The short name for each property is given in parentheses.
If a program that is using an LGR to determine the validity of a label encounters a property that it does not support, it MUST abort with an error.
A class of code points may also be declared by listing the code points that are a member of the class. This is useful when tagging cannot be used because code points are not listed individually as part of the eligible set of code points for the given LGR, for example because they only occur in code point sequences.
To define a class in terms of an explicit list of code points use a space separated list of hexadecimal code point values:
<class name="abcd">0061 0062 0063 0064</class>
This defines a class named "abcd" containing the code points for characters "a", "b", "c" and "d". The ordering of the code points is not material, but it is RECOMMENDED to list them in ascending order.
In a class definition, ranges of code points are represented by a hexadecimal start and end value separated by a hyphen. The following declaration is equivalent to the preceding:
<class name="abcd">0061-0064</class>
Range and code point declarations can be freely intermixed:
<class name="abcd">0061 0062-0063 0064</class>
The contents of a class differ from a repertoire in that the latter may contain sequences as elements, while the former may not. Instead, they closely resemble character classes as found in regular expressions.
Classes may be combined using operators for set complement, union, intersection, difference and symmetric difference (exclusive-or). Because classes fundamentally function like sets, the union of several character classes is itself a class, for example.
+-------------------+----------------------------------------------+ | Logical Operation | Example | +-------------------+----------------------------------------------+ | Complement | <complement><class by-ref="xxx"></complement>| +-------------------+----------------------------------------------+ | Union | <union> | | | <class by-ref="class-1"/> | | | <class by-ref="class-2"/> | | | <class by-ref="class-3"/> | | | </union> | +-------------------+----------------------------------------------+ | Intersection | <intersection> | | | <class by-ref="class-1"/> | | | <class by-ref="class-2"/> | | | </intersection> | +-------------------+----------------------------------------------+ | Difference | <difference> | | | <class by-ref="class-1"/> | | | <class by-ref="class-2"/> | | | </difference> | +-------------------+----------------------------------------------+ | Symmetric | <symmetric-difference> | | Difference | <class by-ref="class-1"/> | | | <class by-ref="class-2"/> | | | </symmetric-difference> | +-------------------+----------------------------------------------+
Set Operators
The elements from this table may be arbitrarily nested inside each other, subject to the following restriction: a "complement" element MUST contain precisely one "class" or one of the operator elements, while an "intersection", "symmetric-difference" or "difference" element MUST contain precisely two, and a "union" element MUST contain two or more of these elements.
An anonymous combined class can be defined directly inside a rule or of the match operator elements that allow child elements (see Section 5.3.2) by using the set combination as the outer element.
<rule>
<union>
<class by-ref="xxx"/>
<class by-ref="yyy"/>
</union>
</rule>
The example shows the definition of an anonymous combined class that represents the union of classes "xxx" and "yyy". There is no need to wrap this union inside another "class" element, and, in fact, set combination elements MUST NOT be nested inside a "class" element.
Lastly, to create a named combined class that can be referenced in other classes or in rules as <class by-ref="xxxyyy"/>, add a "name" attribute to the set combination element, for example <union name="xxxyyy" /> and place it at the top level immediately below the "rules" element (see Section 5.2.1).
<rules>
<union name="xxxyyy">
<class by-ref="xxx"/>
<class by-ref="yyy"/>
</union>
. . .
</rules>
Because (as for ordinary sets) a combination of classes is itself a class, no matter by what combinations of set operators a combined class is created, a reference to it always uses the "class" element as described in Section 5.2.1. That is, a named class is always referenced via an empty "class" element using the "by-ref" attribute containing the name of the class to be referenced.
Each rule is comprised of a series of matching operators that must be satisfied in order to determine whether a label meets a given condition. Rules may reference other rules or character classes defined elsewhere in the table.
A matching rule is defined by a "rule" element, the child elements of which are one of the match operators from Section 5.3.2. In evaluating a rule, each child element is matched in order. Rule elements may be nested.
Rules may optionally be named using a "name" attribute containing a single identifier string with no spaces. A named rule may be incorporated into another rule by reference. If the "name" attribute is omitted, the rule is anonymous and may not be incorporated by reference into another rule or referenced by an action or "when" attribute.
A simple rule to match a label where all characters are members of the class "preferred":
<rule name="preferred">
<start />
<class by-ref="preferred" count="1+"/>
<end />
</rule>
Rules are paired with explicit and implied actions, triggering these actions when a rule matches a label. For example, a simple explicit action for the rule shown above would be:
<action disp="allocatable" match="preferred" />
This has the effect of setting the policy disposition for a label made up entirely of "preferred" code points to "allocatable". Explicit actions are further discussed in Section 6 and the use of rules in conditional contexts for implied actions is discussed in Section 4.3.5 and Section 6.5.
The child elements of a rule are a series of match operators, which are listed here by type and name and with a basic example or two.
+------------+-------------+------------------------------------+ | Type | Operator | Examples | +------------+-------------+------------------------------------+ | logical | any | <any /> | | +-------------+------------------------------------+ | | choice | <choice> | | | | <rule by-ref="alternative1"/> | | | | <rule by-ref="alternative2"/> | | | | </choice> | +--------------------------+------------------------------------+ | positional | start | <start /> | | +-------------+------------------------------------+ | | end | <end /> | +--------------------------+------------------------------------+ | literal | char | <char cp="0061 0062 0063" /> | +--------------------------+------------------------------------+ | set | class | <class by-ref="class1" /> | | | | <class>0061 0064-0065</class> | +--------------------------+------------------------------------+ | group | rule | <rule by-ref="rule1" /> | | | | <rule><any /><rule /> | +--------------------------+------------------------------------+ | contextual | anchor | <anchor /> | | +-------------+------------------------------------+ | | look-ahead | <look-ahead><any /></look-ahead> | | +-------------+------------------------------------+ | | look-behind | <look-behind><any /></look-behind> | +--------------------------+------------------------------------+
Match Operators
Any element defining an anonymous class can be used as a match operator, including any of the set combination operators (see Section 5.2.5) as well as references to named classes.
All match operators shown as empty elements in the Examples column of the table above do not support child elements of their own; otherwise match operators may be nested. In particular, anonymous "rule" elements can be used for grouping.
The optional "count" attribute specifies the minimally required or maximal permitted number of times a match operator is used to match input. If the "count" attribute is
If there is no "count" attribute, the match operator matches the input exactly once.
In matching, greedy evaluation is used in the sense defined for regular expressions: beyond the required number or times, the input is matched as many times as possible, but not so often as to prevent a match of the remainder of the rule.
The optional "count" attribute MUST NOT be applied to any element that contains a "name" attribute, but MAY be applied to operators such as "class" that declare anonymous classes (including combined classes) or invoke any predefined classes by reference. The count attribute MUST not be applied to any "class" element, or element defining a combined class, when it is nested inside a combined class.
The count attribute MUST NOT be applied to match operators of type "start", "end", "anchor", "look-ahead" or "look-behind" or to any operators, such as "rule" or "choice" that contain a nested instance of them. This limitation applies recursively, and irrespective of whether a rule element containing these nested instances is declared in place or used by reference.
However, the "count" attribute MAY be applied to any other instances of either an anonymous "rule" element or of a "choice" element, including those instances nested inside other match operators. It MAY also be applied to the elements "any" and "char", when used as match operators.
Like classes (see Section 5.2.1), rules declared as immediate child elements of the "rules" element MUST be named using a unique "name" attribute, and all other instances MUST NOT be named. Anonymous rules and classes or reference to named rules and classes can be nested inside other match operators by reference.
To reference a named rule or class inside a rule or match operator use a "rule" or "class" element with an optional "by-ref" attribute containing the name of the referenced element. It is an error to reference a rule or class for which the complete definition has not been seen. In other words, it is explicitly not possible to define recursive rules or class definitions. The "by-ref" attribute cannot appear in the same element as the "name" attribute, or in an element that has any child elements.
Here's an example of a rule requiring that all labels be letters (optionally followed by combining marks) and possibly digits. The example shows rules and classes referenced by name.
<class name="letter" property="gc:L"/>
<class name="combining-mark" property="gc:M"/>
<class name="digit" property="gc:Nd" />
<rule name="letter-grapheme">
<class by-ref="letter" count="1+"/>
<class by-ref="combining-mark" count="0+"/>
</rule>
The "choice" element is used to represent a list of two or more alternatives:
<rule name="ldh">
<choice count="1+">
<class by-ref="letters"/>
<class by-ref="digits"/>
<char cp="002D" comment="literal HYPHEN"/>
</choice>
</rule>
Each child element of a "choice" represents one alternative. The first matching alternative determines the match for the "choice" element. To express a choice where an alternative itself consists of a sequence of elements, the sequence must be wrapped in an anonymous rule.
A literal code point sequence matches a single code point or a sequence. It is defined by a "char" element, with the code point or sequence to be matched given by the "cp" attribute. When used as a literal, a "char" element may contain a "count" in addition to the "cp" attribute and optional "comment" or "ref" attributes. No other attributes or child elements are permitted.
The "any" element matches any single code point. It may have a "count" attribute. For an example see Section 5.3.9
Unlike a literal, the "any" element" may not have a "ref" attribute.
To match the beginning or end of a label, use the "start" or "end" element. An empty label would match this rule:
<rule name="empty-label">
<start/>
<end/>
</rule>
Conceptually, Whole Label Evaluation Rules evaluate the label as a whole, but in practice, many rules do not actually need to be specified to match the entire label. For example, to express a requirement of not starting a label with a digit, a rule needs to describe only the initial part of a label.
This example uses the previously defined rules, together with start and end tag, to define a rule that requires that an entire label is well-formed. For this example that means, that it must start with a letter and contains no leading digits or combining marks, nor combining marks placed on digits.
<rule name="leading-letter" >
<start />
<rule by-ref="letter-grapheme" count="1"/>
<choice count="0+">
<rule by-ref="letter-grapheme" count="0+"/>
<class by-ref="digit" count="0+"/>
</choice>
<end />
</rule>
Each "start" or "end" element occurs at most once in a rule, except if nested inside a "choice" element in such a way that in matching each alternative at most one occurrence of each is encountered. Otherwise, the result is an error; as is any case where a "start" or "end" element is not encountered as first or last element to be matched, respectively, in matching a rule. Start and end elements do not have a "count" or any other attribute. It is an error for any match operator enclosing a nested "start" or "end" element to have a "count" attribute.
This is an example of the whole label evaluation rule from [RFC5892] forbidding the mixture of the Arabic-Indic and extended Arabic-Indic digits in the same label. The example also demonstrates several instances of the use of anonymous rules for grouping.
<data>
<range first-cp="0660" last-cp="0669" not-when="mixed-digits"
tag="arabic-indic-digits" />
<range first-cp="06F0" last-cp="06F9" not-when="mixed-digits"
tag="extended-arabic-indic-digits" />
</data>
<rules>
<rule name="mixed-digits">
<choice>
<rule>
<class from-tag="arabic-indic-digits"/>
<any count="0+"/>
<class from-tag="extended-arabic-indic-digits"/>
</rule>
<rule>
<class from-tag="extended-arabic-indic-digits"/>
<any count="0+"/>
<class from-tag="arabic-indic-digits"/>
</rule>
</choice>
</rule>
</rules>
The effect of this example is that a label containing a code point from either of the two digit ranges is invalid for any label matching the "mixed-digits" rule, that is, anytime a code point from the other range is also present. Note that this is not the same as invalidating the definition of the "range" elements.
A special type of rule provides a context for evaluating the validity of a code point or variant mapping. This rule is invoked by the "when" or "not-when" attributes described in Section 4.2. For "char" and "range" elements, an action implied by a context rule always has a disposition of "invalid" whenever the rule given by the "when" attribute is not matched (see Section 6.5). Conversely, a "not-when" attribute results in a disposition of "invalid" whenever the rule is matched.
Such parameterized context or "When Rules" may contain a special place holder represented by an "anchor" element. As each When Rule is evaluated, the "anchor" element is replaced by a literal corresponding to the "cp" attribute of the element containing the "when" (or "not-when") attribute. The match to the "anchor" element must be at the same position in the label as the code point or variant mapping triggering the When Rule.
For example, the Greek lower numeral sign is invalid if not immediately preceding a character in the Greek script. This is most naturally addressed with a When Rule using look-ahead:
<char cp="0375" when="preceding-greek"/>
...
<class name="greek-script" property="sc:Grek"/>
<rule name="preceding-greek">
<anchor/>
<look-ahead>
<class by-ref="greek-script"/>
</look-ahead>
</rule>
In evaluating this rule, the "anchor" element is treated as if it was replaced by a literal
<char cp="0375"/>
but only the instance of U+0375 at the given position is evaluated. If a label had two instances of U+0375 with the first one matching the rule and the second not, then evaluating the When Rule MUST succeed for the first and fail for the second instance.
Unlike other rules, When Rules containing an "anchor" element MUST only be invoked via the "when" or "not-when" attributes on code points or variants; otherwise their "anchor" elements cannot be evaluated. However, it is possible to invoke rules not containing an "anchor" element from a "when" or "not-when" attribute. (See Section 5.4.3)
Context rules use the "look-behind" and "look-ahead" elements to define context before and after the code point sequence matched by the "anchor" element. If the "anchor" element is omitted, neither the "look-behind" nor the "look-ahead" element may be present.
Here is an example of a rule that defines an "initial" context for an Arabic code point:
<class name="transparent" property="jt:T"/>
<class name="right-joining" property="jt:R"/>
<class name="left-joining" property="jt:L"/>
<class name="dual-joining" property="jt:D"/>
<class name="non-joining" property="jt:U"/>
<rule name="Arabic-initial">
<look-behind>
<choice>
<start/>
<rule>
<class by-ref="transparent" count="0+"/>
<class by-ref="non-joining"/>
</rule>
</choice>
</look-behind>
<anchor/>
<look-ahead>
<class by-ref="transparent" count="0+" />
<choice>
<class by-ref="right-joining" />
<class by-ref="dual-joining" />
</choice>
</look-ahead>
</rule>
A "when rule" contains any combination of "look-behind", "anchor" and "look-ahead" elements in that order. Each of these elements occurs at most once, except if nested inside a "choice" element in such a way that in matching each alternative at most one occurrence of each is encountered. Otherwise, the result is undefined. None of these elements takes a "count" attribute, nor does any enclosing match operator. Otherwise, the result is undefined. If a context rule contains a "look-ahead" or "look-behind" element, it MUST contain an "anchor" element. If, because of a choice element, a required anchor is not actually encountered, the results are undefined.
If the "anchor" element is omitted, the evaluation of the context rule is not tied to the position of the code point or sequence associated with the "when" attribute.
According to [RFC5892] Katakana middle dot is invalid in any label not containing at least one Japanese character anywhere in the label. Because this requirement is independent of the position of the middle dot, the rule does not require an "anchor" element.
<char cp="30FB" when="japanese-in-label"/>
<rule name="japanese-in-label">
<union>
<class property="sc:Hani"/>
<class property="sc:Kata"/>
<class property="sc:Hira"/>
</union>
</rule>
The Katakana middle dot is used only with Han, Katakana or Hiragana. The corresponding When Rule requires that at least one code point in the label is in one of these scripts, but the position of that code point is independent of the location of the middle dot and no anchor therefore required. (Note that the Katakana middle dot itself is of script Common, that is, "sc:Zyyy").
The purpose of a rule is to trigger a specific action. Often, the action simply results in blocking or invalidating a label that does not match a rule. An example of an action invalidating a label because it does not match a rule named "leading-letter" is as follows:
<action disp="invalid" not-match="leading-letter"/>
If an action is to be triggered on matching a rule, a "match" attribute is used instead. Actions are evaluated in the order that they appear in the XML file. Once an action is triggered by a label, the disposition defined in the "disp" attribute is assigned to the label and no other actions are evaluated for that label.
The goal of the Label Generation Rules is to identify all labels and variant labels and to assign them disposition values. These dispositions are then fed into a further process that ultimately implements all aspects of policy. To allow this specification to be used with the widest range of policies, the permissible values for the "disp" attribute are neither defined nor restricted. Nevertheless a set of commonly used disposition values is RECOMMENDED. (See Section 6.3)
A "match" or "not-match" attribute specify a rule that must be matched or not matched as a condition for triggering an action. Only a single rule may be named as the value of a "match" or "not-match" attribute. Because rules may be composed of other rules, this restriction to a single attribute value does not impose any limitation on the contexts that can trigger an action.
An action may contain a "match" or a "not-match" attribute, but not both. An action without any attributes is triggered by all labels unconditionally. For a very simple LGR, the following action would allocate all labels that match the repertoire:
<action disp="allocatable" />
Since rules are evaluated for all labels, whether they are the original label or computed by permuting the defined and valid variant mappings for the label's code points, actions based on matching or not matching a rule may be triggered for both original and variant labels, but they the rules are not affected by the disposition attributes of the variant mappings. To trigger any actions base on these dispositions requires the use additional optional attributes for actions described next.
An action may contain one of the optional attributes "any-variant", "all-variants", or "only-variants" defining triggers based on variant types. The permitted value for these attributes consists of one or more variant type values, separated by spaces. When a variant label is generated, these variant type values are compared to the set of type values on the variant mappings used to generate the particular variant label (see Section 7).
Any single match may trigger an action that contains an "any-variant" attribute, while for an "all-variants" or "only-variants" attribute, the variant type for all variant code points must match one or several of the type values specified in the attribute to trigger the action. There is no requirement that the entire list of variant type values be matched, as long as all variant code points match at least one of the values.
An "only-variants" attribute will trigger the action only if all code points of the variant label have variant mappings from the original code points. In other words, the label contains no original code points other than those with a reflexive mapping (see Section 4.3.4).
<char cp="0078" comment="x" />
<var cp="0078" type="allocatable" comment="reflexive" />
<var cp="0079" type="blocked" />
</char>
<char cp="0079" comment="y"/>
<var cp="0078" type="allocatable" />
</char>
. . .
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" only-variants="allocatable" />
<action disp="some-type" any-variant="allocatable" />
In the example above, the label "xx" would have variant labels "xx", "xy", "yx" and "yy". The first action would result in blocking any variant label containing "y", because the variant mapping from "x" to "y" is of type "blocked", triggering the "any-variant" condition. Because in this example "x" has a reflexive variant mapping to itself of type "allocatable" the original label "xx" has a reflexive variant "xx" that would trigger the "only_variants" condition on the second action.
A label "yy" would have the variants "xy", "yx" and "xx". Because the variant mapping from "y" to "x" is of type "allocatable" and a mapping from "y" to "y" is not defined, the labels "xy" and "yx" trigger the "any-variant" condition on the third label. The variant "xx", being generated using the mapping from "y" to "x" of type "allocatable", would trigger the "only-variants" condition on the section action. As there is no reflexive variant "yy", the original label "yy" cannot trigger any variant type triggers. However, it could still trigger an action defined as matching or not matching a rule.
In each action, one variant type trigger may be present by itself or in conjunction with an attribute matching or not-matching a rule. If variant triggers and rule-matching triggers are used together, the label MUST "match" or respectively "not-match" the specified rule, AND satisfy the conditions on the variant type values given by the "any-variant", "all-variants", or "only-variants" attribute.
A useful convention combines the "any-variant" trigger with reflexive variant mappings (Section 4.3.4). This convention is used, for example, when multiple LGRs are defined within the same registry and for overlapping repertoire. In some cases, the delegation of a label from one LGR must prohibit the delegation of another label in some other LGR. This can be done using a variant of type "blocked" as in this example from an Armenian LGR, where the Armenian, Latin and Cyrillic letters all look identical:
<char cp="0570" comment="Armenian small letter HO">
<var cp="0068" type="blocked" comment="Latin small letter H" />
<var cp="04BB" type="blocked"
comment="Cyrillic small letter SSHA" />
</char>
The issue is that the target code points for these two variants are both outside the Armenian repertoire. By using a reflexive variant with the following convention:
<char cp="0068" comment="not part of repertoire">
<var cp="0068" type="out-of-repertoire-var"
comment="reflexive mapping" />
<var cp="04BB" type="blocked" />
<var cp="0570" type="blocked" />
</char>
...
and associating this with an action of the form:
<action disp="invalid" any-variant="out-of-repertoire-var" />
it is possible to list the symmetric and transitive variant mappings in the LGR even where they involve out-of-repertoire code points. By associating the action shown with the special type for these reflexive mappings any original labels containing one or more of the out-of-repertoire code points are filtered out — just as if these code points had not been listed in the LGR in the first place. Nevertheless, they do participate in the permutation of variant labels for n-repertoire labels (Armenian in the example), and these permuted variants can be used to detect collisions with out-of-repertoire labels (see Section 7).
This section gives an example of using variant type triggers, combined with variants with reflexive mappings (Section 4.3.4) to achieve LGRs that implement tables like those defined according to [RFC3743] where the goal is to allow as variants only labels that consist entirely of simplified or traditional variants, in addition to the original label.
Assuming an LGR where all variants have been given suitable "type" attributes of "blocked", "simplified", "traditional", or "both", similar to the ones discussed in Appendix B. Given such an LGR, the following example actions evaluate the disposition for the variant label:
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" only-variants="simplified both" />
<action disp="allocatable" only-variants="traditional both" />
<action disp="blocked" all-variants="simplified traditional " />
<action disp="allocatable" />
The first action matches any variant label for which at least one of the code point variants is of type "blocked". The second matches any variant label for which all of the code point variants are of type "simplified" or "both", in other words an all-simplified label. The third matches any label for which all variants are of type "traditional" or "both", that is all traditional. These two actions are not triggered by any variant labels containing some original code points, unless each of those code points has a variant defined with a reflexive mapping (Section 4.3.4).
The final two actions rely on the fact that actions are evaluated in sequence, and that the first action triggered also defines the final disposition for a variant label (see Section 6.4). They further rely on the assumption that the only variants with type "both" are also reflexive variants.
Given these assumptions, any remaining simplified or traditional variants must then be part of a mixed label, and so are blocked; all labels surviving to the last action are original code points only (that is the original label). The example assumes that an original label may be a mixed label; if that is not the case, the disposition for the last action would be set to "blocked".
There are exceptions where the assumption on reflexive mappings made above does not hold, so this basic scheme needs some refinements to cover all cases. For a more complete example, see Appendix B.
The precise nature of the policy action taken in response to a disposition and the name of the corresponding "disp" attributes are only partially defined here. It is strongly RECOMMENDED to use the following dispositions only with their conventional sense.
Actions are applied in the order of their appearance in the file. This defines their relative precedence. The first action triggered by a label defines the disposition for that label. To define a specific order of precedence, list the actions in the desired order. The conventional order of precedence for the actions defined in Section 6.3 is "invalid", "blocked", "allocatable", then "activated". This default precedence is used for the default actions defined in Section 6.6.
The context rules on code points ("not-when" or "when" rules) carry an implied action with a disposition of "invalid" (not eligible). These rules are evaluated at the time the code points for a label or its variant labels are checked for validity (see Section 7). In other words, they are evaluated before any of the whole-label evaluation rules and with higher precedence. The context rules for variant mappings are evaluated when variants are generated and/or when variant tables are made symmetric and transitive. They have an implied action with a disposition of "invalid" which means a putative variant mapping does not exist whenever the given context matches a "not-when" rule or fails to match a "when" rule specified for that mapping. The result of that disposition is that the variant mapping is ignored in generating variant labels and the value is therefore not accessible to trigger any explicit actions.
Note that such non-existing variant mapping is different from a blocked variant, which is a variant code point mapping that exists but results in a label that may not be allocated.
As described in Section 6 any variant mapping may be given a "type" attribute. An action containing an "any-variant" or "all-variants" attribute relates these type values to a resulting disposition for the entire variant label.
If no actions are defined for the standard disposition values of "invalid", "blocked", "allocatable" and "activated", then the following default actions exist that are shown below in their default order of precedence (see Section 6.4). This default order for evaluating dispositions applies only to labels that triggered no explicitly defined actions and which are therefore handled by default actions. Default actions have a lower order of precedence than explicit actions (see Section 7.3).
The default actions for variant labels are defined as follows:
<action disp="invalid" any-variant="invalid"/>
<action disp="blocked" any-variant="blocked"/>
<action disp="allocatable" any-variant="allocatable"/>
<action disp="activated" all-variants="activate"/>
A final default action sets the disposition to "allocatable" for any label matching the repertoire for which no other action has been triggered (catch-all).
<action disp="allocatable" />
In order to test a specific label for membership in the LGR, a consumer of the LGR must iterate through each code point within a given label, and test that each code point is a member of the LGR. If any code point is not a member of the LGR, the label shall be deemed as invalid.
An individual code point is deemed a member of the LGR when it is listed using a "char" element, or is part of a range defined with a "range" element, and all necessary condition in any "when" or "not-when" attributes are correctly satisfied.
Alternatively, a code point is also deemed a member of the LGR when it forms part of a sequence that corresponds to a sequence listed using a "char" element for which the "cp" attribute defines a sequence, and all necessary condition in any "when" or "not-when" attributes are correctly satisfied.
A label must also not trigger any action that results in a disposition of "invalid", otherwise it is deemed not eligible. (This step may need to be deferred, until variant code point dispositions have been determined).
For LGRs that contain reflexive variant mappings (defined in Section 4.3.4), the final evaluation of eligibility for the label must be deferred until variants are generated. In essence, LGRs that use this feature treat the original label as the (identity) variant of itself. For such LGRs, the ordinary iteration over code points would generally only exclude a subset of invalid labels, but it could be used effectively as a pre-screening.
To check the validity of a label with reflexive mappings, it is not necessary to generate all variant labels. Only a single variant needs to be created, where for each code point, any reflexive variants are applied, and the label disposition is evaluated. A disposition of "invalid" results in the label being not eligible. (In the exceptional case where context rules are present on reflexive mappings, multiple reflexive variants may be defined, but for each original label, at most one of these can be valid at each code position. However, see Section 7.4).
For a given eligible label, the set of variant labels is deemed to consist of each possible permutation of original code points and substituted code points or sequences defined in "var" elements, whereby all "when" and "not-when" attributes are correctly satisfied for each "char" or "var" element in the given permutation and all applicable whole label evaluation rules are satisfied as follows:
The number of potential permutations can be very large. In practice, implementations would use suitable optimizations to avoid having to actually create all permutations.
For a given label (variant or original), its disposition is determined by evaluating in order of their appearance all actions for which the label or variant label satisfies the conditions.
or
For a poorly designed LGR, it is possible to generate duplicate variant labels from the same input label, but with different, and potentially conflicting dispositions. Implementations MUST treat any duplicate variant labels encountered as an error, irrespective of their dispositions.
This situation can arise in two ways. One is described in Section 4.3.5 and involves defining the same variant mapping with two context rules that are formally distinct, but nevertheless overlap so that they are not mutually exclusive for the same label.
The other case involves variants defined for sequences, where one sequence is a prefix of another (see Section 4.3.1). The following shows such an example resulting in conflicting reflexive variants:
<char cp="0061">
<var cp="0061" type="allocatable"/>
</char>
<char cp="0062"/>
<char cp="0061 0062">
<var cp="0061 0062" type="blocked"/>
</char>
A label "ab" would generate the variant labels "{a}{b}" and "{ab}" where the curly braces show the sequence boundaries as they were applied during variant mapping. The result is a duplicate variant label "ab", one based on a variant of type "allocatable" plus an original code point "b" that has no variant, and another one based on a single variant of type "blocked", thus creating two variant labels with conflicting dispositions.
In the general case it is difficult to impossible to prove by mechanical inspection of the LGR that duplicate variant labels will never occur, so implementations have to be prepared to detect this error during variant label generation. The condition is easily avoided by careful design of context rules and special attention to the relation among code point sequences with variants.
The obvious method for checking collision between labels is to generate the fully permuted set of variants for one of them and see whether it contains the other label as a member. As discussed above, this can be prohibitive, and is not necessary.
Because of symmetry and transitivity, all variant mappings form disjoint sets in which each of them is a variant of all other members. As a consequence, the set of variant labels likewise forms disjoint subsets, based on which set of mappings was used.
Instead of generating all permutations, that is, use each variant mapping in each set, it is sufficient to substitute an "index" mapping, in effect identifying the set of variant code points. Such an index mapping could be, for example, the variant mapping for which the target code point (or sequence) comes first n some sorting order.
To check collision then means generating a single variant label from the original, by substituting the "index" value as the target for mapping from any code point. This results in an "index label". Two labels collide whenever the index labels for them are the same.
Both [RFC3743] and [RFC4290] provide different grammars for IDN tables. These formats are unable to fully cater for the increased requirements of contemporary IDN variant policies.
This specification is a superset of functionality provided by these IDN table formats, thus any table expressed in those formats can be expressed in this format. Automated conversion can be conducted between tables conformant with the grammar specified in each document.
For notes on how to translate an RFC 3743-style table, see Appendix B.
Transmission of a well-formed LGR in accordance with this specification SHOULD be transmitted with a media type of "application/lgr+xml". This media type will signal to an LGR-aware client that the content is designed to be interpreted as an LGR.
This document requests the following actions from IANA:
The media type "application/lgr+xml" should be registered to denote transmission of label generation rulesets that are compliant with this specification, in accordance with [RFC6838].
Type name: application
Subtype name: lgr+xml
Required parameters: None.
Optional parameters: charset (as for application/xml per [RFC7303])
Security considerations: As for application/xml per [RFC7303]
Interoperability considerations: As for application/xml per [RFC7303]
Published specification: This document.
Applications which use this media type: Software using label generation rulesets, including registry applications and client validators.
Additional information: None.
Magic number: None.
File extension: .lgr or .xml
Macintosh file-type code: None.
Object Identifiers: None.
Intended Usage: Common
Personal and email address for further information: See the Authors of this document.
Change Controller: World Wide Web Consortium (W3C)
This specification uses a URN to describe the XML namespace, in accordance with [RFC3688].
URI: urn:ietf:params:xml:ns:lgr-1.0
Registrant Contact: See the Authors of this document.
XML: None.
This document establishes a vocabulary of "Label Generation Ruleset Dispositions" which should be reflected as a new IANA registry. This registry should be divided into two sub-registries:
All private dispositions MUST be registered using the prefix-colon notation to distinguish them from standard dispositions.
The IANA registry should provide data on the name of the disposition, the intended purposes, and the registrant or defining specification for the disposition.
If a system that is querying an identifier list (such as a domain zone) that uses the rules in this memo, and those rules are not implemented correctly, and that system is relying on the rules being applied, the system might fail if the rules are not applied in a predictable fashion. This could cause security problems for the querying system.
A naive implementation attempting to generate all variant labels for a given label could lead to the possibility of exhausting the resources on the machine running the LGR processor, potentially causing a DoS on the server. For many operations, brute force generation can be avoided by optimization, and if needed, the number of permuted labels can be estimated more cheaply ahead of time.
The implementation of Whole Label Evaluation rules, using certain backtracking algorithms, can take exponential time for pathological rules or labels and exhaust stack resources. This can be mitigated by proper implementation and enforcing the restrictions on permissible label length.
| [RFC3339] | Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002. |
| [RFC3688] | Mealling, M., "The IETF XML Registry", BCP 81, RFC 3688, DOI 10.17487/RFC3688, January 2004. |
| [RFC5646] | Phillips, A. and M. Davis, "Tags for Identifying Languages", BCP 47, RFC 5646, DOI 10.17487/RFC5646, September 2009. |
| [RFC6838] | Freed, N., Klensin, J. and T. Hansen, "Media Type Specifications and Registration Procedures", BCP 13, RFC 6838, DOI 10.17487/RFC6838, January 2013. |
| [RFC7303] | Thompson, H. and C. Lilley, "XML Media Types", RFC 7303, DOI 10.17487/RFC7303, July 2014. |
| [UAX42] | Unicode Consortium, "Unicode Character Database in XML" |
| [Unicode-Stability] | Unicode Consortium, "Unicode Encoding Stability Policy, Property Value Stability" |
| [Unicode-Versions] | Unicode Consortium, "Unicode Version Numbering" |
| [XML] | World Wide Web Consortium, "Extensible Markup Language (XML) 1.0" |
The following presents a minimal LGR table defining the lower case LDH (letter-digit-hyphen) repertoire and containing no rules or metadata elements. Many simple LGR tables will look quite similar, except that they would contain some metadata.
<?xml version="1.0" encoding="utf-8"?>
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
<data>
<char cp="002D" comment="HYPHEN (-)" />
<range first-cp="0030" last-cp="0039"
comment="DIGIT ZERO - DIGIT NINE" />
<range first-cp="0061" last-cp="007A"
comment="LATIN SMALL LETTER A - LATIN SMALL LETTER Z" />
</data>
</lgr>
The following sample LGR shows a more complete collection of the elements and attributes defined in this specification in a somewhat typical context.
<?xml version="1.0" encoding="utf-8"?>
<!-- This example uses a large subset of the features of this
specification. It does not include every set operator,
match operator element, or action trigger attribute, their
use being largely parallel to the ones demonstrated. -->
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
<!-- meta element with all optional elements -->
<meta>
<version comment="initial version">1</version>
<date>2010-01-01</date>
<language>sv</language>
<scope type="domain">example.com</scope>
<validity-start>2010-01-01</validity-start>
<validity-end>2013-12-31</validity-end>
<description type="text/html">
<![CDATA[
This language table was developed with the
<a href="http://swedish.example/">Swedish
examples institute</a>.
]]>
</description>
<unicode-version>6.3.0</unicode-version>
<references>
<reference id="0" comment="the most recent" >The
Unicode Standard 6.2</reference>
<reference id="1" >RFC 5892</reference>
<reference id="2" >Big-5: Computer Chinese Glyph
and Character Code Mapping Table, Technical Report
C-26, 1984</reference>
</references>
</meta>
<!-- the data section describing the repertoire -->
<data>
<!-- single code point "char" element -->
<char cp="002D" ref="1" comment="HYPHEN" />
<!-- range elements for contiguous code points, with tags -->
<range first-cp="0030" last-cp="0039" ref="1" tag="digit" />
<range first-cp="0061" last-cp="007A" ref ="1" tag="letter" />
<!-- code point sequence -->
<char cp="006C 00B7 006C" comment="catalan middle dot" />
<!-- alternatively use a when rule -->
<char cp="00B7" when="catalan-middle-dot" />
<!-- code point with context rule -->
<char cp="200D" when="joiner" ref="2" />
<!-- code points with variants -->
<char cp="4E16" tag="preferred" ref="0">
<var cp="4E17" type="blocked" ref="2" />
<var cp="534B" type="allocatable" ref="2" />
</char>
<char cp="4E17" ref="0">
<var cp="4E16" type="allocatable" ref="2" />
<var cp="534B" type="allocatable" ref="2" />
</char>
<char cp="534B" ref="0">
<var cp="4E16" type="allocatable" ref="2" />
<var cp="4E17" type="blocked" ref="2" />
</char>
</data>
<!-- Context and whole label rules -->
<rules>
<!-- Require the given code point to be between two 006C -->
<rule name="catalan-middle-dot" ref="0">
<look-behind>
<char cp="006C" />
</look-behind>
<anchor />
<look-ahead>
<char cp="006C" />
</look-ahead>
</rule>
<!-- example of a context rule based on property -->
<class name="virama" property="ccc:9" />
<rule name="joiner" ref="1" >
<look-behind>
<class by-ref="virama" />
</look-behind>
<anchor />
</rule>
<!-- example of using set operators -->
<!-- Subtract vowels from letters to get
consonant, demonstrating the different
set notations and the difference operator -->
<difference name="consonants">
<class comment="all letters">0061-007A</class>
<class comment="all vowels">
0061 0065 0069 006F 0075
</class>
</difference>
<!-- by using the start and end, rule matches whole label -->
<rule name="three-or-more-consonants">
<start />
<!-- reference the class defined by the difference
and require three or more matches -->
<class by-ref="consonants" count="3+" />
<end />
</rule>
<!-- rule for negative matching -->
<rule name="non-preferred"
comment="matches any non-preferred code point">
<complement comment="non-preferred" >
<class from-tag="preferred" />
</complement>
</rule>
<!-- actions triggered by matching rules and/or
variant types -->
<action disp="consonants"
match="three-or-more-consonants" />
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" all-variants="allocatable"
not-match="non-preferred" />
</rules>
</lgr>
As a background, the [RFC3743] rules work as follows:
To illustrate by example, here is one of the more complicated set of variants:
U+4E7E
U+4E81
U+5E72
U+5E79
U+69A6
U+6F27
The following shows the relevant section of the Chinese language table published by the .ASIA registry [ASIA-TABLE]. Its entries read:
<codepoint>;<simpl-variant(s)>;<trad-variant(s)>;<other-variant(s)>
These are the lines corresponding to the set of variants listed above
U+4E7E;U+4E7E,U+5E72;U+4E7E;U+4E81,U+5E72,U+6F27,U+5E79,U+69A6 U+4E81;U+5E72;U+4E7E;U+5E72,U+6F27,U+5E79,U+69A6 U+5E72;U+5E72;U+5E72,U+4E7E,U+5E79;U+4E7E,U+4E81,U+69A6,U+6F27 U+5E79;U+5E72;U+5E79;U+69A6,U+4E7E,U+4E81,U+6F27 U+69A6;U+5E72;U+69A6;U+5E79,U+4E7E,U+4E81,U+6F27 U+6F27;U+4E7E;U+6F27;U+4E81,U+5E72,U+5E79,U+69A6
The corresponding data section XML format would look like this:
<data>
<char cp="4E7E">
<var cp="4E7E" type="both" comment="identity" />
<var cp="4E81" type="blocked" />
<var cp="5E72" type="simp" />
<var cp="5E79" type="blocked" />
<var cp="69A6" type="blocked" />
<var cp="6F27" type="blocked" />
</char>
<char cp="4E81">
<var cp="4E7E" type="trad" />
<var cp="5E72" type="simp" />
<var cp="5E79" type="blocked" />
<var cp="69A6" type="blocked" />
<var cp="6F27" type="blocked" />
</char>
<char cp="5E72">
<var cp="4E7E" type="trad"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="both" comment="identity"/>
<var cp="5E79" type="trad"/>
<var cp="69A6" type="blocked"/>
<var cp="6F27" type="blocked"/>
</char>
<char cp="5E79">
<var cp="4E7E" type="blocked"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="simp"/>
<var cp="5E79" type="trad" comment="identity"/>
<var cp="69A6" type="blocked"/>
<var cp="6F27" type="blocked"/>
</char>
<char cp="69A6">
<var cp="4E7E" type="blocked"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="simp"/>
<var cp="5E79" type="blocked"/>
<var cp="69A6" type="trad" comment="identity"/>
<var cp="6F27" type="blocked"/>
</char>
<char cp="6F27">
<var cp="4E7E" type="simp"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="blocked"/>
<var cp="5E79" type="blocked"/>
<var cp="69A6" type="blocked"/>
<var cp="6F27" type="trad" comment="identity"/>
</char>
</data>
Here the simplified variants have been given a type of "simp", the traditional variants one of "trad" and all other ones are given "blocked".
Because some variant mappings show in more than one column, while the XML format allows only a single type value, they have been given the type of "both".
Note that some variant mappings map to themselves (identity), that is the mapping is reflexive (see Section 4.3.4). In creating the permutation of all variant labels, these mappings have no effect, other than adding a value to the variant type list for the variant label containing them.
In the example so far, all of the entries with type="both" are also mappings where source and target are identical. That is, they are reflexive mappings as defined in Section 4.3.4.
Given a label "U+4E7E U+4E81", the following labels would be ruled allocatable under [RFC3743] based on how that standard is commonly implemented in domain registries:
Original label: U+4E7E U+4E81
Simplified label 1: U+4E7E U+5E72
Simplified label 2: U+5E72 U+5E72
Traditional label: U+4E7E U+4E7E
However, if allocatable labels were generated simply by a straight permutation of all variants with type other than type="blocked" and without regard to the simplified and traditional variants, we would end up with an extra allocatable label of "U+5E72 U+4E7E". This label is comprised of a both Simplified Chinese character and a Traditional Chinese code point and therefore shouldn't be allocatable.
To more fully resolve the dispositions requires several actions to be defined as described in Section 6.2.2 which will override the default actions from Section 6.6. After blocking all labels that contain a variant with type "blocked", these actions will set to allocatable labels based on the following variant types: "simp", "trad" and "both". Note that these variant types do not directly relate to dispositions for the variant label, but that the actions will resolve them to the standard dispositions on labels, to with "blocked" and "allocatable".
To resolve label dispositions requires five actions to be defined (in the rules section of this document) these actions apply in order and the first one triggered, defines the disposition for the label. The actions are:
The rules declarations would be represented as:
<rules>
<!--Action elements - order defines precedence-->
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" only-variants="simp both" />
<action disp="allocatable" only-variants="trad both" />
<action disp="blocked" any-variant="simp trad" />
<action disp="allocatable" comment="catch-all" />
</rules>
Up to now, variants with type "both" have occurred only associated with reflexive variant mappings. The "action" elements defined above rely on the assumption that this is always the case. However, consider the following set of variants:
U+62E0;U+636E;U+636E;U+64DA
U+636E;U+636E;U+64DA;U+62E0
U+64DA;U+636E;U+64DA;U+62E0
The corresponding XML would be:
<char cp="62E0">
<var cp="636E" type="both" comment="both, but not reflexive" />
<var cp="64DA" type="blocked" />
</char>
<char cp="636E">
<var cp="636E" type="simp" comment="reflexive, but not both" />
<var cp="64DA" type="trad" />
<var cp="62E0" type="blocked" />
</char>
<char cp="64DA">
<var cp="636E" type="simp" />
<var cp="64DA" type="trad" comment="reflexive" />
<var cp="62E0" type="blocked" />
</char>
To make such variant sets work requires a way to selectively trigger an action based on whether a variant type is associated with an identity or reflexive mapping, or is associated with an ordinary variant mapping. This can be done by adding a prefix "r-" to the "type" attribute on reflexive variant mappings. For example the "trad" for code point U+64DA in the preceding figure would become "r-trad".
With the dispositions prepared in this way, only a slight modification to the actions is needed to yield the correct set of allocatable labels:
<action disp="blocked" any-variant="blocked" /> <action disp="allocatable" only-variants="simp r-simp both r-both" /> <action disp="allocatable" only-variants="trad r-trad both r-both" /> <action disp="blocked" all-variants="simp trad both" /> <action disp="allocatable" />
The first three actions get triggered by the same labels as before.
The fourth action blocks any label that combines an original code point with any mix of ordinary variant mappings; however no labels that are a combination of only original code points (code points having either no variant mappings or a reflexive mapping) would be affected. These are the original labels and they are allocated in the last action.
Using this scheme of assigning types to ordinary and reflexive variants, all RFC 3743-style tables can be converted to XML. By defining a set of actions as outlined above, the LGR will yield the correct set of allocatable variants: all variants consisting completely of variant code points preferred for simplified or traditional, respectively, will be allocated, as will be the original label. All other variant labels will be blocked.
In LGRs for Indic scripts it may be desirable to restrict valid labels to sequences of valid Indic syllables, or aksharas. This appendix gives a sample set of rules designed to enforce this restriction.
An example of a BNF from for an akshara which has been published in “Devanagari Script Behavior for Hindi" [TDIL-HINDI]. The rules for other languages and scripts used in India are expected to be generally similar.
For Hindi, the BNF has the form:
V[m]|{C[N]H}C[N](H|[v][m])
Where:
By using the Unicode property "InSC" or "Indic_Syllabic_Category" which corresponds rather directly to the classification of characters in the BNF above, we can translate the BNF into a set of WLE rules matching the definition of an akshara.
<rules>
<!--Character Class Definitions go here-->
<class name="halant" property="InSC:Virama" />
<union name="vowel-modifier">
<class property="InSC:Visarga" />
<class property="InSC:Bindu" comment="includes anusvara" />
</union>
<!--Whole label evaluation and Context rules go here-->
<rule name="consonant-with-optional-nukta">
<class by-ref="InSC:Consonant" />
<class by-ref="InSC:Nukta" count="0:1"/>
</rule>
<rule name="independent-vowel-with-optional-modifier">
<class by-ref="InSC:Vowel_Independent" />
<class by-ref="vowel-modifier" count="0:1" />
</rule>
<rule name="optional-dependent-vowel-with-opt-modifier" >
<class by-ref="InSC:Vowel_Dependent" count="0:1" />
<class by-ref="vowel-modifier" count="0:1" />
</rule>
<rule name="consonant-cluster">
<rule count="0+">
<rule by-ref="consonant-with-optional-nukta" />
<class by-ref="halant" />
</rule>
<rule by-ref="consonant-with-optional-nukta" />
<choice>
<class by-ref="halant" />
<rule by-ref="optional-dependent-vowel-with-opt-modifier" />
</choice>
</rule>
<rule name="akshara">
<choice>
<rule by-ref="independent-vowel-with-optional-modifier" />
<rule by-ref="consonant-cluster" />
</choice>
</rule>
<rule name="WLE-akshara-or-other" comment="series of one or
more aksharas, possibly alternating with other types of
code points such as digits">
<start />
<choice count="1+">
<class property="InSC:other" />
<rule by-ref="akshara" />
</choice>
<end />
</rule>
<!--Action elements go here - order defines precedence-->
<action disp="invalid" not-match="WLE-akshara-or-other" />
</rules>
With the rules and classes as defined above, the final action assigns a disposition of "invalid" to all labels that are not composed of a sequence of well-formed aksharas, optionally interspersed with other characters, perhaps digits, for example.
The relevant Unicode property could be replicated by tagging repertoire values directly in the LGR which would remove the dependency on any specific version of the Unicode Standard.
Generally, dependent vowels may only follow consonant expressions, however, for some scripts, like Bengali, the Unicode standard supports sequences of dependent vowels or their application on independent vowels. This makes the definition of akshara less restrictive.
It is possible to reduce the complexity of these rules by defining alternate rules which simply define the permissible pair-wise context of adjacent code points by character class—such as the rule that a Halant can only follow a (nuktated) consonant. (See the example in [WLE-RULES]).
%schema%
This format builds upon the work on documenting IDN tables by many different registry operators. Notably, a comprehensive language table for Chinese, Japanese and Korean was developed by the "Joint Engineering Team" [RFC3743] that is the basis of many registry policies; and a set of guidelines for Arabic script registrations [RFC5564] was published by the Arabic-language community.
Contributions that have shaped this document have been provided by Francisco Arias, Mark Davis, Martin Duerst, Paul Hoffman, Alexander Mayrhofer, Nicholas Ostler, Thomas Roessler, Audric Schiltknecht, Steve Sheng, Michel Suignard, Andrew Sullivan, Wil Tan and John Yunker.
This appendix to be removed prior to final publication.
These are being tracked at http://trac.tools.ietf.org/wg/lager/trac/report/1