Internet DRAFT - draft-sullivan-lucid-prob-stmt
draft-sullivan-lucid-prob-stmt
Internet Engineering Task Force A. Sullivan
Internet-Draft Dyn
Intended status: Informational A. Freytag
Expires: September 10, 2015 ASMUS Inc.
March 9, 2015
A Problem Statement to Motivate Work on Locale-free Unicode Identifiers
draft-sullivan-lucid-prob-stmt-00
Abstract
Internationalization techniques that the IETF has adopted depended on
some assumptions about the way characters get added to Unicode. Some
of those assumptions turn out not to have been true. Discussion is
necessary to determine how the IETF should respond to the new
understanding of how Unicode works.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 10, 2015.
Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
Sullivan & Freytag Expires September 10, 2015 [Page 1]
Internet-Draft LUCID Problem Statement March 2015
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1. The Inclusion Mechanism . . . . . . . . . . . . . . . . . 3
2.2. The Difference Between Theory and Practice . . . . . . . 4
2.2.1. Confusability . . . . . . . . . . . . . . . . . . . . 4
2.2.1.1. Not everything can be solved . . . . . . . . . . 5
2.2.2. The Problem Now Before Us . . . . . . . . . . . . . . 6
3. Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Types of Identifiers . . . . . . . . . . . . . . . . . . 8
4. Possible Nature of Problem . . . . . . . . . . . . . . . . . 9
4.1. Just a Species of Confusables . . . . . . . . . . . . . . 9
4.2. Just a Species of Homoglyphs . . . . . . . . . . . . . . 9
4.3. Separate Problem . . . . . . . . . . . . . . . . . . . . 10
4.4. Unimportant Problem . . . . . . . . . . . . . . . . . . . 10
5. Possible Ways Forward . . . . . . . . . . . . . . . . . . . . 10
5.1. Find the Cases, Disallow New Ones, and Deal
With Old Ones . . . . . . . . . . . . . . . . . . . . . . 10
5.2. Disallow Certain Combining Sequences
Absolutely . . . . . . . . . . . . . . . . . . . . . . . 11
5.3. Do Nothing, Possibly Warn . . . . . . . . . . . . . . . . 11
5.4. Identify Enough Commonality for a New
Property . . . . . . . . . . . . . . . . . . . . . . . . 12
5.5. Create an IETF-only Normalization Form . . . . . . . . . 12
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12
7. Informative References . . . . . . . . . . . . . . . . . . . 12
Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 15
1. Introduction
Among its features, IDNA2008 [RFC5890] [RFC5891] [RFC5892] [RFC5893]
[RFC5894] [RFC5895] provides a way of using Unicode [Unicode]
characters without regard to the version of Unicode available. The
same approach is generalized for protocols other than DNS by the
PRECIS framework [I-D.ietf-precis-framework].
The mechanism used is called "inclusion", and is outlined in
Section 2.1 below. We call the general strategy "inclusion-based
identifier internationalization" or "i3" for short. I3 depends on
certain assumptions made in the IETF at the time it was being
developed. Some of those assumptions were about the relationships
between various characters and the likelihood that similar such
relationships would get added to future versions of Unicode. Those
Sullivan & Freytag Expires September 10, 2015 [Page 2]
Internet-Draft LUCID Problem Statement March 2015
assumptions turn out not to have been true in every case. This
raises a question, therefore, about whether the current approach
meets the needs of the IETF for internationalizing identifiers.
This memo attempts to give enough background about the situation so
that IETF participants can participate in a discussion about what (if
anything) to do about the state of affairs; the discussion is
expected to happen as part of the LUCID BoF at IETF 92. The reader
is assumed to be familiar with the terminology in [RFC6365]. This
memo owes a great deal to the exposition in
[I-D.klensin-idna-5892upd-unicode70].
2. Background
The intent of Unicode is to encode all known writing systems into a
single coded character set. One consequence of that goal is that
Unicode encodes an enormous number of characters. Another is that
the work of Unicode does not end until every writing system is
encoded; even after that, it needs to continue to track any changes
in those writing systems. Unicode encodes abstract characters, not
glyphs. Because of the way Unicode was built up over time, there are
sometimes multiple ways to encode the same abstract character. If
Unicode encodes an abstract character in more than one way, then for
most purposes the different encodings should all be treated as though
they're the same character. This is called "canonical equivalence".
A lack of a defined canonical equivalence is tantamount to an
assertion by Unicode that the two encodings do not represent the same
abstract character, even if both happen to result in the same
appearance.
Every encoded character in Unicode (that is, every code point) is
associated with a set of properties. The properties define what
script a code point is in, whether it is a letter or a number or
punctuation and so forth, what direction it is written in, to what
other code point or code point sequence it is canonically equivalent,
and many other properties. These properties are important to the
inclusion mechanism.
2.1. The Inclusion Mechanism
Because of both the enormous number of characters in Unicode and the
many purposes it must serve, Unicode contains characters that are not
well-suited for use as part of identifiers for network protocols.
The inclusion mechanism starts by assuming an empty set of
characters. It then evaluates Unicode characters not individually,
but instead by classifying them according to their properties. This
Sullivan & Freytag Expires September 10, 2015 [Page 3]
Internet-Draft LUCID Problem Statement March 2015
classification provides the "derived properties" that IDNA2008 and
PRECIS rely upon.
In practice, the inclusion mechanism includes code points that are
letters or digits. There are some ways to include or exclude
characters that otherwise would be excluded or included
(respectively); but it is impractical to evaluate each character, so
most characters are included or excluded based on the properties they
have.
I3 depends on the assumption that strings that will be used in
identifiers will not have any ambiguous matching to other strings.
In practice, this means that input strings to the protocol are
expected to be in Normalization Form C. This way, any alternative
sequences of code points for the same characters will be normalized
to a single form. Assuming then that those characters are all
included by the inclusion mechanism, the string is eligible to be an
identifier under the protocol.
2.2. The Difference Between Theory and Practice
In principle, under i3 identifiers should be unambiguous. It has
always been recognized, however, that for humans some ambiguity was
inevitable, because of the vagaries of writing systems and of human
perception.
Normalization Form NFC removes the ambiguities based on dual or
multiple encoding for the same abstract character. However,
characters are not the same as their glyphs. This means that it is
possible for certain abstract characters to share a glyph. We call
such abstract characters "homoglyphs". While this looks at first
like something that should be handled (or should have been handled)
by normalization (NFC or something else), there are important
differences; the situation is in some sense an extreme case of a
spectrum of ambiguity discussed in the following section.
2.2.1. Confusability
While Unicode deals in abstract characters and i3 works on Unicode
code points, users interact with the characters as actually rendered:
glyphs. There are characters that, depending on font, sometimes look
quite similar to one another (such as "l" and "1"); any character
that is like this is often called "visually similar". More difficult
are characters that, in any normal rendering, always look the same as
one another. The shared history of Cyrillic, Greek, and Latin
scripts, for example, means that there are characters in each script
that function similarly and that are usually indistinguishable from
one another, though they are not the same abstract character. These
Sullivan & Freytag Expires September 10, 2015 [Page 4]
Internet-Draft LUCID Problem Statement March 2015
are examples of "homoglyphs." Any character that can be confused for
another one can be called confusable, and confusability can be
thought of as a spectrum with "visually similar" at one end, and
"homoglyphs" at the other. (We use the term "homoglyph" strictly:
code points that normally use the same glyph when rendered.)
Most of the time, there is some characteristic that can help to
mitigate confusion. Mitigation may be as simple as using a font
designed to distinguish among different characters. For homoglyphs,
a large number of cases (but not all of them) turn out to be in
different scripts. As a result, there is an operational convention
that identifiers should always be in a single script. (This strategy
can be less than successful in cases where each identifier is in a
single script, but the repertoire used in operation allows multiple
scripts, because of whole string confusables -- strings made up
entirely of homoglyphs of another string in a different script.)
There is another convention that operators should only ever use the
smallest repertoire of code points possible for their environment.
So, for example, if there is a code point that is sometimes used but
is perhaps a little obscure, it is better to leave it out and gain
some experience with other cases first. In particular, code points
used in a language with which the administrator is not familiar
should probably be excluded. In the case of IDNA, some client
programs restrict display of U-labels to top-level domains known to
have policies about single-script labels. None of these policies or
convention will do anything to help strict homoglyphs of each other
in the same script (see Appendix A for some example cases.)
2.2.1.1. Not everything can be solved
Before continuing, it is worth noting that there are some cases that,
regardless of mitigation, are fundamentally impossible to solve.
There are certainly cases of two strings in which all the code points
in one script in the first string, and all the code points in another
script in the second string, are respectively confusable with one
another. In that case, the strings cannot be distinguished by a
reader, and the whole string is confusable. Further, human
perception is easily tricked, so that entirely unrelated character
sequences can become confusable, for example "rn" being confused with
"m".
Given the facts of history and the contingencies of writing systems,
one cannot defend against all of these cases; and it seems all but
certain that many of these cases cannot successfully be addressed on
the protocol level alone. In general, the i3 strategy can only
define rules for one identifier at a time, and has no way to offer
guidance about how different identifiers under the same scheme ought
Sullivan & Freytag Expires September 10, 2015 [Page 5]
Internet-Draft LUCID Problem Statement March 2015
to interact. Humans are likely to respond according to the entire
identifier string, so there seems to be a deep tension between the
narrow focus of i3, and the actual experience of users.
In addition, several factors limit the ability to ensure that any
solution adopted is final and complete: the sheer complexity of
writing systems, the fact that many of them are not equally well
understood as Latin or Han, and that many less developed writing
systems are potentially susceptible to paradigm changes as digital
support for them becomes more widespread. Detailed knowledge about,
and implementation experience for, these writing systems only emerges
over time; disruptive changes are neither predictable ahead of time
nor preventable. In essence, any solution to eliminate ambiguity can
be expected to get some detail wrong.
Nobody should imagine that the present discussion takes as its goal
the complete elimination of all possible confusion. The failure to
achieve such a goal does not mean, however, that we should do
nothing, any more than the low chances of ever arresting all grifters
means that we should not enact laws against fraud. Our discussion,
then, must focus on those problems that are able to be addressed in
the constraint of the protocols; and, in particular, the subset that
are suitable for that
2.2.2. The Problem Now Before Us
During the expert review necessary for supporting Unicode 7.0.0 for
use with IDNA, a new code point U+08A1, ARABIC LETTER BEH WITH HAMZA
ABOVE came in for some scrutiny. Using versions of Unicode up to and
including 7.0.0, it is possible to combine ARABIC LETTER BEH (U+0628)
and ARABIC HAMZA ABOVE (U+0654) to produce a glyph that is
indistinguishable from the one produced by U+08A1. But U+08A1 and
\u'0628'\u'0654' are not canonically equivalent. (For more
discussion of this issue, see [I-D.klensin-idna-5892upd-unicode70].)
Further investigation reveals that there are several similar cases.
ARABIC HAMZA ABOVE (U+0654) turns out to be implicated in some cases,
but not all of them. There are cases in Latin (see Appendix A for
examples). There are certainly cases in other scripts (some examples
are provided in Appendix A). The majority of cases all have a
handful of things in common:
o There are at least two forms by which the same glyph is produced.
o One of the forms uses a combining sequence and another form is a
precomposed character, or else one of the forms is a digraph.
[[CREF1: Is this true? Are there any cases that don't match it?
--ajs]]
Sullivan & Freytag Expires September 10, 2015 [Page 6]
Internet-Draft LUCID Problem Statement March 2015
o The results when rendered as glyphs cannot be distinguished from
one another.
o The two forms are not canonically equivalent.
o All of the relevant code points have the same script property, or
else inherit the script property of the previous character so that
it is not possible to select on the basis of the script.
o Competent users of the writing system in a language do not treat
one of the combining sequence or the precomposed character as
reasonable. To writers for whom the combining sequence is
"wrong", it is not a case of a base character modified by an
additional mark, but instead a separate letter. Conversely, to
writers for whom the precomposed character is "wrong", it is
definitely a matter of adding something to a character that
otherwise stands on its own. (Not every possible combination
would normally be used by anyone, of course, and sometimes -- not
infrequently -- one of the alternatives is not used by any
orthography.)
Cases that match these conditions might be considered to involve
"non-normalizable diacritics", because most of the combining marks in
question are non-spacing marks that are or act like diacritics.
3. Identifiers
Part of the reason i3 works from the assumption that not all Unicode
code points are appropriate for identifiers is that identifiers do
not work like words of phrases in a language. First, identifiers
often appear in contexts where there is no way to tell the language
of the identifiers. Indeed, many identifiers are not really "in a
language" at all. Second, and partly because of that lack of
linguistic root, identifiers are often either not words or use
unusual orthography precisely to differentiate themselves.
In ordinary language use, the ambiguity identified in Section 2.2 may
well create no difficulty. Running text has two properties that make
this so. First, because there is a linguistic context (the rest of
the text), it is possible to detect code points that are used in an
unusual way and flag them or, even, create automatic rules to "fix"
such issues. Second, linguistic context comes with spelling rules
that automatically determine whether something is written the right
way. Because of these facts, it is often possible even without a
locale identifier to work out what the locale of the text ought to
be. So, even in cases where passages of text need to be compared, it
is possible to mitigate the issue.
Sullivan & Freytag Expires September 10, 2015 [Page 7]
Internet-Draft LUCID Problem Statement March 2015
The same locale-detection approach does not work for identifiers.
Worse, identifiers, by their very nature, are things that must
provide reliable exact matches. The whole point of an identifier is
that it provides a reliable way of uniquely naming the thing to be
identified. Partial matches and heuristics are inadequate for those
purposes. Identifiers are often used as part of the security
practices for a protocol, and therefore ambiguity in matching
presents a risk for the security of any protocol relying on the
identifier.
3.1. Types of Identifiers
It is worth observing that not all identifiers are of the same type.
There are four relevant dimensions in which identifiers can differ in
type:
1. Scope
(a) Internet-wide
(b) Unique within a context (often a site)
(c) Link-local only
2. Management
(a) Centrally managed
(b) Contextually managed (e.g. registering a nickname with a
server for a session)
(c) Unmanaged
3. Durability
(a) Permanent
(b) Durable but with possible expiration
(c) Temporary
(d) Ephemeral
4. Authority
(a) Single authority
(b) Multiple authorities (possibly within a hierarchy)
Sullivan & Freytag Expires September 10, 2015 [Page 8]
Internet-Draft LUCID Problem Statement March 2015
(c) No authority
These different dimensions present ways in which mitigation of the
identified issue might be possible. For instance, a protocol that
uses only link-local identifiers that are unmanaged, temporary, and
configured automatically does not really present a problem, because
for practical purposes its linguistic context is constrained to the
social realities of the LAN in question. A durable Internet-wide
identifier centrally managed by multiple authorities will present a
greater issue unless locale information comes along with the
identifier.
4. Possible Nature of Problem
We may regard this problem as one of several different kinds, and
depending on how we view it we will have different approaches to
addressing it.
4.1. Just a Species of Confusables
Under this interpretation, the current issue is no different to any
other confusable case, except in detail. Since there is no way to
solve the general problem of confusables, there is no way to solve
this problem either. Moreover, to the degree that confusables are
solved outside protocols, by administration and policy, the current
issue might be addressed by the same strategy.
This interpretation seems unsatisfying, because there exist some
partial mitigations, and if suitable further mitigations are possible
it would be wise to apply them.
4.2. Just a Species of Homoglyphs
Under this interpretation, the current issue is no different than any
other homoglyph case. After all, the basic problem is that there is
no way for a user to tell which codepoint is represented by what the
user sees in either case.
There is some merit to this view, but it has the problem that many of
the homoglyph issues (admittedly not all of them) can be mitigated
through registration rules, and those rules can be established
without examining the particular code points in question (that is,
they can operate just on the properties of code points, such as
script membership). The current issue does not allow such mitigation
given the properties that are currently available. At the same time,
it may be that it is impossible to deal with this adequately, and
some judgement will be needed for what is adequate. This is an area
where more discussion is clearly needed.
Sullivan & Freytag Expires September 10, 2015 [Page 9]
Internet-Draft LUCID Problem Statement March 2015
4.3. Separate Problem
Under this interpretation, there is a definable problem, and its
boundaries can be specified.
That we can list some necessary conditions for the problem suggests
that it is a separable problem. The list of factors in Section 2.2.2
seems to indicate that it is possible to describe the bounds of a
problem that can be addressed separately.
What is not clear is whether it is separable enough to make it worth
treating separately.
4.4. Unimportant Problem
Under this interpretation, while it is possible to describe the
problem, it is not a problem worth addressing since nobody would ever
create such identifiers on purpose.
The problem with this approach, for identifiers, is that it
represents an opportunity for phishing and other similar attacks.
While mitigation will not stop all such attacks, we should try to
understand opportunities for those attacks and close when we have
identified them and it is practical to do so.
Whether phishing or other attacks using confusable code points "pay
off" depends to some extent on the popularity or frequency of the
code points in question. While it may be worth to address the
generalized issue, individual edge cases may have no practical
consequences. The inability to address them then, should not hold up
progress on a solution for the more common, general case.
5. Possible Ways Forward
There are a few ways that this issue could be mitigated. Note that
this section is closely related to Section 3 in
[I-D.klensin-idna-5892upd-unicode70].
5.1. Find the Cases, Disallow New Ones, and Deal With Old Ones
In this case, it is necessary to enumerate all the cases, add
exceptions to DISALLOW any new cases from happening, and make a
determination about what to do for every past case. There are two
reasons to doubt whether this approach will work.
1. The IETF did not catch these issues during previous
internationalization efforts, and it seems unlikely that in the
Sullivan & Freytag Expires September 10, 2015 [Page 10]
Internet-Draft LUCID Problem Statement March 2015
meantime it has acquired enough expertise in writing systems to
do a proper job of it this time.
2. This approach blunts the effectiveness of being Unicode version-
agnostic, since it would effectively block any future additions
to Unicode that had any interaction with the present version.
So, this approach does not seem too promising.
5.2. Disallow Certain Combining Sequences Absolutely
In this case, instead of treating all the code points in Unicode, the
IETF would need only to look at all combining characters. While the
IETF obviously does not have the requisite expertise in writing
systems to do this unilaterally, the Unicode Consortium does. In
fact the Unicode Technical Committee has a clear understanding that
some combining sequences are never intended to be used for
orthographic purposes. Any glyph needed for an orthography or
writing system will, once identified, be added as a single code point
with "pre-composed" glyph.
In principle there is no obstacle, in these cases, to asking Unicode
to express this understanding in form of a character property, which
then means that IETF could DISALLOW the combining marks having such a
property.
5.3. Do Nothing, Possibly Warn
One possibility is to accept that there is nothing one can do in
general here, and that therefore the best one can do is warn people
to be careful.
The problem with this approach, of course, is that it all but
guarantees future problems with ambiguous identifiers. It would
provide a good reason to reject all internationalized identifiers as
representing a significant security risk, and would therefore mean
that internationalized identifiers would become "second class".
Unfortunately, however, the demand for internationalized identifiers
would not likely be reduced by this decision, so some people would
end up using identifiers with known security problems.
This approach may be the only possible in some of the borderline
cases where mitigation approaches are not successful.
Sullivan & Freytag Expires September 10, 2015 [Page 11]
Internet-Draft LUCID Problem Statement March 2015
5.4. Identify Enough Commonality for a New Property
There is reason to suppose that, if the IETF can come up with clear
and complete conditions under which code points causing an issue
could be classified, the Unicode Technical Committee would add such a
property to code points in future versions of the Unicode Standard.
Assuming the conditions were clear, future additions to the Standard
could also be assigned appropriate values of the property, meaning
that the IETF could revert to making decisions about code points
based on derived properties. Beyond the property mentioned in
Section 5.2 this property could cover certain combining marks in the
Arabic script.
If this is possible, it seems a desirable course of action.
5.5. Create an IETF-only Normalization Form
Under this approach, the IETF creates a special normalization form
that it maintains outside the Unicode Standard. For the sake of the
discussion, we'll call this "NFI".
This option does not seem workable. The IETF would have to evaluate
every new release of Unicode to discover the extent to which the new
release interacts with NFI. Because it would be independently
maintained, Unicode stability guarantees would not apply to NFI; the
results would be unpredictable. As a result, either the IETF would
have to ignore new additions to Unicode, or else it would need UTC to
take NFI into account. If UTC were able to do so, this option
reduces to the option in Section 5.4. The UTC might not be able to
do this, however, because the very principles that Unicode uses to
assign new characters in certain situations guarantees that new
characters will be added that cannot be so normalized and yet are
essential for still-to-be-encoded writing systems. Communities for
which these new characters would be added would also not accept any
existing code point sequence as equivalent. This also means that
Unicode cannot create a stability policy to take into account the
needs of such an NFI.
6. Acknowledgements
The discussion in this memo owes a great deal to the IAB
Internationalization program, and particularly to John Klensin.
7. Informative References
Sullivan & Freytag Expires September 10, 2015 [Page 12]
Internet-Draft LUCID Problem Statement March 2015
[I-D.ietf-precis-framework]
Saint-Andre, P. and M. Blanchet, "PRECIS Framework:
Preparation, Enforcement, and Comparison of
Internationalized Strings in Application Protocols",
draft-ietf-precis-framework-23 (work in progress),
February 2015.
[I-D.klensin-idna-5892upd-unicode70]
Klensin, J. and P. Faeltstroem, "IDNA Update for Unicode
7.0.0", draft-klensin-idna-5892upd-unicode70-03 (work in
progress), January 2015.
[RFC5890] Klensin, J., "Internationalized Domain Names for
Applications (IDNA): Definitions and Document Framework",
RFC 5890, August 2010.
[RFC5891] Klensin, J., "Internationalized Domain Names in
Applications (IDNA): Protocol", RFC 5891, August 2010.
[RFC5892] Faltstrom, P., "The Unicode Code Points and
Internationalized Domain Names for Applications (IDNA)",
RFC 5892, August 2010.
[RFC5893] Alvestrand, H. and C. Karp, "Right-to-Left Scripts for
Internationalized Domain Names for Applications (IDNA)",
RFC 5893, August 2010.
[RFC5894] Klensin, J., "Internationalized Domain Names for
Applications (IDNA): Background, Explanation, and
Rationale", RFC 5894, August 2010.
[RFC5895] Resnick, P. and P. Hoffman, "Mapping Characters for
Internationalized Domain Names in Applications (IDNA)
2008", RFC 5895, September 2010.
[RFC6365] Hoffman, P. and J. Klensin, "Terminology Used in
Internationalization in the IETF", BCP 166, RFC 6365,
September 2011.
[Unicode] "The Unicode Standard",
http://www.unicode.org/versions/Unicode7.0.0/, .
Appendix A. Examples
There are a number of cases that illustrate the combining sequence or
digraph issue:
Sullivan & Freytag Expires September 10, 2015 [Page 13]
Internet-Draft LUCID Problem Statement March 2015
U+08A1 vs \u'0628'\u'0654' This case is ARABIC LETTER BEH WITH HAMZA
ABOVE, which is the one that was detected during expert review
that caused the IETF to notice the issue. The issue existed
before this, but we did not know it. For detailed discussion of
this case and some of the following ones, see
[I-D.klensin-idna-5892upd-unicode70]
U+0681 vs \u'062D'\u'0654' This case is ARABIC LETTER HAH WITH HAMZA
ABOVE, which (like U+08A1) does not have a canonical equivalent.
In both cases, the places where hamza above are used are
specialized enough that the combining marks can be excluded in
some cases (for example, the root zone under IDNA).
U+0623 vs \u'0627'\u'0654' This case is ARABIC LETTER ALEF WITH
HAMZA ABOVE. Unlike the previous two cases, it does have a
canonical equivalence with the combining sequence. In the past,
the IETF misunderstood the reasons for the difference between this
pair and the previous two cases.
U+09E1 vs u\'098C'u\'09E2' This case is BENGALI LETTER VOCALIC LL.
This is an example in Bengali script of a case without a canonical
equivalence to the combining sequence. Per Unicode, the single
code point should be used to represent vowel letters in text, and
the sequence of code points should not be used. But it is not a
simple matter of disallowing the combining vowel mark in cases
like this; where the combination does not exist and the use of the
sequence is already established, Unicode is unlikely to encode the
combination.
U+019A vs \u'006C'\u'0335' This case is LATIN SMALL LETTER L WITH
BAR. In at least some fonts, there is a detectable difference
with the combining sequence, but only if one types them one after
another and compares them. There is no canonical equivalence
here. Unicode has a principle of encoding barred letters as
composites when needed for any writing system.
U+00F8 vs \u'006F'\u'0337' This is LATIN SMALL LETTER O WITH STROKE.
The effect are similar to the previous case. Unicode has a
principle of encoding stroked letters as composites when needed
for any writing system.
U+02A6 vs \u'0074'\u'0073' This is LATIN SMALL LETTER TS DIGRAPH,
which is not canonically equivalent to the letters t and s. The
intent appears to be that the digraph shows the two shapes as
kerned, but the difference may be slight out of context.
U+01C9 vs \u'006C'\u'006A' Unlike the TS digraph, the LJ digraph has
a relevant compatibility decomposition, so it fails the relevant
Sullivan & Freytag Expires September 10, 2015 [Page 14]
Internet-Draft LUCID Problem Statement March 2015
stability rules under i3 and is therefore DISALLOWED. This
illustrates the way that consistencies that might be natural to
some users of a script are not necessarily found in it, possibly
because of uses by another writing system.
U+06C8 vs u\'0648'u\'0670' ARABIC LETTER YU is an example where the
normally-rendered character looks just like a combining sequence,
but are named differently. In other words, this is an example
where the simple fact of the Unicode name would have concealed the
apparent relationship from the casual observer.
U+069 vs \u'0069'\u'0307' LATIN SMALL LETTER I followed by COMBINING
DOT ABOVE by definition, renders exactly the same as LATIN SMALL
LETTER I by itself and does so in practice for any good font. The
same would be true if "i" was replaced with any of the other
Soft_Dotted characters defined in Unicode. The character sequence
\u'0069'\u'0307' (followed by no other combining mark) is
reportedly rather common on the Internet. Because base character
and stand-alone code point are the same in this case, and the code
points affected have the Soft_Dotted property already, this could
be mitigated separately via a context rule affecting U+0307.
Other cases test the claim that the issue lies primarily with
combining sequences at all:
U+0B95 vs U+0BE7 The TAMIL LETTER KA and TAMIL DIGIT ONE are always
indistinguishable, but needed to be encoded separately because one
is a letter and the other is a digit.
Arabic-Indic Digits vs. Extended Arabic-Indic Digits Seven
digits of these two sequences have entirely identical shapes.
This case is an example of something dealt with in i3 that
nevertheless can lead to confusions that are not fully mitigated.
IDNA, for example, contains context rules restricting the digits
to one set or another; but such rules apply only to a single
label, not to an entire name. Moreover, it provides no way of
distinguishing between two labels that both conform to the context
rule, but where each contains one of the seven identical shapes.
U+53E3 vs U+56D7 These are two Han characters (roughly rectangular)
that are different when laid side by side; but they may be
impossible to distinguish out of context or in small print.
Authors' Addresses
Sullivan & Freytag Expires September 10, 2015 [Page 15]
Internet-Draft LUCID Problem Statement March 2015
Andrew Sullivan
Dyn
150 Dow St.
Manchester, NH 03101
US
Email: asullivan@dyn.com
Asmus Freytag
ASMUS Inc.
Email: asmus@unicode.org
Sullivan & Freytag Expires September 10, 2015 [Page 16]
