Overview

The std. uni module provides an implementation of fundamental Unicode algorithms and data structures. This doesn't include UTF encoding and decoding primitives, see std.utf.decode and std.utf.encode in std.utf for this functionality.

All primitives listed operate on Unicode characters and sets of characters. For functions which operate on ASCII characters and ignore Unicode characters, see std.ascii. For definitions of Unicode character, code point and other terms used throughout this module see the terminology section below.

The focus of this module is the core needs of developing Unicode-aware applications. To that effect it provides the following optimized primitives:

• Character classification by category and common properties: isAlpha , isWhite and others.
• Case-insensitive string comparison (sicmp , icmp ).
• Converting text to any of the four normalization forms via normalize .
• Decoding (decodeGrapheme ) and iteration (graphemeStride ) by user-perceived characters, that is by Grapheme clusters.
• Decomposing and composing of individual character(s) according to canonical or compatibility rules, see compose and decompose , including the specific version for Hangul syllables composeJamo and decomposeHangul .

It's recognized that an application may need further enhancements and extensions, such as less commonly known algorithms, or tailoring existing ones for region specific needs. To help users with building any extra functionality beyond the core primitives, the module provides:

• CodepointSet , a type for easy manipulation of sets of characters. Besides the typical set algebra it provides an unusual feature: a D source code generator for detection of code points in this set. This is a boon for meta-programming parser frameworks, and is used internally to power classification in small sets like isWhite .
• A way to construct optimal packed multi-stage tables also known as a special case of Trie . The functions codepointTrie , codepointSetTrie construct custom tries that map dchar to value. The end result is a fast and predictable Ο(1) lookup that powers functions like isAlpha and combiningClass , but for user-defined data sets.
• Generally useful building blocks for customized normalization: combiningClass for querying combining class and allowedIn for testing the Quick_Check property of a given normalization form.
• Access to a large selection of commonly used sets of code points. Supported sets include Script, Block and General Category. The exact contents of a set can be observed in the CLDR utility, on the property index page of the Unicode website. See unicode for easy and (optionally) compile-time checked set queries.

Synopsis

import std.uni;
void main()
{
    // initialize code point sets using script/block or property name

    // now 'set' contains code points from both scripts.

    auto set = unicode("Cyrillic") | unicode("Armenian");
    // same thing but simpler and checked at compile-time

    auto ascii = unicode.ASCII;
    auto currency = unicode.Currency_Symbol;

    // easy set ops

    auto a = set & ascii;
    assert(a.empty); // as it has no intersection with ascii

    a = set | ascii;
    auto b = currency - a; // subtract all ASCII, Cyrillic and Armenian


    // some properties of code point sets

    assert(b.length > 45); // 46 items in Unicode 6.1, even more in 6.2

    // testing presence of a code point in a set

    // is just fine, it is O(logN)

    assert(!b['$']);
    assert(!b['\u058F']); // Armenian dram sign

    assert(b['¥']);

    // building fast lookup tables, these guarantee O(1) complexity

    // 1-level Trie lookup table essentially a huge bit-set ~262Kb

    auto oneTrie = toTrie!1(b);
    // 2-level far more compact but typically slightly slower

    auto twoTrie = toTrie!2(b);
    // 3-level even smaller, and a bit slower yet

    auto threeTrie = toTrie!3(b);
    assert(oneTrie['£']);
    assert(twoTrie['£']);
    assert(threeTrie['£']);

    // build the trie with the most sensible trie level

    // and bind it as a functor

    auto cyrilicOrArmenian = toDelegate(set);
    auto balance = find!(cyrilicOrArmenian)("Hello ընկեր!");
    assert(balance == "ընկեր!");
    // compatible with bool delegate(dchar)

    bool delegate(dchar) bindIt = cyrilicOrArmenian;

    // Normalization

    string s = "Plain ascii (and not only), is always normalized!";
    assert(s is normalize(s));// is the same string


    string nonS = "A\u0308ffin"; // A ligature

    auto nS = normalize(nonS); // to NFC, the W3C endorsed standard

    assert(nS == "Äffin");
    assert(nS != nonS);
    string composed = "Äffin";

    assert(normalize!NFD(composed) == "A\u0308ffin");
    // to NFKD, compatibility decomposition useful for fuzzy matching/searching

    assert(normalize!NFKD("2¹⁰") == "210");
}

Terminology

The following is a list of important Unicode notions and definitions. Any conventions used specifically in this module alone are marked as such. The descriptions are based on the formal definition as found in ($WEB http://www.unicode.org/versions/Unicode6.2.0/ch03.pdf, chapter three of The Unicode Standard Core Specification.)

• When representing data, the nature of that data is generally symbolic as opposed to some other kind of data (for example, visual).



• An abstract character has no concrete form and should not be confused with a glyph.



• An abstract character does not necessarily correspond to what a user thinks of as a “character” and should not be confused with a Grapheme .



• The abstract characters encoded (see Encoded character) are known as Unicode abstract characters.



• Abstract characters not directly encoded by the Unicode Standard can often be represented by the use of combining character sequences.

• All characters with non-zero canonical combining class are combining characters, but the reverse is not the case: there are combining characters with a zero combining class.
• These characters are not normally used in isolation unless they are being described. They include such characters as accents, diacritics, Hebrew points, Arabic vowel signs, and Indic matras.

• The grapheme cluster represents a horizontally segmentable unit of text, consisting of some grapheme base (which may consist of a Korean syllable) together with any number of nonspacing marks applied to it.
• A grapheme cluster typically starts with a grapheme base and then extends across any subsequent sequence of nonspacing marks. A grapheme cluster is most directly relevant to text rendering and processes such as cursor placement and text selection in editing, but may also be relevant to comparison and searching.
• For many processes, a grapheme cluster behaves as if it was a single character with the same properties as its grapheme base. Effectively, nonspacing marks apply graphically to the base, but do not change its properties.

This module defines a number of primitives that work with graphemes: Grapheme , decodeGrapheme and graphemeStride . All of them are using extended grapheme boundaries as defined in the aforementioned standard annex.

Normalization

The concepts of canonical equivalent or compatibility equivalent characters in the Unicode Standard make it necessary to have a full, formal definition of equivalence for Unicode strings. String equivalence is determined by a process called normalization, whereby strings are converted into forms which are compared directly for identity. This is the primary goal of the normalization process, see the function normalize to convert into any of the four defined forms.

A very important attribute of the Unicode Normalization Forms is that they must remain stable between versions of the Unicode Standard. A Unicode string normalized to a particular Unicode Normalization Form in one version of the standard is guaranteed to remain in that Normalization Form for implementations of future versions of the standard.

The Unicode Standard specifies four normalization forms. Informally, two of these forms are defined by maximal decomposition of equivalent sequences, and two of these forms are defined by maximal composition of equivalent sequences.

• Normalization Form D (NFD): The canonical decomposition of a character sequence.
• Normalization Form KD (NFKD): The compatibility decomposition of a character sequence.
• Normalization Form C (NFC): The canonical composition of the canonical decomposition of a coded character sequence.
• Normalization Form KC (NFKC): The canonical composition of the compatibility decomposition of a character sequence

The choice of the normalization form depends on the particular use case. NFC is the best form for general text, since it's more compatible with strings converted from legacy encodings. NFKC is the preferred form for identifiers, especially where there are security concerns. NFD and NFKD are the most useful for internal processing.

Construction of lookup tables

The Unicode standard describes a set of algorithms that depend on having the ability to quickly look up various properties of a code point. Given the the codespace of about 1 million code points, it is not a trivial task to provide a space-efficient solution for the multitude of properties.

Common approaches such as hash-tables or binary search over sorted code point intervals (as in InversionList ) are insufficient. Hash-tables have enormous memory footprint and binary search over intervals is not fast enough for some heavy-duty algorithms.

The recommended solution (see Unicode Implementation Guidelines) is using multi-stage tables that are an implementation of the Trie data structure with integer keys and a fixed number of stages. For the remainder of the section this will be called a fixed trie. The following describes a particular implementation that is aimed for the speed of access at the expense of ideal size savings.

Taking a 2-level Trie as an example the principle of operation is as follows. Split the number of bits in a key (code point, 21 bits) into 2 components (e.g. 15 and 8). The first is the number of bits in the index of the trie and the other is number of bits in each page of the trie. The layout of the trie is then an array of size 2^^bits-of-index followed an array of memory chunks of size 2^^bits-of-page/bits-per-element.

The number of pages is variable (but not less then 1) unlike the number of entries in the index. The slots of the index all have to contain a number of a page that is present. The lookup is then just a couple of operations - slice the upper bits, lookup an index for these, take a page at this index and use the lower bits as an offset within this page.

Assuming that pages are laid out consequently in one array at pages, the pseudo-code is:

auto elemsPerPage = (2 ^^ bits_per_page) / Value.sizeOfInBits;
pages[index[n >> bits_per_page]][n & (elemsPerPage - 1)];

Where if elemsPerPage is a power of 2 the whole process is a handful of simple instructions and 2 array reads. Subsequent levels of the trie are introduced by recursing on this notion - the index array is treated as values. The number of bits in index is then again split into 2 parts, with pages over 'current-index' and the new 'upper-index'.

For completeness a level 1 trie is simply an array. The current implementation takes advantage of bit-packing values when the range is known to be limited in advance (such as bool). See also BitPacked for enforcing it manually. The major size advantage however comes from the fact that multiple identical pages on every level are merged by construction.

The process of constructing a trie is more involved and is hidden from the user in a form of the convenience functions codepointTrie , codepointSetTrie and the even more convenient toTrie . In general a set or built-in AA with dchar type can be turned into a trie. The trie object in this module is read-only (immutable); it's effectively frozen after construction.

Unicode properties

This is a full list of Unicode properties accessible through unicode with specific helpers per category nested within. Consult the CLDR utility when in doubt about the contents of a particular set.

General category sets listed below are only accessible with the unicode shorthand accessor.

Sets for other commonly useful properties that are accessible with unicode :

Bellow is the table with block names accepted by unicode.block . Note that the shorthand version unicode requires "In" to be prepended to the names of blocks so as to disambiguate scripts and blocks.

Bellow is the table with script names accepted by unicode.script and by the shorthand version unicode :

Bellow is the table of names accepted by unicode.hangulSyllableType .

License, Boost License 1.0.

Authors, Dmitry Olshansky

Standards, Unicode v6.2