The D Programming Language

Encode and decode UTF-8, UTF-16 and UTF-32 strings.

UTF character support is restricted to '\u0000' <= character <= '\U0010FFFF'.

See Also
Wikipedia
http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
http://anubis.dkuug.dk/JTC1/SC2/WG2/docs/n1335
License
Boost License 1.0.
Authors
Walter Bright and Jonathan M Davis
Source:
std/utf.d

class  UTFException: object.Exception;

Exception thrown on errors in std.utf functions.


pure nothrow @safe bool  isValidDchar(dchar c);

Returns whether c is a valid UTF-32 character.

'\uFFFE' and '\uFFFF' are considered valid by  isValidDchar, as they are permitted for internal use by an application, but they are not allowed for interchange by the Unicode standard.


uint  stride(S)(auto ref S str, size_t index) if (is(S : const(char[])) || isRandomAccessRange!S && is(Unqual!(ElementType!S) == char));
uint  stride(S)(auto ref S str) if (is(S : const(char[])) || isInputRange!S && is(Unqual!(ElementType!S) == char));

 stride returns the length of the UTF-8 sequence starting at index in str.

 stride works with both UTF-8 strings and ranges of char. If no index is passed, then an input range will work, but if an index is passed, then a random-access range is required.

index defaults to 0 if none is passed.

Returns
The number of bytes in the UTF-8 sequence.
Throws
May throw a UTFException if str[index] is not the start of a valid UTF-8 sequence.
Notes:
 stride will only analize the first str[index] element. It will not fully verify the validity of UTF-8 sequence, nor even verify the presence of the sequence: it will not actually guarantee that index +  stride(str, index) <= str.length.

uint  strideBack(S)(auto ref S str, size_t index) if (is(S : const(char[])) || isRandomAccessRange!S && is(Unqual!(ElementType!S) == char));
uint  strideBack(S)(auto ref S str) if (is(S : const(char[])) || isRandomAccessRange!S && hasLength!S && is(Unqual!(ElementType!S) == char));

 strideBack returns the length of the UTF-8 sequence ending one code unit before index in str.

 strideBack works with both UTF-8 strings and bidirectional ranges of char. If no index is passed, then a bidirectional range will work, but if an index is passed, then a random-access range is required.

index defaults to str.length if none is passed.

Returns
The number of bytes in the UTF-8 sequence.
Throws
May throw a UTFException if str[index] is not one past the end of a valid UTF-8 sequence.
Notes:
 strideBack will not fully verify the validity of the UTF-8 sequence. It will, however, guarantee that index - stride(str, index) is a valid index.

uint  stride(S)(auto ref S str, size_t index) if (is(S : const(wchar[])) || isRandomAccessRange!S && is(Unqual!(ElementType!S) == wchar));
pure @safe uint  stride(S)(auto ref S str) if (is(S : const(wchar[])));

 stride returns the length of the UTF-16 sequence starting at index in str.

 stride works with both UTF-16 strings and ranges of wchar. If no index is passed, then an input range will work, but if an index is passed, then a random-access range is required.

index defaults to 0 if none is passed.

Returns
The number of bytes in the UTF-16 sequence.
Throws
May throw a UTFException if str[index] is not the start of a valid UTF-16 sequence.
Notes:
 stride will only analize the first str[index] element. It will not fully verify the validity of UTF-16 sequence, nor even verify the presence of the sequence: it will not actually guarantee that index +  stride(str, index) <= str.length.

uint  strideBack(S)(auto ref S str, size_t index) if (is(S : const(wchar[])) || isRandomAccessRange!S && is(Unqual!(ElementType!S) == wchar));
uint  strideBack(S)(auto ref S str) if (is(S : const(wchar[])) || isBidirectionalRange!S && is(Unqual!(ElementType!S) == wchar));

 strideBack returns the length of the UTF-16 sequence ending one code unit before index in str.

 strideBack works with both UTF-16 strings and ranges of wchar. If no index is passed, then a bidirectional range will work, but if an index is passed, then a random-access range is required.

index defaults to str.length if none is passed.

Returns
The number of bytes in the UTF-16 sequence.
Throws
May throw a UTFException if str[index] is not one past the end of a valid UTF-16 sequence.
Notes:
stride will only analize the element at str[index - 1] element. It will not fully verify the validity of UTF-16 sequence, nor even verify the presence of the sequence: it will not actually guarantee that stride(str, index) <= index.

uint  stride(S)(auto ref S str, size_t index = 0) if (is(S : const(dchar[])) || isInputRange!S && is(Unqual!(ElementEncodingType!S) == dchar));

 stride returns the length of the UTF-32 sequence starting at index in str.

 stride works with both UTF-32 strings and ranges of dchar.

Returns
The number of bytes in the UTF-32 sequence (always 1).
Throws
Never.

uint  strideBack(S)(auto ref S str, size_t index) if (isRandomAccessRange!S && is(Unqual!(ElementEncodingType!S) == dchar));
uint  strideBack(S)(auto ref S str) if (isBidirectionalRange!S && is(Unqual!(ElementEncodingType!S) == dchar));

 strideBack returns the length of the UTF-32 sequence ending one code unit before index in str.

 strideBack works with both UTF-32 strings and ranges of dchar. If no index is passed, then a bidirectional range will work, but if an index is passed, then a random-access range is required.

index defaults to str.length if none is passed.

Returns
The number of bytes in the UTF-32 sequence (always 1).
Throws
Never.

pure @safe size_t  toUCSindex(C)(const(C)[] str, size_t index) if (isSomeChar!C);

Given index into str and assuming that index is at the start of a UTF sequence,  toUCSindex determines the number of UCS characters up to index. So, index is the index of a code unit at the beginning of a code point, and the return value is how many code points into the string that that code point is.

Examples
    assert(toUCSindex(`hello world`, 7) == 7);
    assert(toUCSindex(`hello world`w, 7) == 7);
    assert(toUCSindex(`hello world`d, 7) == 7);

    assert(toUCSindex(`Ma Chérie`, 7) == 6);
    assert(toUCSindex(`Ma Chérie`w, 7) == 7);
    assert(toUCSindex(`Ma Chérie`d, 7) == 7);

    assert(toUCSindex(`さいごの果実 / ミツバチと科学者`, 9) == 3);
    assert(toUCSindex(`さいごの果実 / ミツバチと科学者`w, 9) == 9);
    assert(toUCSindex(`さいごの果実 / ミツバチと科学者`d, 9) == 9);

pure @safe size_t  toUTFindex(C)(const(C)[] str, size_t n) if (isSomeChar!C);

Given a UCS index n into str, returns the UTF index. So, n is how many code points into the string the code point is, and the array index of the code unit is returned.

Examples
    assert(toUTFindex(`hello world`, 7) == 7);
    assert(toUTFindex(`hello world`w, 7) == 7);
    assert(toUTFindex(`hello world`d, 7) == 7);

    assert(toUTFindex(`Ma Chérie`, 6) == 7);
    assert(toUTFindex(`Ma Chérie`w, 7) == 7);
    assert(toUTFindex(`Ma Chérie`d, 7) == 7);

    assert(toUTFindex(`さいごの果実 / ミツバチと科学者`, 3) == 9);
    assert(toUTFindex(`さいごの果実 / ミツバチと科学者`w, 9) == 9);
    assert(toUTFindex(`さいごの果実 / ミツバチと科学者`d, 9) == 9);

dchar  decode(S)(auto ref S str, ref size_t index) if (!isSomeString!S && isRandomAccessRange!S && hasSlicing!S && hasLength!S && isSomeChar!(ElementType!S));

Decodes and returns the code point starting at str[index]. index is advanced to one past the decoded code point. If the code point is not well-formed, then a UTFException is thrown and index remains unchanged.

 decode will only work with strings and random access ranges of code units with length and slicing, whereas decodeFront will work with any input range of code units.

Throws
UTFException if str[index] is not the start of a valid UTF sequence.

dchar  decodeFront(S)(ref S str, out size_t numCodeUnits) if (!isSomeString!S && isInputRange!S && isSomeChar!(ElementType!S));
dchar  decodeFront(S)(ref S str) if (isInputRange!S && isSomeChar!(ElementType!S));

 decodeFront is a variant of decode which specifically decodes the first code point. Unlike decode ,  decodeFront accepts any input range of code units (rather than just a string or random access range). It also takes the range by ref and pops off the elements as it decodes them. If numCodeUnits is passed in, it gets set to the number of code units which were in the code point which was decoded.

Throws
UTFException if str.front is not the start of a valid UTF sequence. If an exception is thrown, then there is no guarantee as to the number of code units which were popped off, as it depends on the type of range being used and how many code units had to be popped off before the code point was determined to be invalid.

pure @safe size_t  encode(ref char[4] buf, dchar c);
pure @safe size_t  encode(ref wchar[2] buf, dchar c);

Encodes c into the static array, buf, and returns the actual length of the encoded character (a number between 1 and 4 for char[4] buffers and a number between 1 and 2 for wchar[2] buffers).

Throws
UTFException if c is not a valid UTF code point.

pure @safe void  encode(ref char[] str, dchar c);
pure @safe void  encode(ref wchar[] str, dchar c);
pure @safe void  encode(ref dchar[] str, dchar c);

Encodes c in str's encoding and appends it to str.

Throws
UTFException if c is not a valid UTF code point.

pure nothrow @safe ubyte  codeLength(C)(dchar c) if (isSomeChar!C);

Returns the number of code units that are required to encode the code point c when C is the character type used to encode it.

Examples
    assert(codeLength!char('a') == 1);
    assert(codeLength!wchar('a') == 1);
    assert(codeLength!dchar('a') == 1);

    assert(codeLength!char('\U0010FFFF') == 4);
    assert(codeLength!wchar('\U0010FFFF') == 2);
    assert(codeLength!dchar('\U0010FFFF') == 1);

size_t  codeLength(C, InputRange)(InputRange input) if (isInputRange!InputRange && is(ElementType!InputRange : dchar));

Returns the number of code units that are required to encode str in a string whose character type is C. This is particularly useful when slicing one string with the length of another and the two string types use different character types.

Examples
    assert(codeLength!char("hello world") ==
           to!string("hello world").length);
    assert(codeLength!wchar("hello world") ==
           to!wstring("hello world").length);
    assert(codeLength!dchar("hello world") ==
           to!dstring("hello world").length);

    assert(codeLength!char(`プログラミング`) ==
           to!string(`プログラミング`).length);
    assert(codeLength!wchar(`プログラミング`) ==
           to!wstring(`プログラミング`).length);
    assert(codeLength!dchar(`プログラミング`) ==
           to!dstring(`プログラミング`).length);

    string haystack = `Être sans la verité, ça, ce ne serait pas bien.`;
    wstring needle = `Être sans la verité`;
    assert(haystack[codeLength!char(needle) .. $] ==
           `, ça, ce ne serait pas bien.`);

pure @safe void  validate(S)(in S str) if (isSomeString!S);

Checks to see if str is well-formed unicode or not.

Throws
UTFException if str is not well-formed.

pure @safe string  toUTF8(in char[] s);
pure @trusted string  toUTF8(in wchar[] s);
pure @trusted string  toUTF8(in dchar[] s);

Encodes string s into UTF-8 and returns the encoded string.


pure @trusted wstring  toUTF16(in char[] s);
pure @safe wstring  toUTF16(in wchar[] s);
pure @trusted wstring  toUTF16(in dchar[] s);

Encodes string s into UTF-16 and returns the encoded string.


pure @trusted dstring  toUTF32(in char[] s);
pure @trusted dstring  toUTF32(in wchar[] s);
pure @safe dstring  toUTF32(in dchar[] s);

Encodes string s into UTF-32 and returns the encoded string.


template  toUTFz(P)

Returns a C-style zero-terminated string equivalent to str. str must not contain embedded '\0''s as any C function will treat the first '\0' that it sees as the end of the string. If str.empty is true, then a string containing only '\0' is returned.

 toUTFz accepts any type of string and is templated on the type of character pointer that you wish to convert to. It will avoid allocating a new string if it can, but there's a decent chance that it will end up having to allocate a new string - particularly when dealing with character types other than char.

Warning 1: If the result of  toUTFz equals str.ptr, then if anything alters the character one past the end of str (which is the '\0' character terminating the string), then the string won't be zero-terminated anymore. The most likely scenarios for that are if you append to str and no reallocation takes place or when str is a slice of a larger array, and you alter the character in the larger array which is one character past the end of str. Another case where it could occur would be if you had a mutable character array immediately after str in memory (for example, if they're member variables in a user-defined type with one declared right after the other) and that character array happened to start with '\0'. Such scenarios will never occur if you immediately use the zero-terminated string after calling  toUTFz and the C function using it doesn't keep a reference to it. Also, they are unlikely to occur even if you save the zero-terminated string (the cases above would be among the few examples of where it could happen). However, if you save the zero-terminate string and want to be absolutely certain that the string stays zero-terminated, then simply append a '\0' to the string and use its ptr property rather than calling  toUTFz.

Warning 2: When passing a character pointer to a C function, and the C function keeps it around for any reason, make sure that you keep a reference to it in your D code. Otherwise, it may go away during a garbage collection cycle and cause a nasty bug when the C code tries to use it.

Examples
    auto p1 = toUTFz!(char*)("hello world");
    auto p2 = toUTFz!(const(char)*)("hello world");
    auto p3 = toUTFz!(immutable(char)*)("hello world");
    auto p4 = toUTFz!(char*)("hello world"d);
    auto p5 = toUTFz!(const(wchar)*)("hello world");
    auto p6 = toUTFz!(immutable(dchar)*)("hello world"w);

const(wchar)*  toUTF16z(C)(const(C)[] str) if (isSomeChar!C);

 toUTF16z is a convenience function for toUTFz!(const(wchar)*).

Encodes string s into UTF-16 and returns the encoded string.  toUTF16z is suitable for calling the 'W' functions in the Win32 API that take an LPWSTR or LPCWSTR argument.


pure @trusted size_t  count(C)(const(C)[] str) if (isSomeChar!C);

Returns the total number of code points encoded in str.

Supercedes:
This function supercedes toUCSindex .
Standards
Unicode 5.0, ASCII, ISO-8859-1, WINDOWS-1252
Throws
UTFException if str is not well-formed.