5 ms·

What is D's type for a unicode character (code point?)

by emmelaich 5y ago

WalterBright 5y ago
A `string`, which is a typedef for `const(char)[]`, or a readonly view of an array of characters.
- tialaramex 5y ago
  Note that because of the above definition of char (any UTF-8 code unit) these "strings" aren't necessarily valid UTF-8 and you might be better off with its dchar type (which is UTF-32 and thus can hold an entire Unicode code point) It's unclear to me whether there's actually enforcement to ensure char can't become say 0xC0 (this byte can't occur in UTF-8 because it claims to be the prefix of a multi-byte sequence, yet it also implies that sequence should only be one byte long... which isn't multi-byte) I spent a while reading the D website and came away if anything more confused on this topic.
  - WalterBright 5y ago
    Checking to see if the string is valid UTF-8 requires code to execute, which gets in the way of performance. Therefore, you can check with functions like: https://dlang.org/phobos/std_utf.html#validate https://dlang.org/phobos/std_utf.html#validate when it is convenient in your code to do so.
    - tialaramex 5y ago
      I don't see much value in a distinct char type which has no real semantics beyond being an unsigned byte. Calling validate on a whole string makes at least some sense, I don't love it, but it's not crazy. But it's obviously never worth "validating" a single char, so why not just call them bytes (or ubytes if that's the preferred D nomenclature)
      - WalterBright 5y ago
        There are some extra semantics that strings have in D, but you have a point. Why make it special? It's because strings are such a basic thing in programming, that baking them into language has a way of standardizing them. Instead of everyone having their own string type, there is one string type. One of the problems with strings not being a builtin type is what does one do with string tokens? (C++ has this problem with its library string type.) Being an array of bytes in a function signature gives no clue if it is intended to be a string, or an array of other data. It's just a big advantage to building it in, and making it interact smoothly with the other core language features.

What is D's type for a unicode character (code point?)

WalterBright 5y ago

A `string`, which is a typedef for `const(char)[]`, or a readonly view of an array of characters.

tialaramex 5y ago
Note that because of the above definition of char (any UTF-8 code unit) these "strings" aren't necessarily valid UTF-8 and you might be better off with its dchar type (which is UTF-32 and thus can hold an entire Unicode code point) It's unclear to me whether there's actually enforcement to ensure char can't become say 0xC0 (this byte can't occur in UTF-8 because it claims to be the prefix of a multi-byte sequence, yet it also implies that sequence should only be one byte long... which isn't multi-byte) I spent a while reading the D website and came away if anything more confused on this topic.
- WalterBright 5y ago
  Checking to see if the string is valid UTF-8 requires code to execute, which gets in the way of performance. Therefore, you can check with functions like: https://dlang.org/phobos/std_utf.html#validate https://dlang.org/phobos/std_utf.html#validate when it is convenient in your code to do so.
  - tialaramex 5y ago
    I don't see much value in a distinct char type which has no real semantics beyond being an unsigned byte. Calling validate on a whole string makes at least some sense, I don't love it, but it's not crazy. But it's obviously never worth "validating" a single char, so why not just call them bytes (or ubytes if that's the preferred D nomenclature)
    - WalterBright 5y ago
      There are some extra semantics that strings have in D, but you have a point. Why make it special? It's because strings are such a basic thing in programming, that baking them into language has a way of standardizing them. Instead of everyone having their own string type, there is one string type. One of the problems with strings not being a builtin type is what does one do with string tokens? (C++ has this problem with its library string type.) Being an array of bytes in a function signature gives no clue if it is intended to be a string, or an array of other data. It's just a big advantage to building it in, and making it interact smoothly with the other core language features.