4 ms·

There are (a few) very good reasons not to use UTF-8. It's a great encoding but not suitable for all cases. For example, constant time subscripting, or improve

by junon 5y ago

There are (a few) very good reasons not to use UTF-8. It's a great encoding but not suitable for all cases. For example, constant time subscripting, or improved length calculations, are made possible by encodings other than utf-8. But when performance isn't critical, utf-8 should be the default. I don't see a reason for any other encoding.

jfk13 5y ago
> For example, constant time subscripting, or improved length calculations, are made possible by encodings other than utf-8. Assuming you mean different encoding forms of Unicode (rather than entirely different and far less comprehensive character sets, such as ASCII or Latin-1), there are very few use cases where "subscripting" or "length calculations" would benefit significantly from using a different encoding form, because it is rare that individual Unicode code points are the most appropriate units to work with. (If you're happy to sacrifice support for most of the world's writing systems in favour of raw performance for a limited subset of scripts and text operations, that's different.)
ninkendo 5y ago
Constant time subscripting is a myth. There's nothing(*) useful to be obtained by adding a fixed offset to the base of your string, in any unicode encoding, including UTF-32. If you're hoping that a fixed offset gives you a user-percieved character boundary, then you're not handling composed characters or zero-width-joiners or any number of other things that may cause a grapheme cluster to be composed of multiple UTF code points. The "fixed" size of code points in encodings like UTF-32 are just that: code points. Whether a code point corresponds with anything useful, like the boundary of a visible character, will always require linear-time indexing of the string, in any encoding. (*) Approximately nothing. If you're in a position where you've somehow already vetted that the text is of a subset of human languages where you're guaranteed to never have grapheme clusters that occupy more than a single code point, then you maybe have a use case for this, but I'd argue you really just have a bunch of bugs waiting to happen.
- irq-1 5y ago
  > Constant time subscripting is a myth. There's nothing(*) useful to be obtained by adding a fixed offset to the base of your string, in any unicode encoding, including UTF-32. What about UTF-256? Maybe not today, maybe not tomorrow, but someday...
  - ts4z 5y ago
    I know you're kidding, but I want to note that UTF-256 isn't enough. There's an Arabic ligature that decomposes into 20 codepoints. That was already in Unicode 20 years ago. You can probably do something even crazier with the family emoji. These make "single characters" that do not have precomposed forms.
    - pjscott 5y ago
      Also, if you want O(1) indexing by grapheme cluster you can get that with less memory overhead by precomputing a lookup table of the location in the string where you can find every k-th grapheme cluster, for some constant k >= 1. (This requires a single O(n) pass through the string to build the index, but you were always going to have do make at least one such pass through the string for other reasons.)
  - wizzwizz4 5y ago
    Some characters are longer than 32 codepoints.
- nirvdrum 5y ago
  I see this mentioned periodically in discussions about UTF-8 and it just doesn't seem to match reality. Very often you can be certain you're not operating with multi-codepoint grapheme clusters. Whether through string literals, conversion from other types (e.g., numeric to string), restrictions on identifiers, specification for file formats, company-policy on language for source files, conversion from strings with an ASCII charset, etc., you very often can be certain about the contents of that string. And optimizing around that information is considerably faster than a naive linear scan for the string, constantly rediscovering properties about that string. E.g., Ruby runtimes scan the bytes in a string and then cache data about them in value called a code range. Knowing the code range, you can optimize many operations to not require additional linear scans of the string. Knowing a UTF-8 string consists only of ASCII characters can allow operations to be just as fast as if the string truly were ASCII-only (Ruby supports 100+ string encodings). And that fact is used throughout the core library to provide fast implementations of many operations (upcase, downcase, capitalize, gsub, substring, and so on). Moreover, a JIT can generate extremely tight code in those situations. Having to take a linear pass through the string to discover codepoint boundaries incurs a huge performance cost. While all strings could be treated uniformly and use Unicode tables for case mapping and such, the extra overhead is brutal. It has a measurable impact on string-heavy applications, such as template rendering and text processing. In the most general case, yes, you know nothing about the string and can't make any assumptions. You can't even be sure the byte sequence is valid UTF-8. But, very often you do know properties of those strings. And you can manage boundaries where strings with known properties are joined with strings with unknown properties (e.g., variable interpolation in a template file).
deleted 5y ago
[deleted]

by junon 5y ago

jfk13 5y ago

> For example, constant time subscripting, or improved length calculations, are made possible by encodings other than utf-8. Assuming you mean different encoding forms of Unicode (rather than entirely different and far less comprehensive character sets, such as ASCII or Latin-1), there are very few use cases where "subscripting" or "length calculations" would benefit significantly from using a different encoding form, because it is rare that individual Unicode code points are the most appropriate units to work with. (If you're happy to sacrifice support for most of the world's writing systems in favour of raw performance for a limited subset of scripts and text operations, that's different.)

ninkendo 5y ago

Constant time subscripting is a myth. There's nothing(*) useful to be obtained by adding a fixed offset to the base of your string, in any unicode encoding, including UTF-32. If you're hoping that a fixed offset gives you a user-percieved character boundary, then you're not handling composed characters or zero-width-joiners or any number of other things that may cause a grapheme cluster to be composed of multiple UTF code points. The "fixed" size of code points in encodings like UTF-32 are just that: code points. Whether a code point corresponds with anything useful, like the boundary of a visible character, will always require linear-time indexing of the string, in any encoding. (*) Approximately nothing. If you're in a position where you've somehow already vetted that the text is of a subset of human languages where you're guaranteed to never have grapheme clusters that occupy more than a single code point, then you maybe have a use case for this, but I'd argue you really just have a bunch of bugs waiting to happen.

irq-1 5y ago
> Constant time subscripting is a myth. There's nothing(*) useful to be obtained by adding a fixed offset to the base of your string, in any unicode encoding, including UTF-32. What about UTF-256? Maybe not today, maybe not tomorrow, but someday...
- ts4z 5y ago
  I know you're kidding, but I want to note that UTF-256 isn't enough. There's an Arabic ligature that decomposes into 20 codepoints. That was already in Unicode 20 years ago. You can probably do something even crazier with the family emoji. These make "single characters" that do not have precomposed forms.
  - pjscott 5y ago
    Also, if you want O(1) indexing by grapheme cluster you can get that with less memory overhead by precomputing a lookup table of the location in the string where you can find every k-th grapheme cluster, for some constant k >= 1. (This requires a single O(n) pass through the string to build the index, but you were always going to have do make at least one such pass through the string for other reasons.)
- wizzwizz4 5y ago
  Some characters are longer than 32 codepoints.
nirvdrum 5y ago
I see this mentioned periodically in discussions about UTF-8 and it just doesn't seem to match reality. Very often you can be certain you're not operating with multi-codepoint grapheme clusters. Whether through string literals, conversion from other types (e.g., numeric to string), restrictions on identifiers, specification for file formats, company-policy on language for source files, conversion from strings with an ASCII charset, etc., you very often can be certain about the contents of that string. And optimizing around that information is considerably faster than a naive linear scan for the string, constantly rediscovering properties about that string. E.g., Ruby runtimes scan the bytes in a string and then cache data about them in value called a code range. Knowing the code range, you can optimize many operations to not require additional linear scans of the string. Knowing a UTF-8 string consists only of ASCII characters can allow operations to be just as fast as if the string truly were ASCII-only (Ruby supports 100+ string encodings). And that fact is used throughout the core library to provide fast implementations of many operations (upcase, downcase, capitalize, gsub, substring, and so on). Moreover, a JIT can generate extremely tight code in those situations. Having to take a linear pass through the string to discover codepoint boundaries incurs a huge performance cost. While all strings could be treated uniformly and use Unicode tables for case mapping and such, the extra overhead is brutal. It has a measurable impact on string-heavy applications, such as template rendering and text processing. In the most general case, yes, you know nothing about the string and can't make any assumptions. You can't even be sure the byte sequence is valid UTF-8. But, very often you do know properties of those strings. And you can manage boundaries where strings with known properties are joined with strings with unknown properties (e.g., variable interpolation in a template file).

deleted 5y ago

[deleted]