3 ms·

It's always the tradeoff, some operations are simpler on UTF-32 but they have additional memory (and therefore cache) footprint and since you typically don't wa

by simias 4y ago

It's always the tradeoff, some operations are simpler on UTF-32 but they have additional memory (and therefore cache) footprint and since you typically don't want to use UTF-32 externally you have to convert back and forth which is not free. I think these days people don't bother with UTF-32 too much because it's not even like you have a clean "one 32bit int, one character" relation anyway since some characters can be built from multiple codepoints. Since generally most code manipulating character strings are interested in characters and not codepoints, UTF-32 is effectively a variable-length encoding too...

tialaramex 4y ago
Right, somebody else might have actual metrics but I'd have guessed actual regular expression patterns are split something like: 90% Only care about ASCII, thus individual bytes in UTF-8, and so UTF-32 just wastes memory 1% Care about individual code points, but spread over multiple bytes (e.g. the double dagger ‡), UTF-32 is perfect 9% Care about multiple code points (to form e.g. a Flag, or é written in combining form, or two women kissing) and so UTF-32 doesn't really help again
- bmn__ 4y ago
  Applications and text-processing libraries are free use an internal text encoding or data structure, if you will, which does not suffer from the drawbacks. Conversions to encodings suitable for data exchange (e.g. UTF-8) are performed at the I/O boundaries.
  - tialaramex 4y ago
    Sure, and if stuff lives inside your "application and text-processing library" for long enough and performs enough of the thusly optimised actions without leaving, you might even amortize the cost of the work you did at the edges. But probably not.
layer8 4y ago
Another factor is that nowadays machine code execution is much faster than memory accesses, so the trade-off of requiring more program logic to process a more compact format makes a lot of sense.

by simias 4y ago

tialaramex 4y ago

Right, somebody else might have actual metrics but I'd have guessed actual regular expression patterns are split something like: 90% Only care about ASCII, thus individual bytes in UTF-8, and so UTF-32 just wastes memory 1% Care about individual code points, but spread over multiple bytes (e.g. the double dagger ‡), UTF-32 is perfect 9% Care about multiple code points (to form e.g. a Flag, or é written in combining form, or two women kissing) and so UTF-32 doesn't really help again

bmn__ 4y ago
Applications and text-processing libraries are free use an internal text encoding or data structure, if you will, which does not suffer from the drawbacks. Conversions to encodings suitable for data exchange (e.g. UTF-8) are performed at the I/O boundaries.
- tialaramex 4y ago
  Sure, and if stuff lives inside your "application and text-processing library" for long enough and performs enough of the thusly optimised actions without leaving, you might even amortize the cost of the work you did at the edges. But probably not.

layer8 4y ago

Another factor is that nowadays machine code execution is much faster than memory accesses, so the trade-off of requiring more program logic to process a more compact format makes a lot of sense.