3 ms·

The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pret

by gok 8y ago

The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.

acdha 8y ago
It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness. Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.
rstuart4133 8y ago
They are still partially right, and I'd go so far as to say were totally right if efficiency wasn't a concern. From the programmers point of view utf-32 is still by far the easiest memory representation to use. Back in the day, when the Unicode people were pushing it as an alternative to whatever the ISO standard is that solved the same problem, they argued it was better than the ISO standard because the was only one encoding, UCS2 whereas ISO defined 8, 16 and 32 bit encodings. They said they could get away with that because 16 bits was enough for all modern languages (read: languages in use). Windows, Java and Javascript among others drank the kool-aid. But they were wrong, 16 bits was not enough. USC2 morphed into UTF-16, which is actually worse than UTF-8 because if has the byte ordering problem. To make matters worse, Unicode decided to abandon grapheme == code point. (This means there is ä and ä. They look identical, but one is '\xe4' and the other is a combining diaeresis, '\61\u0308', and the latter is now preferred.) I don't know why that did that as it makes string handling fragile, but I notice the encoding chosen for UTF-16 drastically reduced the number of code points available, so I wonder if they were worried about running out. The end result means that makes utf-32 actually harder to parse than utf-8 (which is quite an achievement), so you may as well use utf-8 for everything.

by gok 8y ago

The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.

acdha 8y ago

It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness. Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.

rstuart4133 8y ago

They are still partially right, and I'd go so far as to say were totally right if efficiency wasn't a concern. From the programmers point of view utf-32 is still by far the easiest memory representation to use. Back in the day, when the Unicode people were pushing it as an alternative to whatever the ISO standard is that solved the same problem, they argued it was better than the ISO standard because the was only one encoding, UCS2 whereas ISO defined 8, 16 and 32 bit encodings. They said they could get away with that because 16 bits was enough for all modern languages (read: languages in use). Windows, Java and Javascript among others drank the kool-aid. But they were wrong, 16 bits was not enough. USC2 morphed into UTF-16, which is actually worse than UTF-8 because if has the byte ordering problem. To make matters worse, Unicode decided to abandon grapheme == code point. (This means there is ä and ä. They look identical, but one is '\xe4' and the other is a combining diaeresis, '\61\u0308', and the latter is now preferred.) I don't know why that did that as it makes string handling fragile, but I notice the encoding chosen for UTF-16 drastically reduced the number of code points available, so I wonder if they were worried about running out. The end result means that makes utf-32 actually harder to parse than utf-8 (which is quite an achievement), so you may as well use utf-8 for everything.