4 ms·

> most modern software uses UTF-8 [citation needed] UTF-8 is used for storage and serialization, yes, but most mainstream programming languages store unicode

> most modern software uses UTF-8 [citation needed] UTF-8 is used for storage and serialization, yes, but most mainstream programming languages store unicode strings in memory as UTF-16. The notable exception is Rust, which does indeed use UTF-8 even in memory.

loeg 3y ago
> most mainstream programming languages store unicode strings in memory as UTF-16 Uh. What languages do you have in mind? C/C++ don't have any preference for UTF-16. Python3 doesn't use UTF-16. As you mentioned, Rust doesn't.
- steeleduncan 3y ago
  java, .net, apple ecosystem (i.e. swift/objc),...
  - Longhanks 3y ago
    Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/
- grishka 3y ago
  Java, C# and the rest of .net, JavaScript, Swift/ObjC. C/C++ don't have a preference but, for example, if you're writing for Windows, you'd probably want to use wide characters. I'm not sure what kind of strings Linux GUIs use but I suspect it's wide characters as well.
  - 7jjjjjjj 3y ago
    Qt uses UTF-16, Gtk uses UTF-8.
  - Longhanks 3y ago
    Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/
- MatmaRex 3y ago
  In addition to the examples already given, JavaScript as well (although modern implementations do wacky stuff with the internal representations, the APIs are required to expose only UTF-16).
chrismorgan 3y ago
> most mainstream programming languages store unicode strings in memory as UTF-16 This is increasingly frequently false. For stupid historical reasons, far too many languages adopted potentially-ill-formed UTF-16 semantics (never well-formed, sadly), but this makes for such a bad representation that they’ve mostly abandoned it as a sole representation, in favour of more complicated arrangements. For example, Java strings can internally be single-byte Latin-1 or two-byte UTF-16. JavaScript engines do similar tricks these days, and although Python 3 just never did UTF-16 at all (it uses code point semantics, which is almost worse than UTF-16 semantics) it does a 1-byte/2-byte/4-byte hybrid representation thing too. For one of the most extreme examples of becoming unglued from UTF-16, the Servo browser engine manages to use WTF-8 (UTF-8 plus lone surrogates, to allow representing ill-formed UTF-16) despite being forced by these historical reasons to expose UTF-16 semantics, and although the performance of some operations are harmed by it, others are improved, and in the balance it was a pretty convincing win when they tested it (which many people did not expect). PyPy is also a great example of changing your encoding to one that has mismatched semantics: it applies UTF-8 (or I suppose it must actually be UTF-8 plus surrogates) to Python, and likewise they found it surprisingly good for performance. These things give me hope that even environments like the JVM and .NET CLR might eventually manage to switch their internal string representations to UTF-8 or almost-UTF-8. But also I emphasise that these are historical languages that are stuck with the horrendous 16-bit decisions of the early-to-mid-’90s. When you look at newer languages, they almost always choose something more sensible, and UTF-8 is almost always at the heart of it.
- grishka 3y ago
  > it uses code point semantics, which is almost worse than UTF-16 semantics What's so bad about storing a unicode string as a series of codepoints? UTF-8 is also a series of codepoints in essence. > When you look at newer languages, they almost always choose something more sensible, and UTF-8 is almost always at the heart of it. I still think that UTF-8 for memory is a bad idea. For example, it wastes a lot of bits to make sure that a string can be decoded starting from an arbitrary byte offset. This is generally a desirable property for files and some networking applications, but it's an absolute waste of space when used in memory.
IshKebab 3y ago
That's arguably because those languages are not modern. Nobody would design a modern language today that uses UTF-16 (unless they really need compatibility with Java or JavaScript). Go also uses UTF-8 btw. I think you're being needlessly pedantic (I know it's unusual for HN).

loeg 3y ago

> most mainstream programming languages store unicode strings in memory as UTF-16 Uh. What languages do you have in mind? C/C++ don't have any preference for UTF-16. Python3 doesn't use UTF-16. As you mentioned, Rust doesn't.

steeleduncan 3y ago
java, .net, apple ecosystem (i.e. swift/objc),...
- Longhanks 3y ago
  Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/
grishka 3y ago
Java, C# and the rest of .net, JavaScript, Swift/ObjC. C/C++ don't have a preference but, for example, if you're writing for Windows, you'd probably want to use wide characters. I'm not sure what kind of strings Linux GUIs use but I suspect it's wide characters as well.
- 7jjjjjjj 3y ago
  Qt uses UTF-16, Gtk uses UTF-8.
- Longhanks 3y ago
  Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/
MatmaRex 3y ago
In addition to the examples already given, JavaScript as well (although modern implementations do wacky stuff with the internal representations, the APIs are required to expose only UTF-16).

chrismorgan 3y ago

> most mainstream programming languages store unicode strings in memory as UTF-16 This is increasingly frequently false. For stupid historical reasons, far too many languages adopted potentially-ill-formed UTF-16 semantics (never well-formed, sadly), but this makes for such a bad representation that they’ve mostly abandoned it as a sole representation, in favour of more complicated arrangements. For example, Java strings can internally be single-byte Latin-1 or two-byte UTF-16. JavaScript engines do similar tricks these days, and although Python 3 just never did UTF-16 at all (it uses code point semantics, which is almost worse than UTF-16 semantics) it does a 1-byte/2-byte/4-byte hybrid representation thing too. For one of the most extreme examples of becoming unglued from UTF-16, the Servo browser engine manages to use WTF-8 (UTF-8 plus lone surrogates, to allow representing ill-formed UTF-16) despite being forced by these historical reasons to expose UTF-16 semantics, and although the performance of some operations are harmed by it, others are improved, and in the balance it was a pretty convincing win when they tested it (which many people did not expect). PyPy is also a great example of changing your encoding to one that has mismatched semantics: it applies UTF-8 (or I suppose it must actually be UTF-8 plus surrogates) to Python, and likewise they found it surprisingly good for performance. These things give me hope that even environments like the JVM and .NET CLR might eventually manage to switch their internal string representations to UTF-8 or almost-UTF-8. But also I emphasise that these are historical languages that are stuck with the horrendous 16-bit decisions of the early-to-mid-’90s. When you look at newer languages, they almost always choose something more sensible, and UTF-8 is almost always at the heart of it.

grishka 3y ago
> it uses code point semantics, which is almost worse than UTF-16 semantics What's so bad about storing a unicode string as a series of codepoints? UTF-8 is also a series of codepoints in essence. > When you look at newer languages, they almost always choose something more sensible, and UTF-8 is almost always at the heart of it. I still think that UTF-8 for memory is a bad idea. For example, it wastes a lot of bits to make sure that a string can be decoded starting from an arbitrary byte offset. This is generally a desirable property for files and some networking applications, but it's an absolute waste of space when used in memory.

IshKebab 3y ago

That's arguably because those languages are not modern. Nobody would design a modern language today that uses UTF-16 (unless they really need compatibility with Java or JavaScript). Go also uses UTF-8 btw. I think you're being needlessly pedantic (I know it's unusual for HN).