4 ms·

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But more broadly, being able to assume $encoding everywhere is unrealistic. Write your programs/whatevers allowing your users to be aware of and configure encodings. It might not be ideal, but such is life.

flohofwoe 7y ago
I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone, and that UTF-16 isn't a fixed-length encoding either). From that perspective, keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing.
- camgunz 7y ago
  I'm gonna do little quotes but, I don't mean to be passive aggressive. It's just that this stuff comes up all the time > I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone... This should be the responsibility of a string library internally, and if you're saving data to disk or sending it over the network, you should be serializing to a specific format. That format can be UTF-8, or it can be whatever, depending on your application's needs. > and that UTF-16 isn't a fixed-length encoding either) We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. > keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing. Well, you're essentially saying "I know about your use case better than you do". It might be important to me to not blow space on UTF-8. But if my platform/libraries have bought into "UTF-8 everywhere" and don't give me knobs to configure the encoding, I have no recourse. And that's the entire basis for this. It's "having to mess with encodings is worse than the application-specific benefits of being able to choose an encoding". I think that's... at best an impossible claim and at worst pretty arrogant. Again here I don't mean you, but this "UTF-8 everywhere" thing.
  - jeltz 7y ago
    > We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. So what do you suggest? UTF-16 and UTF-32 encourage this even more.
    - camgunz 7y ago
      Yeah, ASCII is such a powerful mental model that I think anyone working with Unicode made a lot of concessions to convert people, no argument there. But I think we need to say we're done with that and move on to phase 2. Here's what I advocate: - Encodings should be configurable. Programmers get to decide what format their strings are internally, users get to decide what encoding programs use when dealing with filenames or saving data to disk, etc. Defaults matter, and we should employ smarts, but we should never say "I know best" and remove those knobs. - Engineers need to internalize that "strings" conceal mountains of complexity (because written language is complex), and default to using libraries, to manage them. We should start view manual string manipulation as an anti-pattern. There isn't an encoding out there that we can all standardize on that makes this untrue, again because written language is complex.
  - eMSF 7y ago
    >We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. Mistaking a variable-width encoding for a fixed-width one is specifically a UTF-16 problem. UTF-8 is so obviously not fixed-width that such an error could not happen by a mistake, because even before widespread use of emojis, multibyte sequences were not in any way a corner case for UTF-8 text (for additional reference, compare UTF-16 String APIs in Java/JavaScript/etc. with UTF-8 ones in, say, Rust and Go, and see which ones allow you to easily split a string where you shouldn't be able to, or access "half-chars" as a datatype called "char".)
    - camgunz 7y ago
      I mean, I think we're both in the realm of [citation needed] here. I would argue that people index into strings quite a lot--whether that's because we thought UCS-2 would be enough for anybody or UTF-8 == ASCII and "it's probably fine" is academic. The solution is the same though: don't index into strings, don't assume an encoding until you've validated. That makes any "advantage" UTF-8 has disappear. If you really think no one made this mistake with UTF-8, just read up on Python 3.
crazygringo 7y ago
> not really a good reason to support UTF-8 over UTF-16 Of course there is, the fact that if you're dealing only with ASCII characters then it's backwards-compatible. Which is a nice convenience in a great number of situations programmers encounter. The minor details of efficiency of an encoding these days isn't particularly relevant -- sure UTF-16 is better for Chinese, but the average webpage usually does have way more markup, CSS and JavaScript than text, and gzip-ing it on delivery will result in a similar payload totally independent of the encoding you choose.
- camgunz 7y ago
  UTF-8's ASCII compatibility is an anti-feature; it's allowed us to continue to use systems that are encoding naive (in practice ASCII-only). It's no substitute for creating encoding-aware programs, libraries, and systems. The vast majority of text is not in HTML or XML, and there's no reason you can't use Chinese characters in JavaScript besides (your strings and variable/class/component/file names will surely outpace your use of keywords).
  - jeltz 7y ago
    The reasons most programmers use English in their source code has nothing to do with file size (for that their are JS minimizes) or supported encodings. It has to do with that two things, English is the most used language in the industry so if you want to cooperate with programmers from other parts of the world English is a good idea and because it frankly looks ugly to mix languages in the same file so when the standard library is in English your source code will be too. So since most source code is in English (and for JS is minimized) UTF-8 works perfectly there too.
  - crazygringo 7y ago
    It's not an anti-feature, it's a benefit that is a huge asset in the real world. For example, you can be on a legacy ASCII system, inspect a modern UTF-8 file, and if it's in a Latin language then it will still be readable as opposed to gibberish. Yes all modern tools should be (and these days generally are) encoding-aware, but in the real world we're stuck with a lot of legacy tools too. And of course the vast majority of transmitted digital text is in HTML and similar! What do you think it's in instead? By sheer quantity of digital words consumed by the average person, it's news and social media delivered in browsers (HTML), followed by apps (still using HTML markup to a huge degree) and ebooks (ePub based on HTML). And of course plenty of JSON and XML wrapping too. And of course you can you Chinese characters in JavaScript/JSON, but development teams are increasingly international and English is the de-facto lingua franca.
jcranmer 7y ago
> There's not really a good reason to support UTF-8 over UTF-16 Two big reasons: 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this. > Write your programs/whatevers allowing your users to be aware of and configure encodings. Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible. But if you're emitting stuff... you should try hard to make sure that UTF-8 is the only output charset you have to support. Letting people select non-UTF-8 charsets for output adds lots of complication (now you have to have error paths for characters that can't be emitted), and you need to have strong justification for why your code needs that complication.
- mark-r 7y ago
  Every program that purports to support Unicode should be tested with a bunch of emoticons.
  - coolreader18 7y ago
    Do you mean emoji? I don't see what the issue would be with [{}:();P\[\],.<>/~-_+=XD]
    - mark-r 7y ago
      Yes, that's what I meant. I knew I was using the wrong word but couldn't remember the right one.
- camgunz 7y ago
  > 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. Eh, realistically if you're doing this, you should be validating it like converting from one encoding to another anyway. I get that people won't and haven't, but that's because UTF-8 has this anti-feature where ASCII is compatible with it, and that's led to a lot of problems. > 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this. This is one of those problems, and it's way worse with UTF-8 because it encodes ASCII the same way ASCII does. It's let programmers stay naive about this stuff for... decades? > Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible. There are all kinds of reasons to not use UTF-8. tialaramex pointed out one above. "UTF-8 everywhere" is simply unrealistic, and it forces a lot of applications to be slower, or to take on unnecessary complexity. Maybe it's worth it to "never have to think about encodings again", but that's pretty hard to verify and there's no way it happens in our lifetimes anyway. > and you need to have strong justification for why your code needs that complication. Yeah see, I strongly disagree with this. I'll choose whatever encoding I like, thanks. Maybe you don't mean to be super prescriptive here, but I think a little more consideration by UTF-8 advocates wouldn't hurt.
jeltz 7y ago
But is it really a plurality? Portuguese, English, Spanish, Turkish, Vietnamese, French, Indonesian and German are stored more efficiently in UTF-8 while Chinese, Korean and Japanese are stored less effeciently. My gut feel is that more people use the Latin script than people using CJK scripts. Indic scripts, Thai, Cyrillic, etc are stored using two bytes in both UTF-8 AND UTF-16. And thus ignores markup which is in ascii.
- camgunz 7y ago
  Looking at the basic multilingual plane [1], UTF-8 will use > 2 bytes to encode essentially anything that isn't: * ASCII/Latin * Cyrillic * Greek * Most of Arabic That leaves out: * China * India * Japan * Korea * All of Southeast Asia Re: markup, think about any text that's in a database, stored in RAM, or stored on a disk--relatively little of it will be in noisy ASCII markup formats like HTML or XML. [1]: https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilin...
  - jeltz 7y ago
    > All of Southeast Asia Did you forget Indonesia, Vietnam, Malaysia, Brunei and the Philippines?
    - camgunz 7y ago
      Again, here's what UTF-8 will use <= 2 bytes for: Basic Latin (Lower half of ISO/IEC 8859-1: ISO/IEC 646:1991-IRV aka ASCII) (0000–007F) Latin-1 Supplement (Upper half of ISO/IEC 8859-1) (0080–00FF) Latin Extended-A (0100–017F) Latin Extended-B (0180–024F) IPA Extensions (0250–02AF) Spacing Modifier Letters (02B0–02FF) Combining Diacritical Marks (0300–036F) Greek and Coptic (0370–03FF) Cyrillic (0400–04FF) Cyrillic Supplement (0500–052F) Armenian (0530–058F) Aramaic Scripts: Hebrew (0590–05FF) Arabic (0600–06FF) Syriac (0700–074F) Arabic Supplement (0750–077F) Thaana (0780–07BF) N'Ko (07C0–07FF) In UTF-8, everything over U+0800 requires > 2 bytes. Am I misunderstanding something? It's possible.

flohofwoe 7y ago

I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone, and that UTF-16 isn't a fixed-length encoding either). From that perspective, keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing.

camgunz 7y ago
I'm gonna do little quotes but, I don't mean to be passive aggressive. It's just that this stuff comes up all the time > I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone... This should be the responsibility of a string library internally, and if you're saving data to disk or sending it over the network, you should be serializing to a specific format. That format can be UTF-8, or it can be whatever, depending on your application's needs. > and that UTF-16 isn't a fixed-length encoding either) We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. > keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing. Well, you're essentially saying "I know about your use case better than you do". It might be important to me to not blow space on UTF-8. But if my platform/libraries have bought into "UTF-8 everywhere" and don't give me knobs to configure the encoding, I have no recourse. And that's the entire basis for this. It's "having to mess with encodings is worse than the application-specific benefits of being able to choose an encoding". I think that's... at best an impossible claim and at worst pretty arrogant. Again here I don't mean you, but this "UTF-8 everywhere" thing.
- jeltz 7y ago
  > We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. So what do you suggest? UTF-16 and UTF-32 encourage this even more.
  - camgunz 7y ago
    Yeah, ASCII is such a powerful mental model that I think anyone working with Unicode made a lot of concessions to convert people, no argument there. But I think we need to say we're done with that and move on to phase 2. Here's what I advocate: - Encodings should be configurable. Programmers get to decide what format their strings are internally, users get to decide what encoding programs use when dealing with filenames or saving data to disk, etc. Defaults matter, and we should employ smarts, but we should never say "I know best" and remove those knobs. - Engineers need to internalize that "strings" conceal mountains of complexity (because written language is complex), and default to using libraries, to manage them. We should start view manual string manipulation as an anti-pattern. There isn't an encoding out there that we can all standardize on that makes this untrue, again because written language is complex.
- eMSF 7y ago
  >We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. Mistaking a variable-width encoding for a fixed-width one is specifically a UTF-16 problem. UTF-8 is so obviously not fixed-width that such an error could not happen by a mistake, because even before widespread use of emojis, multibyte sequences were not in any way a corner case for UTF-8 text (for additional reference, compare UTF-16 String APIs in Java/JavaScript/etc. with UTF-8 ones in, say, Rust and Go, and see which ones allow you to easily split a string where you shouldn't be able to, or access "half-chars" as a datatype called "char".)
  - camgunz 7y ago
    I mean, I think we're both in the realm of [citation needed] here. I would argue that people index into strings quite a lot--whether that's because we thought UCS-2 would be enough for anybody or UTF-8 == ASCII and "it's probably fine" is academic. The solution is the same though: don't index into strings, don't assume an encoding until you've validated. That makes any "advantage" UTF-8 has disappear. If you really think no one made this mistake with UTF-8, just read up on Python 3.

crazygringo 7y ago

> not really a good reason to support UTF-8 over UTF-16 Of course there is, the fact that if you're dealing only with ASCII characters then it's backwards-compatible. Which is a nice convenience in a great number of situations programmers encounter. The minor details of efficiency of an encoding these days isn't particularly relevant -- sure UTF-16 is better for Chinese, but the average webpage usually does have way more markup, CSS and JavaScript than text, and gzip-ing it on delivery will result in a similar payload totally independent of the encoding you choose.

camgunz 7y ago
UTF-8's ASCII compatibility is an anti-feature; it's allowed us to continue to use systems that are encoding naive (in practice ASCII-only). It's no substitute for creating encoding-aware programs, libraries, and systems. The vast majority of text is not in HTML or XML, and there's no reason you can't use Chinese characters in JavaScript besides (your strings and variable/class/component/file names will surely outpace your use of keywords).
- jeltz 7y ago
  The reasons most programmers use English in their source code has nothing to do with file size (for that their are JS minimizes) or supported encodings. It has to do with that two things, English is the most used language in the industry so if you want to cooperate with programmers from other parts of the world English is a good idea and because it frankly looks ugly to mix languages in the same file so when the standard library is in English your source code will be too. So since most source code is in English (and for JS is minimized) UTF-8 works perfectly there too.
- crazygringo 7y ago
  It's not an anti-feature, it's a benefit that is a huge asset in the real world. For example, you can be on a legacy ASCII system, inspect a modern UTF-8 file, and if it's in a Latin language then it will still be readable as opposed to gibberish. Yes all modern tools should be (and these days generally are) encoding-aware, but in the real world we're stuck with a lot of legacy tools too. And of course the vast majority of transmitted digital text is in HTML and similar! What do you think it's in instead? By sheer quantity of digital words consumed by the average person, it's news and social media delivered in browsers (HTML), followed by apps (still using HTML markup to a huge degree) and ebooks (ePub based on HTML). And of course plenty of JSON and XML wrapping too. And of course you can you Chinese characters in JavaScript/JSON, but development teams are increasingly international and English is the de-facto lingua franca.

jcranmer 7y ago

> There's not really a good reason to support UTF-8 over UTF-16 Two big reasons: 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this. > Write your programs/whatevers allowing your users to be aware of and configure encodings. Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible. But if you're emitting stuff... you should try hard to make sure that UTF-8 is the only output charset you have to support. Letting people select non-UTF-8 charsets for output adds lots of complication (now you have to have error paths for characters that can't be emitted), and you need to have strong justification for why your code needs that complication.

mark-r 7y ago
Every program that purports to support Unicode should be tested with a bunch of emoticons.
- coolreader18 7y ago
  Do you mean emoji? I don't see what the issue would be with [{}:();P\[\],.<>/~-_+=XD]
  - mark-r 7y ago
    Yes, that's what I meant. I knew I was using the wrong word but couldn't remember the right one.
camgunz 7y ago
> 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. Eh, realistically if you're doing this, you should be validating it like converting from one encoding to another anyway. I get that people won't and haven't, but that's because UTF-8 has this anti-feature where ASCII is compatible with it, and that's led to a lot of problems. > 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this. This is one of those problems, and it's way worse with UTF-8 because it encodes ASCII the same way ASCII does. It's let programmers stay naive about this stuff for... decades? > Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible. There are all kinds of reasons to not use UTF-8. tialaramex pointed out one above. "UTF-8 everywhere" is simply unrealistic, and it forces a lot of applications to be slower, or to take on unnecessary complexity. Maybe it's worth it to "never have to think about encodings again", but that's pretty hard to verify and there's no way it happens in our lifetimes anyway. > and you need to have strong justification for why your code needs that complication. Yeah see, I strongly disagree with this. I'll choose whatever encoding I like, thanks. Maybe you don't mean to be super prescriptive here, but I think a little more consideration by UTF-8 advocates wouldn't hurt.

jeltz 7y ago

But is it really a plurality? Portuguese, English, Spanish, Turkish, Vietnamese, French, Indonesian and German are stored more efficiently in UTF-8 while Chinese, Korean and Japanese are stored less effeciently. My gut feel is that more people use the Latin script than people using CJK scripts. Indic scripts, Thai, Cyrillic, etc are stored using two bytes in both UTF-8 AND UTF-16. And thus ignores markup which is in ascii.

camgunz 7y ago
Looking at the basic multilingual plane [1], UTF-8 will use > 2 bytes to encode essentially anything that isn't: * ASCII/Latin * Cyrillic * Greek * Most of Arabic That leaves out: * China * India * Japan * Korea * All of Southeast Asia Re: markup, think about any text that's in a database, stored in RAM, or stored on a disk--relatively little of it will be in noisy ASCII markup formats like HTML or XML. [1]: https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilin...
- jeltz 7y ago
  > All of Southeast Asia Did you forget Indonesia, Vietnam, Malaysia, Brunei and the Philippines?
  - camgunz 7y ago
    Again, here's what UTF-8 will use <= 2 bytes for: Basic Latin (Lower half of ISO/IEC 8859-1: ISO/IEC 646:1991-IRV aka ASCII) (0000–007F) Latin-1 Supplement (Upper half of ISO/IEC 8859-1) (0080–00FF) Latin Extended-A (0100–017F) Latin Extended-B (0180–024F) IPA Extensions (0250–02AF) Spacing Modifier Letters (02B0–02FF) Combining Diacritical Marks (0300–036F) Greek and Coptic (0370–03FF) Cyrillic (0400–04FF) Cyrillic Supplement (0500–052F) Armenian (0530–058F) Aramaic Scripts: Hebrew (0590–05FF) Arabic (0600–06FF) Syriac (0700–074F) Arabic Supplement (0750–077F) Thaana (0780–07BF) N'Ko (07C0–07FF) In UTF-8, everything over U+0800 requires > 2 bytes. Am I misunderstanding something? It's possible.