3 ms·

Could we all agree to not use the word "unicode" when talking about encoding (ie. how code points are serialized to bytes)? Unicode (ie. the standard set forth

Could we all agree to not use the word "unicode" when talking about encoding (ie. how code points are serialized to bytes)? Unicode (ie. the standard set forth by the unicode consortium) has nothing to say about encoding. I think what you mean is "Go's 'everything is a UTF-8 string' approach", but I'm not familiar enough with Go's internal encoding to know. For instance, you mention "Don't handle the decode errors", but I can only assume by that you mean UTF-8's decode errors, since UTF-8 has the possibility of having encoding errors, where things like UTF-32 do not. They're both Unicode encodings, so it makes no sense to say it's "Unicode"'s decode errors. I think the author of this article falls into the same trap all over the place. He uses the word "Unicode" to refer to an encoding all over the place. Until he is able to distinguish the difference between the Unicode standard and all its various encoding methods, there's not much point in reading his article. (And no, Microsoft's braindead decision to use the word "Unicode" to mean "UCS-2", and later ret-conning it to mean "UTF-16", doesn't count. Don't perpetuate the stupid.)

captaincrowbar 13y ago
"Unicode (ie. the standard set forth by the unicode consortium) has nothing to say about encoding." Yes it does. The UTF-8/16/32 encodings are defined in the Unicode standard (chapter 3.9, page 89), they're not some kind of aftermarket add-on. "UTF-8 has the possibility of having encoding errors, where things like UTF-32 do not." Also not true. The set of valid Unicode code points is the same regardless of the encoding (0-0xd7ff and 0xe000-0x10ffff); anything outside those two ranges is just as invalid in UTF-32 as its UTF-8 encoded equivalent would be. A stream containing nothing but 0xff bytes, for example, would be illegal regardless of which UTF you tried to interpret it as.
- ninkendo 13y ago
  > Yes it does. The UTF-8/16/32 encodings are defined in the Unicode standard (chapter 3.9, page 89), they're not some kind of aftermarket add-on. Fair enough, but would you not agree that the word is overloaded? The fact that a few encodings are listed in the Unicode standard doesn't make it ok to wave your hand and call it all "Unicode" in essays like these where it very much matters what actual encoding you're talking about. If you mean UTF-8, stay UTF-8, etc.

captaincrowbar 13y ago

"Unicode (ie. the standard set forth by the unicode consortium) has nothing to say about encoding." Yes it does. The UTF-8/16/32 encodings are defined in the Unicode standard (chapter 3.9, page 89), they're not some kind of aftermarket add-on. "UTF-8 has the possibility of having encoding errors, where things like UTF-32 do not." Also not true. The set of valid Unicode code points is the same regardless of the encoding (0-0xd7ff and 0xe000-0x10ffff); anything outside those two ranges is just as invalid in UTF-32 as its UTF-8 encoded equivalent would be. A stream containing nothing but 0xff bytes, for example, would be illegal regardless of which UTF you tried to interpret it as.

ninkendo 13y ago
> Yes it does. The UTF-8/16/32 encodings are defined in the Unicode standard (chapter 3.9, page 89), they're not some kind of aftermarket add-on. Fair enough, but would you not agree that the word is overloaded? The fact that a few encodings are listed in the Unicode standard doesn't make it ok to wave your hand and call it all "Unicode" in essays like these where it very much matters what actual encoding you're talking about. If you mean UTF-8, stay UTF-8, etc.