5 ms·

tl;dr It's nearly 2013, if you're still stuck at the byte/Unicode level you're doing it wrong: nowadays people are using programming languages which have abstr

tl;dr It's nearly 2013, if you're still stuck at the byte/Unicode level you're doing it wrong: nowadays people are using programming languages which have abstraction for strings and characters. I can't answer about parent about Go specifically but I may have the beginning of an answer. Do you realize all your questions are really low-level details that are entirely taken care of by default APIs in quite some languages? The whole byte/Unicode you keep insisting on is a strawman: at the right level of abstraction you're dealing with neither. You're dealing with "strings" made of "characters". I don't remember when was the last time I had to manually read bytes / data and "convert" them using whatever character encoding. You're questions just feel weird. As if the early nineties called and wanted their byte/UTF-8 encoders developers back or something. I realize this may be a very true concern when working with languages that didn't deal correctly with the string/characters abstraction and which are still stuck in the byte/Unicode(byte or bytes) mindset but you have to realize that several languages and techniques allow to simply dodge these matters altogether. For example, Java doesn't specify the encoding of Java source file: great, we mandate (and enforce with scripts/hooks) that all .java file contain only ASCII characters. We use properties file (whose encoding is defined) to contain our UTF-8 characters. Wanna transmit text? Well, use a format which allows to specify the encoding of your text. It's really that simple and, on the other end of the wire, it's going to become automagically again "strings" and "characters". Another "detail": Go is one the only language to specify that source file have to be encoded in UTF-8. This is a major win too. (if you've never seen ISO-Latin-1 / UTF-8 / MacRoman / etc. mismatch in source file then you've never worked in a diversified enough team / environment ; )

wrl 14y ago
Do you realize all your questions are really low-level details that are entirely taken care of by default APIs in quite some languages? Somebody's got to write those APIs. d0mine's questions are valid.
dalke 14y ago
"low-level details that are entirely taken care of by default APIs in quite some languages" They are not "taken care of"; it's only that the sharp edges are smoothed down a bit. Here's a hard one. In Go, if you read a directory to get a list of filenames, do you get a list of (Unicode) strings, or a list of byte sequences? Does it depend on the filesystem? What if the filesystem is nominally UTF-8 encoded, or UTF-16 encoded, but the underlying filesystem API does no validation, so that illegal UTF-8/-16 sequences are allowed? Do you have a way to get the filenames as bytes for that case? How do you show those strings in a GUI? Can someone enter a path name for a byte sequence which can't be represented in Unicode? The Python developers worked hard to get a reasonably good solution, but there are still hard details worry about. Can you point out the Go solution? Or is this one of the corner cases you've not had to deal with? Suppose you've read three strings, one with "Å" (LATIN CAPITAL LETTER A WITH RING ABOVE), the second with "Å" (ANGSTROM SIGN), and the third using the code point sequence "U+0041 U+030A" (LATIN LETTER A and COMBINING RING ABOVE ("°")). Are these the same strings? Is there a way to normalize the Unicode string to make them the same? A toolkit can implement the normalizations, but can't know when to do them. In your code, have you personally dealt with Unicode normalizations? And now I'm curious - can you use all three Å representations as different variables in Go? ... Ahh, here is it "The text is not canonicalized, so a single accented code point is distinct from the same character constructed from combining an accent and a letter; those are treated as two code points." This means your Go code COULD have problems distinguishing different representation of the same character, rather like the problems you saw with ISO-Latin-1 / UTF-8 / MacRoman . Perhaps you just haven't worked with a diversified enough team / environment in your Go code? ;) The fact that you think Unicode is a "string" of "characters" suggests that you don't know much about Unicode and mostly have experience using the simple Latin-like subset of Unicode. If that's the case, then you should have written "This all makes me very happy I work with Latin-like languages day to day" rather than "..I program in Go..."
to3m 14y ago
This depends very much on the sort of work one does, I think. If there are only strings made of characters, it can be difficult to form certain sequences of bytes. For example, (uint8_t[]){0x80,0x80} isn't valid UTF-8; one could create the string "\x80\x80", but this is actually four bytes in size. Not great for efficiency, no good for interoperability. To create every possible array of bytes, arrays of bytes must be a data type too. And these arrays of bytes must be able to turn into strings, and vice versa, because the data read from and written to sockets and files and streams and what-have-you is most naturally of array-of-byte type rather than string. Perhaps mandating UTF-8 as the string encoding, leaving alternative encodings up to the programmer, would be reasonable these days, I suppose - if a little rude. Another advantage to having arrays of bytes in addition to strings of chars is that arrays of bytes support efficient random access. (Javascript looks like an example of this playing itself out, from the little I've used it so far. Websockets has alternative text/binary frames, which appear to be a later addition, and there's this ArrayBuffer thing which I believe was prompted by WebGL's requirement for efficient arrays of unadorned bytes.)
- snogglethorpe 14y ago
  > Perhaps mandating UTF-8 as the string encoding, leaving alternative encodings up to the programmer, would be reasonable these days, I suppose - if a little rude. It really works quite well, especially "these days" (maybe not so much in the '90s), and has the—huge—advantage that it's very simple and lightweight, and avoids the very tricky issue of re-encoding (Guaranteed To Make Your Life Hell). The python3/java/microsoft/whatever solution is to add an enormously heavyweight and complex layer that attempts to interpret and re-encode the contents of strings, with often dubious benefits. My experience suggest that the latter method is in many cases a classic case of over-engineering, where for a wide swathe of applications, the downsides—complexity and cost—exceed the upsides. But because these languages (or systems, in the case of MS) require this at a fundamental level, all applications pay the costs, even those that don't gain any benefit. [Given the encoding jungle of the '80s/'90s, it's to some degree understandable that MS/Java went the way they did (though UTF-8 was around back then, and would have at least made a much better canonical representation). That python3 dove into that swamp is a bit more mystifying given the rather clear signs that the future will essentially be Unicode encoded with UTF-8...]
bazzargh 14y ago
We use properties file (whose encoding is defined) to contain our UTF-8 characters. Yup...but it's defined as ISO-8859-1, not UTF-8 ( http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load%28java.io.InputStream%29 http://docs.oracle.com/javase/6/docs/api/java/util/Propertie... ). You need to escape everything else. Also the escapes are only for BMP characters, so supplementary characters need multiple unicode escapes to represent a single 'character'. There's too many rough edges in interoperability. You do still need to know this stuff if you work outside of Latin-1.
dalke 14y ago
For everyone hoping for a reply or dialog, it appears based on history that martinced is a fly-by commenter who doesn't do follow-ups.
SiVal 14y ago
No, martinced, it doesn't work like that. Encodings are data formats, and with the Internet connecting everything to everything, you have data generated by every programming language ever written and every protocol ever used swirling around you all the time. What a delight it would be if all text data were UTF-8, but it's not. So, you have to take what you get as input, do whatever processing you do with your part in the system, and output what your receiver demands. In the general case, you can't control the whole system, regardless of what tools you choose to use. I specialize in software that deals with non-Western scripts (writing systems), and my life would be easier (and less profitable) if everyone used a Unicode encoding for everything (I don't care WHICH Unicode encoding), but the legacy stuff will be around for years. In the meantime, it doesn't matter whether you use Go or Java or Python 3, you'll still have to deal with the fact that others out there use Python 2 or C or a MacRoman Web app (yes, I had to deal with one) or SQLServer's broken "UTF-8" or whatever to generate encoding mess you have to deal with. The fact that your language does things well internally is a great feature but won't insulate you from trouble if you have to deal with the outside world, as most of us do.

wrl 14y ago

Do you realize all your questions are really low-level details that are entirely taken care of by default APIs in quite some languages? Somebody's got to write those APIs. d0mine's questions are valid.

dalke 14y ago

"low-level details that are entirely taken care of by default APIs in quite some languages" They are not "taken care of"; it's only that the sharp edges are smoothed down a bit. Here's a hard one. In Go, if you read a directory to get a list of filenames, do you get a list of (Unicode) strings, or a list of byte sequences? Does it depend on the filesystem? What if the filesystem is nominally UTF-8 encoded, or UTF-16 encoded, but the underlying filesystem API does no validation, so that illegal UTF-8/-16 sequences are allowed? Do you have a way to get the filenames as bytes for that case? How do you show those strings in a GUI? Can someone enter a path name for a byte sequence which can't be represented in Unicode? The Python developers worked hard to get a reasonably good solution, but there are still hard details worry about. Can you point out the Go solution? Or is this one of the corner cases you've not had to deal with? Suppose you've read three strings, one with "Å" (LATIN CAPITAL LETTER A WITH RING ABOVE), the second with "Å" (ANGSTROM SIGN), and the third using the code point sequence "U+0041 U+030A" (LATIN LETTER A and COMBINING RING ABOVE ("°")). Are these the same strings? Is there a way to normalize the Unicode string to make them the same? A toolkit can implement the normalizations, but can't know when to do them. In your code, have you personally dealt with Unicode normalizations? And now I'm curious - can you use all three Å representations as different variables in Go? ... Ahh, here is it "The text is not canonicalized, so a single accented code point is distinct from the same character constructed from combining an accent and a letter; those are treated as two code points." This means your Go code COULD have problems distinguishing different representation of the same character, rather like the problems you saw with ISO-Latin-1 / UTF-8 / MacRoman . Perhaps you just haven't worked with a diversified enough team / environment in your Go code? ;) The fact that you think Unicode is a "string" of "characters" suggests that you don't know much about Unicode and mostly have experience using the simple Latin-like subset of Unicode. If that's the case, then you should have written "This all makes me very happy I work with Latin-like languages day to day" rather than "..I program in Go..."

to3m 14y ago

This depends very much on the sort of work one does, I think. If there are only strings made of characters, it can be difficult to form certain sequences of bytes. For example, (uint8_t[]){0x80,0x80} isn't valid UTF-8; one could create the string "\x80\x80", but this is actually four bytes in size. Not great for efficiency, no good for interoperability. To create every possible array of bytes, arrays of bytes must be a data type too. And these arrays of bytes must be able to turn into strings, and vice versa, because the data read from and written to sockets and files and streams and what-have-you is most naturally of array-of-byte type rather than string. Perhaps mandating UTF-8 as the string encoding, leaving alternative encodings up to the programmer, would be reasonable these days, I suppose - if a little rude. Another advantage to having arrays of bytes in addition to strings of chars is that arrays of bytes support efficient random access. (Javascript looks like an example of this playing itself out, from the little I've used it so far. Websockets has alternative text/binary frames, which appear to be a later addition, and there's this ArrayBuffer thing which I believe was prompted by WebGL's requirement for efficient arrays of unadorned bytes.)

snogglethorpe 14y ago
> Perhaps mandating UTF-8 as the string encoding, leaving alternative encodings up to the programmer, would be reasonable these days, I suppose - if a little rude. It really works quite well, especially "these days" (maybe not so much in the '90s), and has the—huge—advantage that it's very simple and lightweight, and avoids the very tricky issue of re-encoding (Guaranteed To Make Your Life Hell). The python3/java/microsoft/whatever solution is to add an enormously heavyweight and complex layer that attempts to interpret and re-encode the contents of strings, with often dubious benefits. My experience suggest that the latter method is in many cases a classic case of over-engineering, where for a wide swathe of applications, the downsides—complexity and cost—exceed the upsides. But because these languages (or systems, in the case of MS) require this at a fundamental level, all applications pay the costs, even those that don't gain any benefit. [Given the encoding jungle of the '80s/'90s, it's to some degree understandable that MS/Java went the way they did (though UTF-8 was around back then, and would have at least made a much better canonical representation). That python3 dove into that swamp is a bit more mystifying given the rather clear signs that the future will essentially be Unicode encoded with UTF-8...]

bazzargh 14y ago

We use properties file (whose encoding is defined) to contain our UTF-8 characters. Yup...but it's defined as ISO-8859-1, not UTF-8 ( http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load%28java.io.InputStream%29 http://docs.oracle.com/javase/6/docs/api/java/util/Propertie... ). You need to escape everything else. Also the escapes are only for BMP characters, so supplementary characters need multiple unicode escapes to represent a single 'character'. There's too many rough edges in interoperability. You do still need to know this stuff if you work outside of Latin-1.

dalke 14y ago

For everyone hoping for a reply or dialog, it appears based on history that martinced is a fly-by commenter who doesn't do follow-ups.

SiVal 14y ago

No, martinced, it doesn't work like that. Encodings are data formats, and with the Internet connecting everything to everything, you have data generated by every programming language ever written and every protocol ever used swirling around you all the time. What a delight it would be if all text data were UTF-8, but it's not. So, you have to take what you get as input, do whatever processing you do with your part in the system, and output what your receiver demands. In the general case, you can't control the whole system, regardless of what tools you choose to use. I specialize in software that deals with non-Western scripts (writing systems), and my life would be easier (and less profitable) if everyone used a Unicode encoding for everything (I don't care WHICH Unicode encoding), but the legacy stuff will be around for years. In the meantime, it doesn't matter whether you use Go or Java or Python 3, you'll still have to deal with the fact that others out there use Python 2 or C or a MacRoman Web app (yes, I had to deal with one) or SQLServer's broken "UTF-8" or whatever to generate encoding mess you have to deal with. The fact that your language does things well internally is a great feature but won't insulate you from trouble if you have to deal with the outside world, as most of us do.