4 ms·

This all makes me very happy I program in Go day to day.

deleted 14y ago
[deleted]
d0mine 14y ago
Could you elaborate why the article is not applicable to Go? Does Go allows to work with binary data (bytes)? Does it allow to work with text (Unicode)? Does it allows to convert one to another using a character encoding? Does it allow to accept data/text input from stdin? Does it allow to output data/text to stdout? Does it allow to read/write data/text from/to file/subprocess? Does it allow to specify filenames as Unicode/bytes? Does it allow to use regexes on bytes/Unicode? ... Do you need to write every single operation twice: once for bytes, once for Unicode?
- martinced 14y ago
  tl;dr It's nearly 2013, if you're still stuck at the byte/Unicode level you're doing it wrong: nowadays people are using programming languages which have abstraction for strings and characters. I can't answer about parent about Go specifically but I may have the beginning of an answer. Do you realize all your questions are really low-level details that are entirely taken care of by default APIs in quite some languages? The whole byte/Unicode you keep insisting on is a strawman: at the right level of abstraction you're dealing with neither. You're dealing with "strings" made of "characters". I don't remember when was the last time I had to manually read bytes / data and "convert" them using whatever character encoding. You're questions just feel weird. As if the early nineties called and wanted their byte/UTF-8 encoders developers back or something. I realize this may be a very true concern when working with languages that didn't deal correctly with the string/characters abstraction and which are still stuck in the byte/Unicode(byte or bytes) mindset but you have to realize that several languages and techniques allow to simply dodge these matters altogether. For example, Java doesn't specify the encoding of Java source file: great, we mandate (and enforce with scripts/hooks) that all .java file contain only ASCII characters. We use properties file (whose encoding is defined) to contain our UTF-8 characters. Wanna transmit text? Well, use a format which allows to specify the encoding of your text. It's really that simple and, on the other end of the wire, it's going to become automagically again "strings" and "characters". Another "detail": Go is one the only language to specify that source file have to be encoded in UTF-8. This is a major win too. (if you've never seen ISO-Latin-1 / UTF-8 / MacRoman / etc. mismatch in source file then you've never worked in a diversified enough team / environment ; )
  - wrl 14y ago
    Do you realize all your questions are really low-level details that are entirely taken care of by default APIs in quite some languages? Somebody's got to write those APIs. d0mine's questions are valid.
  - dalke 14y ago
    "low-level details that are entirely taken care of by default APIs in quite some languages" They are not "taken care of"; it's only that the sharp edges are smoothed down a bit. Here's a hard one. In Go, if you read a directory to get a list of filenames, do you get a list of (Unicode) strings, or a list of byte sequences? Does it depend on the filesystem? What if the filesystem is nominally UTF-8 encoded, or UTF-16 encoded, but the underlying filesystem API does no validation, so that illegal UTF-8/-16 sequences are allowed? Do you have a way to get the filenames as bytes for that case? How do you show those strings in a GUI? Can someone enter a path name for a byte sequence which can't be represented in Unicode? The Python developers worked hard to get a reasonably good solution, but there are still hard details worry about. Can you point out the Go solution? Or is this one of the corner cases you've not had to deal with? Suppose you've read three strings, one with "Å" (LATIN CAPITAL LETTER A WITH RING ABOVE), the second with "Å" (ANGSTROM SIGN), and the third using the code point sequence "U+0041 U+030A" (LATIN LETTER A and COMBINING RING ABOVE ("°")). Are these the same strings? Is there a way to normalize the Unicode string to make them the same? A toolkit can implement the normalizations, but can't know when to do them. In your code, have you personally dealt with Unicode normalizations? And now I'm curious - can you use all three Å representations as different variables in Go? ... Ahh, here is it "The text is not canonicalized, so a single accented code point is distinct from the same character constructed from combining an accent and a letter; those are treated as two code points." This means your Go code COULD have problems distinguishing different representation of the same character, rather like the problems you saw with ISO-Latin-1 / UTF-8 / MacRoman . Perhaps you just haven't worked with a diversified enough team / environment in your Go code? ;) The fact that you think Unicode is a "string" of "characters" suggests that you don't know much about Unicode and mostly have experience using the simple Latin-like subset of Unicode. If that's the case, then you should have written "This all makes me very happy I work with Latin-like languages day to day" rather than "..I program in Go..."
  - 14y ago
- voidlogic 14y ago
  I think the way go handles unicode is very sane- of course two of the three primary creators of Go, Ken Thompson and Rob Pike, are the same two co-creator's of UTF-8, so that is probably unsurprising. Take a look at the Go spec: https://golang.org/ref/spec https://golang.org/ref/spec And std libs: https://golang.org/pkg/ https://golang.org/pkg/
charliesome 14y ago
No matter what language you program in, you still need to have an understanding of character encodings if you ever intend to write software that interfaces with the real world.
gcr 14y ago
Racket does this too. In Racket, "this is a string, which is a sequence of characters" #"this is a bytestring, a sequence of numbers between 0 and 255" You can convert back and forth with string->bytes/utf-8 or bytes->string/utf-8, substituting utf-8 for whichever character encoding you want. You can use functions like string-ref, which returns a unicode character and bytes-ref, which returns an integer. Functions like string-normalize-nfc (or -nfd or -nfkc or etc) will normalize combining characters, and you can capitalize uppercase/lowercase strings according to locale, etc. Read from an input port with read-bytes and read-string, and writing to an output port works similarly. Racket does does a great job keeping these things as explicit as they should be.

deleted 14y ago

[deleted]

Could you elaborate why the article is not applicable to Go? Does Go allows to work with binary data (bytes)? Does it allow to work with text (Unicode)? Does it allows to convert one to another using a character encoding? Does it allow to accept data/text input from stdin? Does it allow to output data/text to stdout? Does it allow to read/write data/text from/to file/subprocess? Does it allow to specify filenames as Unicode/bytes? Does it allow to use regexes on bytes/Unicode? ... Do you need to write every single operation twice: once for bytes, once for Unicode?

martinced 14y ago
tl;dr It's nearly 2013, if you're still stuck at the byte/Unicode level you're doing it wrong: nowadays people are using programming languages which have abstraction for strings and characters. I can't answer about parent about Go specifically but I may have the beginning of an answer. Do you realize all your questions are really low-level details that are entirely taken care of by default APIs in quite some languages? The whole byte/Unicode you keep insisting on is a strawman: at the right level of abstraction you're dealing with neither. You're dealing with "strings" made of "characters". I don't remember when was the last time I had to manually read bytes / data and "convert" them using whatever character encoding. You're questions just feel weird. As if the early nineties called and wanted their byte/UTF-8 encoders developers back or something. I realize this may be a very true concern when working with languages that didn't deal correctly with the string/characters abstraction and which are still stuck in the byte/Unicode(byte or bytes) mindset but you have to realize that several languages and techniques allow to simply dodge these matters altogether. For example, Java doesn't specify the encoding of Java source file: great, we mandate (and enforce with scripts/hooks) that all .java file contain only ASCII characters. We use properties file (whose encoding is defined) to contain our UTF-8 characters. Wanna transmit text? Well, use a format which allows to specify the encoding of your text. It's really that simple and, on the other end of the wire, it's going to become automagically again "strings" and "characters". Another "detail": Go is one the only language to specify that source file have to be encoded in UTF-8. This is a major win too. (if you've never seen ISO-Latin-1 / UTF-8 / MacRoman / etc. mismatch in source file then you've never worked in a diversified enough team / environment ; )
- wrl 14y ago
  Do you realize all your questions are really low-level details that are entirely taken care of by default APIs in quite some languages? Somebody's got to write those APIs. d0mine's questions are valid.
- dalke 14y ago
  "low-level details that are entirely taken care of by default APIs in quite some languages" They are not "taken care of"; it's only that the sharp edges are smoothed down a bit. Here's a hard one. In Go, if you read a directory to get a list of filenames, do you get a list of (Unicode) strings, or a list of byte sequences? Does it depend on the filesystem? What if the filesystem is nominally UTF-8 encoded, or UTF-16 encoded, but the underlying filesystem API does no validation, so that illegal UTF-8/-16 sequences are allowed? Do you have a way to get the filenames as bytes for that case? How do you show those strings in a GUI? Can someone enter a path name for a byte sequence which can't be represented in Unicode? The Python developers worked hard to get a reasonably good solution, but there are still hard details worry about. Can you point out the Go solution? Or is this one of the corner cases you've not had to deal with? Suppose you've read three strings, one with "Å" (LATIN CAPITAL LETTER A WITH RING ABOVE), the second with "Å" (ANGSTROM SIGN), and the third using the code point sequence "U+0041 U+030A" (LATIN LETTER A and COMBINING RING ABOVE ("°")). Are these the same strings? Is there a way to normalize the Unicode string to make them the same? A toolkit can implement the normalizations, but can't know when to do them. In your code, have you personally dealt with Unicode normalizations? And now I'm curious - can you use all three Å representations as different variables in Go? ... Ahh, here is it "The text is not canonicalized, so a single accented code point is distinct from the same character constructed from combining an accent and a letter; those are treated as two code points." This means your Go code COULD have problems distinguishing different representation of the same character, rather like the problems you saw with ISO-Latin-1 / UTF-8 / MacRoman . Perhaps you just haven't worked with a diversified enough team / environment in your Go code? ;) The fact that you think Unicode is a "string" of "characters" suggests that you don't know much about Unicode and mostly have experience using the simple Latin-like subset of Unicode. If that's the case, then you should have written "This all makes me very happy I work with Latin-like languages day to day" rather than "..I program in Go..."
- 14y ago
voidlogic 14y ago
I think the way go handles unicode is very sane- of course two of the three primary creators of Go, Ken Thompson and Rob Pike, are the same two co-creator's of UTF-8, so that is probably unsurprising. Take a look at the Go spec: https://golang.org/ref/spec https://golang.org/ref/spec And std libs: https://golang.org/pkg/ https://golang.org/pkg/

charliesome 14y ago

No matter what language you program in, you still need to have an understanding of character encodings if you ever intend to write software that interfaces with the real world.

gcr 14y ago

Racket does this too. In Racket, "this is a string, which is a sequence of characters" #"this is a bytestring, a sequence of numbers between 0 and 255" You can convert back and forth with string->bytes/utf-8 or bytes->string/utf-8, substituting utf-8 for whichever character encoding you want. You can use functions like string-ref, which returns a unicode character and bytes-ref, which returns an integer. Functions like string-normalize-nfc (or -nfd or -nfkc or etc) will normalize combining characters, and you can capitalize uppercase/lowercase strings according to locale, etc. Read from an input port with read-bytes and read-string, and writing to an output port works similarly. Racket does does a great job keeping these things as explicit as they should be.