3 ms·

I think my perspective on the "standard unix" approach of "it's bytes with ASCII or whatever" is different, because my native language uses the "whatever" part.

by pornel 4y ago

I think my perspective on the "standard unix" approach of "it's bytes with ASCII or whatever" is different, because my native language uses the "whatever" part. In pre-Unicode times I couldn't reliably grep for "żółw", or consume a CSV file with it, or share a file with anyone on a different platform, because it may have been in ISO-8859-2 encoding, or Windows-1252 which was subtly incompatible, or in MacCE, or in one of three AmigaOS encodings. I've had printers that had "it's just bytes" encoding, so I had to convert files to a character encoding used by a particular font file, which varied between font sources and printer brands. I've had to deal with files from a popular MS-DOS editor which invented its own character encoding. I've had to convert files to encoding of "Reverend John Pikul", because he was a prolific author of printer drivers, and created his own encoding. The "strings are just bytes" systems were a shitshow for me. It's a broken approach that is impossible to fix. Nobody cares what those "garbage" bytes actually mean. So when strings are UTF-8-or-garbage, these non-UTF-8 bytes are someone's "żółw" becoming "???w", or a `String + bstr` is "¿ó³w".

burntsushi 4y ago
Yeah I think that's a different time and a different place. At least the GNU Unix tooling can all be configured in the UTF-8 locale, which I think is pretty standard these days. So I guess from my view, Unix tooling isn't quite "it's all just bytes" these days, but rather, "everything is UTF-8, except for some things and we still want to handle those things in some sensible way." I don't think the byte-string-mostly-UTF-8 approach leads to the sadness you describe. What you describe I think was a result of a very different time before the ubiquity of UTF-8. I might have a very different opinion if we were transported back in time, absolutely. Go has the same byte-string-mostly-UTF-8 approach as bstr (and indeed, bstr was inspired by it), and I don't think they're having the kinds of problems you describe. Go strings might not be valid UTF-8, but the language itself pushes you toward UTF-8 and they otherwise have very good Unicode support throughout their standard library and supporting libraries.

by pornel 4y ago

burntsushi 4y ago

Yeah I think that's a different time and a different place. At least the GNU Unix tooling can all be configured in the UTF-8 locale, which I think is pretty standard these days. So I guess from my view, Unix tooling isn't quite "it's all just bytes" these days, but rather, "everything is UTF-8, except for some things and we still want to handle those things in some sensible way." I don't think the byte-string-mostly-UTF-8 approach leads to the sadness you describe. What you describe I think was a result of a very different time before the ubiquity of UTF-8. I might have a very different opinion if we were transported back in time, absolutely. Go has the same byte-string-mostly-UTF-8 approach as bstr (and indeed, bstr was inspired by it), and I don't think they're having the kinds of problems you describe. Go strings might not be valid UTF-8, but the language itself pushes you toward UTF-8 and they otherwise have very good Unicode support throughout their standard library and supporting libraries.