4 ms·

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

halostatue 17y ago
The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-MapUTF.html http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.
- pmjordan 17y ago
  That page says there are duplicates in CP-932 because of different vendor variants. If those characters are otherwise entirely identical, calling it a single encoding seems wrong. Wouldn't you just have a CP-932-IBM and a CP-932-NEC encoding?
  - halostatue 17y ago
    Because someone unified two similar encodings into CP-932/ShiftJIS. That someone who unified them wasn't IBM or NEC (both of whom made competing systems and had made different encoding choices, but whose choices were mostly compatible). Rules for dealing with legacy encodings: 1. They make no sense. 2. If you think they make sense, remember that you weren't there so refer to rule 1.
- 3pt14159 17y ago
  General question: Why didn't the UTF-8 boys and girls make it safe in the first place? This doesn't sound like rocket science. "This character maps to that character, this character to that one." I don't understand how we have unicode snowmen, but we can't safely round trip characters.
  - flogic 17y ago
    There isn't a good idea that a standards body hasn't fucked up. Maybe it's because these problems are harder than they seem. Or maybe they're just plain screwed up. Or maybe it's Adobe.
  - halostatue 17y ago
    Some of it has to do with Han Unification (http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification). Mostly, though, it's because some of these characters are overloaded. If you've got a Windows system, go into the DOS window and type "chcp 932" (you may need the Japanese language files installed). When you type '\', you'll get '¥' (making "C:\Program Files\" look like "C:¥Program Files¥"). In the systems where what become CP932 were first used, the backslash wasn't necessary in Japanese, so that character point was used to encode the yen symbol. Other systems used the backslash, so it was encoded as a different point. When JIS unified the existing Japanese code pages, it couldn't very well go back in time to change all that old data, so it merged the two encodings on many things. So, there's only one Unicode codepoint for the yen glyph ¥, but in this one encoding there's two different characters for it. This is the most blatant example of a problem with Unicode transcoding, but as far as I know, it's not the only one. See http://www.mail-archive.com/linux-utf8@nl.linux.org/msg02337.html http://www.mail-archive.com/linux-utf8@nl.linux.org/msg02337... for what could be done, but probably won't.
- alextgordon 17y ago
  Are these encodings common enough that they're worth supporting in the basic String class? Surely another class could be provided to handle these edge cases, they don't have to be natively handled. There's no need for any data loss to occur - the String class would merely not support converting from non round-trippable encodings.
  - halostatue 17y ago
    If they're not natively handled, you can't regex them. If they're not natively handled, you can't convert any numbers that might be in them to numeric values. Yes, they're common enough (especially in Japan) and encodings have to be baked deeply in if you really want to use everything that a Rubyist expects to be able to use.
- bhiggins 17y ago
  Practically speaking, what data are you talking about? Which characters don't map? The mapping here looks pretty complete, except for "DBCS LEAD BYTE". Is that a problem? Other non-round trip mappings look like it's because different vendors have different meanings for that character. That's not Unicode's fault. http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOW...
  - halostatue 17y ago
    If there's ~398 byte combinations that can be translated only one way, but not the other, you're still losing data. (The big one that always gets pointed out? '\' and '¥'.)
    - earthboundkid 17y ago
      "(The big one that always gets pointed out? '\' and '¥’.)” The point there though is that Japanese DON’T want \ and ¥ to map properly. They want \ and ¥ to be considered the same. So Unicode isn’t losing information, it’s forcing a distinction the Japanese don’t want to be able to make.
deleted 17y ago
[deleted]
donaldc 17y ago
Is there a technical reason why Ruby doesn't do this? More likely it's political. Matz (the creator of Ruby), and many of its early contributors, are said not to like Unicode.
- halostatue 17y ago
  That doesn't jive with what I know of Matz. Japan (and much of Asia) adopted Unicode much slower because both UTF-8 and UTF-16 are less efficient than many of the native encodings for representing much of what they need; there was definitely some cultural tone-deafness in the Unicode community when it did Han unification (which may be a good idea, but could have been handled better). Matz (and the people he works with who use Ruby to get their jobs done) needs access to data that's Not Unicode. Painfully Not Unicode as in it doesn't necessarily round-trip.
pmjordan 17y ago
I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to allow multiple implementations for performance reasons (e.g. because reading UTF-8 from file, converting it to UTF-16, processing it, converting it back to UTF-8 to send it over the network is inefficient) then that's fine by me. I'll still want to tell it what encoding that byte stream from which I'm writing is, and what that stream I'm writing to expects. If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. That was a bad idea in C and with the complexity that unicode adds, this is an even worse idea nowadays.
- DougBTX 17y ago
  If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. My understanding was just the opposite: now that strings are associated with encodings, he can no longer assume that a1 + a2 results in a string with the same encoding as a1 and a2, since a1 and a2 can have different encodings.
  - pmjordan 17y ago
    The implication is that the programmer must know what is used internally, though, because things will go horribly wrong otherwise. That's a sign of a very leaky abstraction - if you need to know the format and behaviour of the underlying byte stream, you'd probably be better off using the byte stream directly, the abstraction is just introducing uncertainty where it should be isolating from such concerns.
    - halostatue 17y ago
      It's also supposed to be an incorrect implication for Ruby most of the time. It should be possible to do a single encoding trivially (e.g., if all of your data is shift_jis or UTF-8), but rather than trying to (badly, and usually unsuccessfully) hide the encoding difficulties from the programmer if you've got mixed encodings, Ruby has chosen to be a little more up front about encodings. The reality may be a bit differently, but I recall seeing an email message from Matz on ruby-core last year suggesting that it was supposed to be trivially easy to work with one encoding (specifically mentioning UTF-8, but implying others).
wedesoft 17y ago
The performance is better if you use an encoding with a fixed byte-length character-set.
- ochuuzu1 17y ago
  No it isn't.
nradov 17y ago
Java stores all strings in UTF-16. It probably seemed like a good idea at the time but has led to problems later. There is now a huge body of broken code out there which assumes that each 16-bit char value represents a complete and separate Unicode character.
- halostatue 17y ago
  Um. Wasn't Java's original choice UCS-2 (same as NTFS originally)? UTF-16 has surrogate characters and the length will be one or two 16-bit values depending on whether it's a surrogate. UCS-2 was a fixed 16-bit character size, long since deemed a mistake.
  - nradov 17y ago
    You're right. Java originally used UCS-2 and later switched to UTF-16. http://java.sun.com/docs/books/jls/first_edition/html/3.doc.html#95413 http://java.sun.com/docs/books/jls/first_edition/html/3.doc.... The problem is that dealing with surrogate pairs is so awkward that most developers don't even bother.
wooster 17y ago
> Is there a technical reason why Ruby doesn't do this? Han Unification. Gory details here: http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification
india 17y ago
And performance... My non-utf8 1 GB file whose number of lines I need to count thanks Ruby 1.9 for not uselessly and forcefully making an in memory utf8 copy of it.

halostatue 17y ago

The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-MapUTF.html http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.") Loss of text data is bad.

pmjordan 17y ago
That page says there are duplicates in CP-932 because of different vendor variants. If those characters are otherwise entirely identical, calling it a single encoding seems wrong. Wouldn't you just have a CP-932-IBM and a CP-932-NEC encoding?
- halostatue 17y ago
  Because someone unified two similar encodings into CP-932/ShiftJIS. That someone who unified them wasn't IBM or NEC (both of whom made competing systems and had made different encoding choices, but whose choices were mostly compatible). Rules for dealing with legacy encodings: 1. They make no sense. 2. If you think they make sense, remember that you weren't there so refer to rule 1.
3pt14159 17y ago
General question: Why didn't the UTF-8 boys and girls make it safe in the first place? This doesn't sound like rocket science. "This character maps to that character, this character to that one." I don't understand how we have unicode snowmen, but we can't safely round trip characters.
- flogic 17y ago
  There isn't a good idea that a standards body hasn't fucked up. Maybe it's because these problems are harder than they seem. Or maybe they're just plain screwed up. Or maybe it's Adobe.
- halostatue 17y ago
  Some of it has to do with Han Unification (http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification). Mostly, though, it's because some of these characters are overloaded. If you've got a Windows system, go into the DOS window and type "chcp 932" (you may need the Japanese language files installed). When you type '\', you'll get '¥' (making "C:\Program Files\" look like "C:¥Program Files¥"). In the systems where what become CP932 were first used, the backslash wasn't necessary in Japanese, so that character point was used to encode the yen symbol. Other systems used the backslash, so it was encoded as a different point. When JIS unified the existing Japanese code pages, it couldn't very well go back in time to change all that old data, so it merged the two encodings on many things. So, there's only one Unicode codepoint for the yen glyph ¥, but in this one encoding there's two different characters for it. This is the most blatant example of a problem with Unicode transcoding, but as far as I know, it's not the only one. See http://www.mail-archive.com/linux-utf8@nl.linux.org/msg02337.html http://www.mail-archive.com/linux-utf8@nl.linux.org/msg02337... for what could be done, but probably won't.
alextgordon 17y ago
Are these encodings common enough that they're worth supporting in the basic String class? Surely another class could be provided to handle these edge cases, they don't have to be natively handled. There's no need for any data loss to occur - the String class would merely not support converting from non round-trippable encodings.
- halostatue 17y ago
  If they're not natively handled, you can't regex them. If they're not natively handled, you can't convert any numbers that might be in them to numeric values. Yes, they're common enough (especially in Japan) and encodings have to be baked deeply in if you really want to use everything that a Rubyist expects to be able to use.
bhiggins 17y ago
Practically speaking, what data are you talking about? Which characters don't map? The mapping here looks pretty complete, except for "DBCS LEAD BYTE". Is that a problem? Other non-round trip mappings look like it's because different vendors have different meanings for that character. That's not Unicode's fault. http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOW...
- halostatue 17y ago
  If there's ~398 byte combinations that can be translated only one way, but not the other, you're still losing data. (The big one that always gets pointed out? '\' and '¥'.)
  - earthboundkid 17y ago
    "(The big one that always gets pointed out? '\' and '¥’.)” The point there though is that Japanese DON’T want \ and ¥ to map properly. They want \ and ¥ to be considered the same. So Unicode isn’t losing information, it’s forcing a distinction the Japanese don’t want to be able to make.

deleted 17y ago

[deleted]

donaldc 17y ago

Is there a technical reason why Ruby doesn't do this? More likely it's political. Matz (the creator of Ruby), and many of its early contributors, are said not to like Unicode.

halostatue 17y ago
That doesn't jive with what I know of Matz. Japan (and much of Asia) adopted Unicode much slower because both UTF-8 and UTF-16 are less efficient than many of the native encodings for representing much of what they need; there was definitely some cultural tone-deafness in the Unicode community when it did Han unification (which may be a good idea, but could have been handled better). Matz (and the people he works with who use Ruby to get their jobs done) needs access to data that's Not Unicode. Painfully Not Unicode as in it doesn't necessarily round-trip.

pmjordan 17y ago

I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to allow multiple implementations for performance reasons (e.g. because reading UTF-8 from file, converting it to UTF-16, processing it, converting it back to UTF-8 to send it over the network is inefficient) then that's fine by me. I'll still want to tell it what encoding that byte stream from which I'm writing is, and what that stream I'm writing to expects. If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. That was a bad idea in C and with the complexity that unicode adds, this is an even worse idea nowadays.

DougBTX 17y ago
If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. My understanding was just the opposite: now that strings are associated with encodings, he can no longer assume that a1 + a2 results in a string with the same encoding as a1 and a2, since a1 and a2 can have different encodings.
- pmjordan 17y ago
  The implication is that the programmer must know what is used internally, though, because things will go horribly wrong otherwise. That's a sign of a very leaky abstraction - if you need to know the format and behaviour of the underlying byte stream, you'd probably be better off using the byte stream directly, the abstraction is just introducing uncertainty where it should be isolating from such concerns.
  - halostatue 17y ago
    It's also supposed to be an incorrect implication for Ruby most of the time. It should be possible to do a single encoding trivially (e.g., if all of your data is shift_jis or UTF-8), but rather than trying to (badly, and usually unsuccessfully) hide the encoding difficulties from the programmer if you've got mixed encodings, Ruby has chosen to be a little more up front about encodings. The reality may be a bit differently, but I recall seeing an email message from Matz on ruby-core last year suggesting that it was supposed to be trivially easy to work with one encoding (specifically mentioning UTF-8, but implying others).

wedesoft 17y ago

The performance is better if you use an encoding with a fixed byte-length character-set.

ochuuzu1 17y ago
No it isn't.

nradov 17y ago

Java stores all strings in UTF-16. It probably seemed like a good idea at the time but has led to problems later. There is now a huge body of broken code out there which assumes that each 16-bit char value represents a complete and separate Unicode character.

halostatue 17y ago
Um. Wasn't Java's original choice UCS-2 (same as NTFS originally)? UTF-16 has surrogate characters and the length will be one or two 16-bit values depending on whether it's a surrogate. UCS-2 was a fixed 16-bit character size, long since deemed a mistake.
- nradov 17y ago
  You're right. Java originally used UCS-2 and later switched to UTF-16. http://java.sun.com/docs/books/jls/first_edition/html/3.doc.html#95413 http://java.sun.com/docs/books/jls/first_edition/html/3.doc.... The problem is that dealing with surrogate pairs is so awkward that most developers don't even bother.

wooster 17y ago

> Is there a technical reason why Ruby doesn't do this? Han Unification. Gory details here: http://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification

india 17y ago

And performance... My non-utf8 1 GB file whose number of lines I need to count thanks Ruby 1.9 for not uselessly and forcefully making an in memory utf8 copy of it.