4 ms·

I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodin

I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to allow multiple implementations for performance reasons (e.g. because reading UTF-8 from file, converting it to UTF-16, processing it, converting it back to UTF-8 to send it over the network is inefficient) then that's fine by me. I'll still want to tell it what encoding that byte stream from which I'm writing is, and what that stream I'm writing to expects. If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. That was a bad idea in C and with the complexity that unicode adds, this is an even worse idea nowadays.

DougBTX 17y ago
If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. My understanding was just the opposite: now that strings are associated with encodings, he can no longer assume that a1 + a2 results in a string with the same encoding as a1 and a2, since a1 and a2 can have different encodings.
- pmjordan 17y ago
  The implication is that the programmer must know what is used internally, though, because things will go horribly wrong otherwise. That's a sign of a very leaky abstraction - if you need to know the format and behaviour of the underlying byte stream, you'd probably be better off using the byte stream directly, the abstraction is just introducing uncertainty where it should be isolating from such concerns.
  - halostatue 17y ago
    It's also supposed to be an incorrect implication for Ruby most of the time. It should be possible to do a single encoding trivially (e.g., if all of your data is shift_jis or UTF-8), but rather than trying to (badly, and usually unsuccessfully) hide the encoding difficulties from the programmer if you've got mixed encodings, Ruby has chosen to be a little more up front about encodings. The reality may be a bit differently, but I recall seeing an email message from Matz on ruby-core last year suggesting that it was supposed to be trivially easy to work with one encoding (specifically mentioning UTF-8, but implying others).
tolmasky 17y ago
I also thought the exact same thing, but I know that encoding stuff is so complex that I may be missing something. For what its worth, I believe Cocoa does things this way: a string is an abstract object of unicode code points, when its time to read or write you must choose an encoding.
- pmjordan 17y ago
  The Java and .NET infrastructures make that choice for you too. I guess I wasn't there when Java first got started, it's possible that there was a backlash[1] against this decision by the people who are using encodings that aren't subsets of unicode. I don't get to see many "enterprise" systems, but I get the impression those applications where it's important to use some kind of not-unicode are sufficiently rare that they're best served by specialty libraries. You can't make everyone happy with one solution, but if a simple solution works for 99%, then I'd call that a roaring success. [1] a backlash other than the OMG-2-bytes-per-ASCII-character hysteria, which is irrelevant for these purposes, they could just as well have chosen UTF-8

DougBTX 17y ago

If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. My understanding was just the opposite: now that strings are associated with encodings, he can no longer assume that a1 + a2 results in a string with the same encoding as a1 and a2, since a1 and a2 can have different encodings.

pmjordan 17y ago
The implication is that the programmer must know what is used internally, though, because things will go horribly wrong otherwise. That's a sign of a very leaky abstraction - if you need to know the format and behaviour of the underlying byte stream, you'd probably be better off using the byte stream directly, the abstraction is just introducing uncertainty where it should be isolating from such concerns.
- halostatue 17y ago
  It's also supposed to be an incorrect implication for Ruby most of the time. It should be possible to do a single encoding trivially (e.g., if all of your data is shift_jis or UTF-8), but rather than trying to (badly, and usually unsuccessfully) hide the encoding difficulties from the programmer if you've got mixed encodings, Ruby has chosen to be a little more up front about encodings. The reality may be a bit differently, but I recall seeing an email message from Matz on ruby-core last year suggesting that it was supposed to be trivially easy to work with one encoding (specifically mentioning UTF-8, but implying others).

tolmasky 17y ago

I also thought the exact same thing, but I know that encoding stuff is so complex that I may be missing something. For what its worth, I believe Cocoa does things this way: a string is an abstract object of unicode code points, when its time to read or write you must choose an encoding.

pmjordan 17y ago
The Java and .NET infrastructures make that choice for you too. I guess I wasn't there when Java first got started, it's possible that there was a backlash[1] against this decision by the people who are using encodings that aren't subsets of unicode. I don't get to see many "enterprise" systems, but I get the impression those applications where it's important to use some kind of not-unicode are sufficiently rare that they're best served by specialty libraries. You can't make everyone happy with one solution, but if a simple solution works for 99%, then I'd call that a roaring success. [1] a backlash other than the OMG-2-bytes-per-ASCII-character hysteria, which is irrelevant for these purposes, they could just as well have chosen UTF-8