4 ms·
If that's true, it's surprising then that Ruby handles encodings so badly. String encoding cause some pretty obtuse errors in non-obvious spots. irb(main)
by paulddraper 9y ago
If that's true, it's surprising then that Ruby handles encodings so badly.
String encoding cause some pretty obtuse errors in non-obvious spots.
irb(main):001:0> str = "\xAA"
=> "\xAA"
irb(main):002:0> str.split('')
ArgumentError: invalid byte sequence in UTF-8
from (irb):2:in `split'
from (irb):2
from /usr/bin/irb:12:in `<main>'
WTF?
- bhaak 9y agoFirst, you're looking at a feature of modern Ruby that got added in 1.9. Before that, Ruby only supported a single encoding. If you're interesting in the details, look up how the $KCODE global variable used to affect in Ruby <= 1.8. Second, what do you expect if you have an invalid byte sequence in your UTF-8 string? str.valid_encoding? would have told you that the string contains invalid characters. String#encode gives you fine grained control over how to handle invalid characters in your strings. Ruby lets you set the encoding by string. > str = "\xa4" > puts str � > puts str.force_encoding("iso-8859-1").encode("utf-8") ¤ > puts str.force_encoding("iso-8859-16").encode("utf-8") €
- lloeki 9y agoYou just made the point of the GP, Ruby does handle encoding, but it does so badly: > str = "\xa4" "" creates a UTF8 string, that one happens to contain an invalid char sequence right off the bat > str.split('') ArgumentError: invalid byte sequence in UTF-8 ... which can only be detected at the worst possible time: when processed at runtime > puts str � ... and not even always, as sometimes ruby decides to handle strings as just bytes > str.force_encoding("iso-8859-1").encode("utf-8") ... and leaves the responsibility of whether they're bytes or strings, (and which encoding of) solely on the programmer. "force_encoding" is a thing that just should not exist on a string. Python 3 and Go do it the correct way: array of bytes are array of bytes and strings are {array of bytes + encoding}. Ruby could have done it too by introducing a new type or something but chose to do otherwise in the name of a theoretical backwards compatibility that doesn't exist in practice since every singe release of Ruby introduced backwards-incompatible changes anyway.
- spoiler 9y agoIt's documented that the default encoding in ruby is UTF-8. > "\xAA".force_encoding('binary').split('') # => ["\xAA"] If you want a binary string, you can do str = String.new('...', encoding: 'binary') NOTE: `'binary'` is an alias for `Encoding::ASCII_8BIT`
- masklinn 9y ago> It's documented that the default encoding in ruby is UTF-8. The complaint is not that the default string has no encoding, it's that proper encoding is not validated when creating the string, but only when processing it (and even then it might depend on the actual operation being applied). So given (1) a string object with (2) a proper encoding, it may still blow up in your face at any point unless you defensively check every string you get with `valid_encoding?`
- bhaak 9y agoAFAIK you are incorrect about Python 3. Python 3 strings are internally represented as ASCII, Latin-1(since 3.3), UCS-2, or UCS-4. So for an arbitrary encoding this statement would be incorrect. Ruby's model look much more like array of bytes + encoding to me, how else could I change the encoding for example of a latin-1 string into an invalid utf-8 string with force_encoding? Ruby holds off the validation of the string until it's really needed. Lazy validation doesn't strike me as odd for a dynamic language. I agree that String#force_encoding would be nice if it were not needed but in the real world you get all kinds of broken encoded data. String#encode most of the time is enough but as a last resort, String#force_encoding is there to use. I'm a bit surprised that you mention Go because if I look at https://blog.golang.org/strings https://blog.golang.org/strings "Similarly, unless [the literal string] contains UTF-8-breaking escapes like those from the previous section, a regular string literal will also always contain valid UTF-8." This is exactly the same behavior as in Ruby. I put the invalid byte sequence there deliberately and Ruby and apparently also Go don't have a problem with that.
- lloeki 9y agoRegarding Go, what matters is that you read/write []byte and transform it to string explicitly. > in the real world you get all kinds of broken encoded data > String#encode most of the time is enough but as a last resort, String#force_encoding is there to use. When your byte arrays and strings get passed around your code soon you don't have a way to validate which one is safe and which is not, which one is a byte array and which is a string. The thing is not about content, it's about communication and contracts. #force_encoding should not be available on a string, and #encode should not be available on a byte array. If you receive something that may be broken, it should be received into a byte array (b"" in python, []byte in Go), then "cross the gate" to be a string, and that moment is where you eventually perform sanitisation (u"" in python, string in Go). From then on a consumer of your string will be able to assume its content and its declared encoding match. > I'm a bit surprised that you mention Go [...] This is exactly the same behavior as in Ruby. Indeed, but you have two types to use and discriminate against, with assorted funcs corresponding to each level of abstraction. From the Go doc [0]: > It's important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes. I am definitely not concerned about what is in a string. I am concerned about distinguishing between "I just read that bunch of data from somewhere and I'm not yet sure what it is" vs "Ok, that data has been through some earlier process that took guarantees as to what format I expect it to be in". Different abstraction levels. Where Go gets it right is that io.Reader/Writer reads/writes []byte, not string. So you're explicitly acting on your own responsibility when you do: var b = []byte{0xA4} var s = string(b) foo.IExpectSomeSpecificEncoding(s) So, when to insert a sanity check becomes obvious, as well as relying on the static type system/method dispatch to check things around for you becomes incredibly useful. and foo.IExpectSomeSpecificEncoding can then use the string as an opaque container. [0]: https://blog.golang.org/strings https://blog.golang.org/strings
- paulddraper 9y ago> First, you're looking at a feature of modern Ruby that got added in 1.9. I thought 1.9 died quite some time ago. > what do you expect if you have an invalid byte sequence in your UTF-8 string? What I expected: An error when creating the invalid UTF-8 string literal. What I got: An when trying to split the UTF-8 string. It's a very odd place to get an error.