4 ms·

This is exactly the sort of encoding issues that the python 2 to 3 transition has flushed out. People get frustrated with python 3, yet the actual failure was t

This is exactly the sort of encoding issues that the python 2 to 3 transition has flushed out. People get frustrated with python 3, yet the actual failure was their mishandling of encoding issues -- papered over by python 2.

xorcist 7y ago
But that's not what frustrates people with the transition. It's that they suddenly get encoding issues where there should have been no encoding to begin with!
- cbsmith 7y ago
  No observed encoding issues.
  - CJefferson 7y ago
    When I treated headers as bytes, there wasn't an "encoding". What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs.
    - cbsmith 7y ago
      > When I treated headers as bytes, there wasn't an "encoding". If you are representing strings as bytes, you are intrinsically using an encoding. > What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs. Yes, it makes a mockery of the notion that "human readable data is easy". In many cases, you don't want to work with the actual strings in the data anyway, so bytes is the right thing to do. But yes, this strategy largely avoids encoding issues... until it doesn't.
      - Dylan16807 7y ago
        > If you are representing strings as bytes, you are intrinsically using an encoding. It's just binary data that might resemble a string. No encoding necessary.
        fiedzia 7y ago
        This is false more often than not. Many programs taking user input will treat it as string, assuming specific encoding or compatibility with screen output/some api, at least in some code paths. For example if you print an error message when you can't open some file, you are very likely to assume its encoded in a way terminal can handle, so its no longer "just binary data".
        CJefferson 7y ago
        Yes, I have to worry about how to make a "best effort" to show it to users, but in all internal code paths it must stay as "just binary data", else I lose information. This is exactly how chrome and Firefox handle headers internally.
        cbsmith 7y ago
        It might resemble a particular encoding of a string... and the way you got that string to that particular sequence of bytes is by... encoding it.
        Dylan16807 7y ago
        > and the way you got that string to that particular sequence of bytes No I didn't. Those bytes came from an external source. My primary job is to preserve the exact sequence, whether I can make sense of it or not.
        cbsmith 7y ago
        In that context, you aren't using strings. You are using bytes. HTML without interpreting it as strings isn't really HTML, nor is it a string. It's just a blob that is passing through.
    - takeda 7y ago
      > When I treated headers as bytes, there wasn't an "encoding". oh, actually there was (either us-ascii or more likely iso-8859-1) the bytes are just values 0-255 what these values mean is the encoding. You're confused because the encoding was implicit, rather than explicit. It would perhaps be clearer to see it if you for example had to chose if you use ASCII or legacy EBCDIC encoding.

xorcist 7y ago

But that's not what frustrates people with the transition. It's that they suddenly get encoding issues where there should have been no encoding to begin with!

cbsmith 7y ago
No observed encoding issues.
- CJefferson 7y ago
  When I treated headers as bytes, there wasn't an "encoding". What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs.
  - cbsmith 7y ago
    > When I treated headers as bytes, there wasn't an "encoding". If you are representing strings as bytes, you are intrinsically using an encoding. > What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs. Yes, it makes a mockery of the notion that "human readable data is easy". In many cases, you don't want to work with the actual strings in the data anyway, so bytes is the right thing to do. But yes, this strategy largely avoids encoding issues... until it doesn't.
    - Dylan16807 7y ago
      > If you are representing strings as bytes, you are intrinsically using an encoding. It's just binary data that might resemble a string. No encoding necessary.
      - fiedzia 7y ago
        This is false more often than not. Many programs taking user input will treat it as string, assuming specific encoding or compatibility with screen output/some api, at least in some code paths. For example if you print an error message when you can't open some file, you are very likely to assume its encoded in a way terminal can handle, so its no longer "just binary data".
        CJefferson 7y ago
        Yes, I have to worry about how to make a "best effort" to show it to users, but in all internal code paths it must stay as "just binary data", else I lose information. This is exactly how chrome and Firefox handle headers internally.
      - cbsmith 7y ago
        It might resemble a particular encoding of a string... and the way you got that string to that particular sequence of bytes is by... encoding it.
        Dylan16807 7y ago
        > and the way you got that string to that particular sequence of bytes No I didn't. Those bytes came from an external source. My primary job is to preserve the exact sequence, whether I can make sense of it or not.
        cbsmith 7y ago
        In that context, you aren't using strings. You are using bytes. HTML without interpreting it as strings isn't really HTML, nor is it a string. It's just a blob that is passing through.
  - takeda 7y ago
    > When I treated headers as bytes, there wasn't an "encoding". oh, actually there was (either us-ascii or more likely iso-8859-1) the bytes are just values 0-255 what these values mean is the encoding. You're confused because the encoding was implicit, rather than explicit. It would perhaps be clearer to see it if you for example had to chose if you use ASCII or legacy EBCDIC encoding.