4 ms·

Interesting you mention http headers. I had a program converted Python 2 -> Python 2 which was crashing occasionally, and it turned out it was because I was bei

Interesting you mention http headers. I had a program converted Python 2 -> Python 2 which was crashing occasionally, and it turned out it was because I was being sent a http request which wasn't valid unicode, so decoding failed. I had to switch back to treating headers as bytes for as long as possible. It is a stupid client which doesn't send valid ascii for http headers of course.

dnautics 7y ago
> It is a stupid client which doesn't send valid ascii for http headers of course. ...or a smart malicious actor.
takeda 7y ago
I believe the headers are encoded using ISO-8859-1 not Unicode. That encoding has 1:1 mapping with bytes so wouldn't break this way. Treating them as UTF-8 was the bug.
- code_biologist 7y ago
  This is exactly the sort of encoding issues that the python 2 to 3 transition has flushed out. People get frustrated with python 3, yet the actual failure was their mishandling of encoding issues -- papered over by python 2.
  - xorcist 7y ago
    But that's not what frustrates people with the transition. It's that they suddenly get encoding issues where there should have been no encoding to begin with!
    - cbsmith 7y ago
      No observed encoding issues.
      - CJefferson 7y ago
        When I treated headers as bytes, there wasn't an "encoding". What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs.
        cbsmith 7y ago
        > When I treated headers as bytes, there wasn't an "encoding". If you are representing strings as bytes, you are intrinsically using an encoding. > What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs. Yes, it makes a mockery of the notion that "human readable data is easy". In many cases, you don't want to work with the actual strings in the data anyway, so bytes is the right thing to do. But yes, this strategy largely avoids encoding issues... until it doesn't.
        Dylan16807 7y ago
        > If you are representing strings as bytes, you are intrinsically using an encoding. It's just binary data that might resemble a string. No encoding necessary.
        fiedzia 7y ago
        This is false more often than not. Many programs taking user input will treat it as string, assuming specific encoding or compatibility with screen output/some api, at least in some code paths. For example if you print an error message when you can't open some file, you are very likely to assume its encoded in a way terminal can handle, so its no longer "just binary data".
        CJefferson 7y ago
- CJefferson 7y ago
  I'll admit, I'm not positive what the encoding should be. However there is a bunch of people who do clearly send UTF-8, and I can also promise you there are headers out there which just have binary nonsense in them. See for example https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/HttpHeaders https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web... If you want to handle all headers, you have to be prepared to just get binary data.
  - takeda 7y ago
    Yes, and using ISO-8859-1 is the way to handle them without issues. You will never get error when decoding it that way. If you are using UTF-8 there are character combinations that are invalid.

dnautics 7y ago

> It is a stupid client which doesn't send valid ascii for http headers of course. ...or a smart malicious actor.

takeda 7y ago

I believe the headers are encoded using ISO-8859-1 not Unicode. That encoding has 1:1 mapping with bytes so wouldn't break this way. Treating them as UTF-8 was the bug.

code_biologist 7y ago
This is exactly the sort of encoding issues that the python 2 to 3 transition has flushed out. People get frustrated with python 3, yet the actual failure was their mishandling of encoding issues -- papered over by python 2.
- xorcist 7y ago
  But that's not what frustrates people with the transition. It's that they suddenly get encoding issues where there should have been no encoding to begin with!
  - cbsmith 7y ago
    No observed encoding issues.
    - CJefferson 7y ago
      When I treated headers as bytes, there wasn't an "encoding". What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs.
      - cbsmith 7y ago
        > When I treated headers as bytes, there wasn't an "encoding". If you are representing strings as bytes, you are intrinsically using an encoding. > What I often want to do when reading user data is not treat it as a "encoded string", but just as a stream of bytes. Most data I work with (HTML files, output of other programs) can't be treated as anything but bytes, because people put junk in files / output of programs. Yes, it makes a mockery of the notion that "human readable data is easy". In many cases, you don't want to work with the actual strings in the data anyway, so bytes is the right thing to do. But yes, this strategy largely avoids encoding issues... until it doesn't.
        Dylan16807 7y ago
        > If you are representing strings as bytes, you are intrinsically using an encoding. It's just binary data that might resemble a string. No encoding necessary.
        fiedzia 7y ago
        This is false more often than not. Many programs taking user input will treat it as string, assuming specific encoding or compatibility with screen output/some api, at least in some code paths. For example if you print an error message when you can't open some file, you are very likely to assume its encoded in a way terminal can handle, so its no longer "just binary data".
        CJefferson 7y ago
CJefferson 7y ago
I'll admit, I'm not positive what the encoding should be. However there is a bunch of people who do clearly send UTF-8, and I can also promise you there are headers out there which just have binary nonsense in them. See for example https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest/HttpHeaders https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Web... If you want to handle all headers, you have to be prepared to just get binary data.
- takeda 7y ago
  Yes, and using ISO-8859-1 is the way to handle them without issues. You will never get error when decoding it that way. If you are using UTF-8 there are character combinations that are invalid.