4 ms·

The incorrect docker-compose file was generated by Java (Jetbrains) but consumed by Python (docker-compose). The GP comment was complaining about Python's stric

The incorrect docker-compose file was generated by Java (Jetbrains) but consumed by Python (docker-compose). The GP comment was complaining about Python's strict Unicode consumption, not Java's invalid Unicode generation.

geofft 5y ago
Oh, I see. But if it was UTF-8 it would have worked... I guess the problem is that JetBrains is generating the file in (e.g.) Windows-1252, and Python needs to be told that? Does it work if you set the environment variable PYTHONENCODING to cp1252? (I suppose I should either contact the author, or try it myself...)
- deleted 5y ago
  [deleted]
- deathanatos 5y ago
  Normally I'd agree, a windows-1252 misencode should be one's default guess when mojibake is afoot. Unfortunately, the errant byte in the error is 0xb3, which is "³" in windows-1252. If you Google, "Mikołaj", > Mikołaj is the Polish cognate of given name Nicholas Then Google, "windows character encoding polish" > Windows-1250 - Wikipedia And 0xb3 is "ł" in that encoding.¹ > Does it work if you set the environment variable PYTHONENCODING (sic) to cp1252? I don't know if setting PYTHONIOENCODING would work here; I don't think it should affect this. Really, fixing the YAML file is the fix. (And fixing the thing that generated it.) ¹it is queries like this that really make me love the search engines of today. This would have been hell in the days of Alta Vista.
- stubish 5y ago
  JetBrains is generating an invalid YAML file, which are UTF-8. If they were using a decent YAML library, it would have crashed at that point. And firmly pointed the finger at the real bug, reading raw bytes from the environment or a .properties file parser and assuming it is valid UTF-8. And this is why you always validate your data when you slurp it in, or else you pass crap down several layers where it crashes or mostly works with the potential for security holes or catastrophic behavior, and a pain in the arse to track down since the actual bug is nowhere near where you are looking.
  - account42 5y ago
    I'd expect a decent YAML library to have functions taking UTF-8 and not wasting time verifying that the data passed is actually UTF-8 in release builds.
    - stubish 5y ago
      You generally don't verify on output, because you verified on input (especially with languages where text strings are Unicode or UTF-8 byte strings like Python3 or Rust). But it would also be a premature optimization when it does make sense to check. For expected YAML use cases I doubt it would be a measurable difference in runtime. And it has to inspect the strings in any case to correctly quote things and deal with indentation if there are newlines.
deathanatos 5y ago
The Docker compose file is YAML. My reading of YAML's standard is that it must be in one of the Unicode encodings, and the smell I get from the article is that it is probably in windows-1250 (the CP Windows would use for Polish; Mikołaj is a Polish name, 0xb3, the octet in the error, is the Windows-1250 encoding of "ł"); thus, it isn't valid YAML. I'm not sure what sane behavior Python could have here besides errorring. > EVERY language should _try_ to handle Unicode such that if a data sequence were valid before it remains valid after. This sequence was never valid, and never will be. > in the article's case, the correct answer is GIGO. Just pass it through and hope it continues to work. Dear God, no; emit a diagnostic and abort. Countless decades of existing code have shown time and again that "plow forward with some hot garbage" is not a good idea. But that ignores that … that that isn't how any of this works; the YAML parse is going to want to emit strings, which the incoming data isn't.
- account42 5y ago
  > This sequence was never valid Neither were four-byte UTF-8 characters at some point. > and never will be. We shall see.

geofft 5y ago

Oh, I see. But if it was UTF-8 it would have worked... I guess the problem is that JetBrains is generating the file in (e.g.) Windows-1252, and Python needs to be told that? Does it work if you set the environment variable PYTHONENCODING to cp1252? (I suppose I should either contact the author, or try it myself...)

deleted 5y ago
[deleted]
deathanatos 5y ago
Normally I'd agree, a windows-1252 misencode should be one's default guess when mojibake is afoot. Unfortunately, the errant byte in the error is 0xb3, which is "³" in windows-1252. If you Google, "Mikołaj", > Mikołaj is the Polish cognate of given name Nicholas Then Google, "windows character encoding polish" > Windows-1250 - Wikipedia And 0xb3 is "ł" in that encoding.¹ > Does it work if you set the environment variable PYTHONENCODING (sic) to cp1252? I don't know if setting PYTHONIOENCODING would work here; I don't think it should affect this. Really, fixing the YAML file is the fix. (And fixing the thing that generated it.) ¹it is queries like this that really make me love the search engines of today. This would have been hell in the days of Alta Vista.
stubish 5y ago
JetBrains is generating an invalid YAML file, which are UTF-8. If they were using a decent YAML library, it would have crashed at that point. And firmly pointed the finger at the real bug, reading raw bytes from the environment or a .properties file parser and assuming it is valid UTF-8. And this is why you always validate your data when you slurp it in, or else you pass crap down several layers where it crashes or mostly works with the potential for security holes or catastrophic behavior, and a pain in the arse to track down since the actual bug is nowhere near where you are looking.
- account42 5y ago
  I'd expect a decent YAML library to have functions taking UTF-8 and not wasting time verifying that the data passed is actually UTF-8 in release builds.
  - stubish 5y ago
    You generally don't verify on output, because you verified on input (especially with languages where text strings are Unicode or UTF-8 byte strings like Python3 or Rust). But it would also be a premature optimization when it does make sense to check. For expected YAML use cases I doubt it would be a measurable difference in runtime. And it has to inspect the strings in any case to correctly quote things and deal with indentation if there are newlines.

deathanatos 5y ago

The Docker compose file is YAML. My reading of YAML's standard is that it must be in one of the Unicode encodings, and the smell I get from the article is that it is probably in windows-1250 (the CP Windows would use for Polish; Mikołaj is a Polish name, 0xb3, the octet in the error, is the Windows-1250 encoding of "ł"); thus, it isn't valid YAML. I'm not sure what sane behavior Python could have here besides errorring. > EVERY language should _try_ to handle Unicode such that if a data sequence were valid before it remains valid after. This sequence was never valid, and never will be. > in the article's case, the correct answer is GIGO. Just pass it through and hope it continues to work. Dear God, no; emit a diagnostic and abort. Countless decades of existing code have shown time and again that "plow forward with some hot garbage" is not a good idea. But that ignores that … that that isn't how any of this works; the YAML parse is going to want to emit strings, which the incoming data isn't.

account42 5y ago
> This sequence was never valid Neither were four-byte UTF-8 characters at some point. > and never will be. We shall see.