4 ms·

Oh, I see. But if it was UTF-8 it would have worked... I guess the problem is that JetBrains is generating the file in (e.g.) Windows-1252, and Python needs to

by geofft 5y ago

Oh, I see. But if it was UTF-8 it would have worked... I guess the problem is that JetBrains is generating the file in (e.g.) Windows-1252, and Python needs to be told that? Does it work if you set the environment variable PYTHONENCODING to cp1252? (I suppose I should either contact the author, or try it myself...)

deleted 5y ago
[deleted]
deathanatos 5y ago
Normally I'd agree, a windows-1252 misencode should be one's default guess when mojibake is afoot. Unfortunately, the errant byte in the error is 0xb3, which is "³" in windows-1252. If you Google, "Mikołaj", > Mikołaj is the Polish cognate of given name Nicholas Then Google, "windows character encoding polish" > Windows-1250 - Wikipedia And 0xb3 is "ł" in that encoding.¹ > Does it work if you set the environment variable PYTHONENCODING (sic) to cp1252? I don't know if setting PYTHONIOENCODING would work here; I don't think it should affect this. Really, fixing the YAML file is the fix. (And fixing the thing that generated it.) ¹it is queries like this that really make me love the search engines of today. This would have been hell in the days of Alta Vista.
stubish 5y ago
JetBrains is generating an invalid YAML file, which are UTF-8. If they were using a decent YAML library, it would have crashed at that point. And firmly pointed the finger at the real bug, reading raw bytes from the environment or a .properties file parser and assuming it is valid UTF-8. And this is why you always validate your data when you slurp it in, or else you pass crap down several layers where it crashes or mostly works with the potential for security holes or catastrophic behavior, and a pain in the arse to track down since the actual bug is nowhere near where you are looking.
- account42 5y ago
  I'd expect a decent YAML library to have functions taking UTF-8 and not wasting time verifying that the data passed is actually UTF-8 in release builds.
  - stubish 5y ago
    You generally don't verify on output, because you verified on input (especially with languages where text strings are Unicode or UTF-8 byte strings like Python3 or Rust). But it would also be a premature optimization when it does make sense to check. For expected YAML use cases I doubt it would be a measurable difference in runtime. And it has to inspect the strings in any case to correctly quote things and deal with indentation if there are newlines.

by geofft 5y ago

deleted 5y ago

[deleted]

deathanatos 5y ago

Normally I'd agree, a windows-1252 misencode should be one's default guess when mojibake is afoot. Unfortunately, the errant byte in the error is 0xb3, which is "³" in windows-1252. If you Google, "Mikołaj", > Mikołaj is the Polish cognate of given name Nicholas Then Google, "windows character encoding polish" > Windows-1250 - Wikipedia And 0xb3 is "ł" in that encoding.¹ > Does it work if you set the environment variable PYTHONENCODING (sic) to cp1252? I don't know if setting PYTHONIOENCODING would work here; I don't think it should affect this. Really, fixing the YAML file is the fix. (And fixing the thing that generated it.) ¹it is queries like this that really make me love the search engines of today. This would have been hell in the days of Alta Vista.

stubish 5y ago

JetBrains is generating an invalid YAML file, which are UTF-8. If they were using a decent YAML library, it would have crashed at that point. And firmly pointed the finger at the real bug, reading raw bytes from the environment or a .properties file parser and assuming it is valid UTF-8. And this is why you always validate your data when you slurp it in, or else you pass crap down several layers where it crashes or mostly works with the potential for security holes or catastrophic behavior, and a pain in the arse to track down since the actual bug is nowhere near where you are looking.

account42 5y ago
I'd expect a decent YAML library to have functions taking UTF-8 and not wasting time verifying that the data passed is actually UTF-8 in release builds.
- stubish 5y ago
  You generally don't verify on output, because you verified on input (especially with languages where text strings are Unicode or UTF-8 byte strings like Python3 or Rust). But it would also be a premature optimization when it does make sense to check. For expected YAML use cases I doubt it would be a measurable difference in runtime. And it has to inspect the strings in any case to correctly quote things and deal with indentation if there are newlines.