4 ms·
> You just need to be aware that in some cases the work is already done for you by the language, for example in python if you open a file (without "b" option, t
by masklinn 7y ago
> You just need to be aware that in some cases the work is already done for you by the language, for example in python if you open a file (without "b" option, the python will do the translation on the fly and you don't need to worry about it)
Sadly they fucked up that part rather thoroughly, because the default encoding is `locale.getpreferredencoding()`, which ensures it's going to be wrong at the least possible convenient time and on the devices least accessible for debugging.
Do not ever use text-mode `open` without specifying an encoding.
- mintplant 7y agoNode.js tries to be helpful in defaulting file writes to UTF-8, but defaults file reads to returning a raw byte buffer [0]. So you have to either remember to treat the two operations differently, or, like in Python, manually specify the encoding for both. [0] I seem to recall that it used to default to the locale's preferred encoding, but I could have my wires crossed with other languages' standard libraries there.
- takeda 7y agoThe locales are provided by LANG and other locate variables, so Python will use whatever is set in environment, you can also specify the encoding in one of open() parameters.
- masklinn 7y ago> The locales are provided by LANG and other locate variables Which is absolutely not what you want when, say, opening your own data files. Even when opening the user’s files it’s likely not what you want. > you can also specify … And what I’m saying is this is not a “can also” it’s a “must”. Not doing so will bite you in the ass, because “whatever random garbage is on the machine” is really not what you want a default to be.
- takeda 7y agoOh I see your point. Looks like they changed the behavior in 3.7 (they added -X UTF-8 option), but being able to set it from the application would be great.