5 ms·

> in python if you open a file (without "b" option, the python will do the translation on the fly and you don't need to worry about it) Of course, if you don't

> in python if you open a file (without "b" option, the python will do the translation on the fly and you don't need to worry about it) Of course, if you don't know what encoding the file was opened with, you don't know what characters can be written to the file. I was bitten by this with Python 3.5 on Windows. I naively assumed the default file encoding would be UTF-8 or UTF-16, but it was actually CP-1252, so my program would crash upon trying to write a non-ASCII character.

mark-r 7y ago
Every Python program should be tested with Emoji characters, they're a real torture test.
- WorldMaker 7y ago
  Every program in general should be tested with Emoji characters at this point.
  - mark-r 7y ago
    Not a bad idea, but I think Python is more likely to have hidden bugs that this will uncover. A language that accepts bytes as input and emits the same on output will probably work fine on UTF-8 for example.
    - WorldMaker 7y ago
      That's the Python 2 mentality and a large part of this discussion was that it didn't work in hindsight, that you can't just be "encoding oblivious", but it usually doesn't show up as an obvious problem until you least expect it. Our input and output devices are aren't always homozygous on byte encoding (and quite possibly very rarely are; we have decades and decades of kludges around this), and testing every program with Emoji has become one of my favorite pastimes for finding failure cases.
- slavik81 7y ago
  Note that you need to test on every platform, as the default file encoding may vary. I missed that bug in part because it worked correctly on Linux.
  - mark-r 7y ago
    Good point. I do almost all of my Python on Windows where it's much easier to get an error.
takeda 7y ago
It defaults to the system encoding. I don't use Python on Windows, but Windows evolved its default encoding over time, the code pages were popular in Windows 9x, starting with NT based (2000, XP...) They used UTF-16 I believe and then Windows 7? It became UTF-8. Perhaps Python needs to be updated to reflect that? You can also specify encoding when calling open.
- Dylan16807 7y ago
  > Windows evolved its default encoding over time, the code pages were popular in Windows 9x, starting with NT based (2000, XP...) They used UTF-16 I believe and then Windows 7? It became UTF-8. They bolted on a separate set of functions that took UCS-2 and now take UTF-16. The actual code pages, to this day, are legacy things that are mostly 8 bits. My system is set to code pages 437 and 1252, for example. They put together a code page for UTF-8 but it's behind a 'beta' warning.
  - ygra 7y ago
    > They bolted on a separate set of functions that took UCS-2 and now take UTF-16. NT actually bolted on 8-bit versions of the native Unicode functions. FooBarA is a wrapper around FooBarW. > They put together a code page for UTF-8 but it's behind a 'beta' warning. Codepage 65001 has been a thing for quite a while. It's just that it's variable-width per character and few applications are ready to handle that when they assume a 1:1 or 2:1 relationship between bytes and characters. It does work sort of for applications that don't do too weird stuff to text, though, and can be a useful workaround in such cases to get UTF-8 support into legacy applications. But in general, Windows is UTF-16LE and the code pages are indeed legacy cruft that no application should touch or even use. Sadly much software ported from Unix-likes notices »Hey, there's a default encoding in Windows too, so let's just use that«.
- slavik81 7y ago
  The default file encoding for Windows was changed to UTF-8 in Python 3.6. That particular problem on that particular platform is now a thing of the past. It was just an example of why implicit conversions in the standard library functions don't save you from having to think about encodings. You get much more robust and user-friendly programs when you explicitly consider your encodings and the error-handling strategies to go with them.

mark-r 7y ago

Every Python program should be tested with Emoji characters, they're a real torture test.

WorldMaker 7y ago
Every program in general should be tested with Emoji characters at this point.
- mark-r 7y ago
  Not a bad idea, but I think Python is more likely to have hidden bugs that this will uncover. A language that accepts bytes as input and emits the same on output will probably work fine on UTF-8 for example.
  - WorldMaker 7y ago
    That's the Python 2 mentality and a large part of this discussion was that it didn't work in hindsight, that you can't just be "encoding oblivious", but it usually doesn't show up as an obvious problem until you least expect it. Our input and output devices are aren't always homozygous on byte encoding (and quite possibly very rarely are; we have decades and decades of kludges around this), and testing every program with Emoji has become one of my favorite pastimes for finding failure cases.
slavik81 7y ago
Note that you need to test on every platform, as the default file encoding may vary. I missed that bug in part because it worked correctly on Linux.
- mark-r 7y ago
  Good point. I do almost all of my Python on Windows where it's much easier to get an error.

takeda 7y ago

It defaults to the system encoding. I don't use Python on Windows, but Windows evolved its default encoding over time, the code pages were popular in Windows 9x, starting with NT based (2000, XP...) They used UTF-16 I believe and then Windows 7? It became UTF-8. Perhaps Python needs to be updated to reflect that? You can also specify encoding when calling open.

Dylan16807 7y ago
> Windows evolved its default encoding over time, the code pages were popular in Windows 9x, starting with NT based (2000, XP...) They used UTF-16 I believe and then Windows 7? It became UTF-8. They bolted on a separate set of functions that took UCS-2 and now take UTF-16. The actual code pages, to this day, are legacy things that are mostly 8 bits. My system is set to code pages 437 and 1252, for example. They put together a code page for UTF-8 but it's behind a 'beta' warning.
- ygra 7y ago
  > They bolted on a separate set of functions that took UCS-2 and now take UTF-16. NT actually bolted on 8-bit versions of the native Unicode functions. FooBarA is a wrapper around FooBarW. > They put together a code page for UTF-8 but it's behind a 'beta' warning. Codepage 65001 has been a thing for quite a while. It's just that it's variable-width per character and few applications are ready to handle that when they assume a 1:1 or 2:1 relationship between bytes and characters. It does work sort of for applications that don't do too weird stuff to text, though, and can be a useful workaround in such cases to get UTF-8 support into legacy applications. But in general, Windows is UTF-16LE and the code pages are indeed legacy cruft that no application should touch or even use. Sadly much software ported from Unix-likes notices »Hey, there's a default encoding in Windows too, so let's just use that«.
slavik81 7y ago
The default file encoding for Windows was changed to UTF-8 in Python 3.6. That particular problem on that particular platform is now a thing of the past. It was just an example of why implicit conversions in the standard library functions don't save you from having to think about encodings. You get much more robust and user-friendly programs when you explicitly consider your encodings and the error-handling strategies to go with them.