4 ms·
Python 3 encoding management was broken, because it tried to impose Unicode semantics on things that were actually byte streams. For anyone actually correctly h
by morelisp 5y ago
Python 3 encoding management was broken, because it tried to impose Unicode semantics on things that were actually byte streams. For anyone actually correctly handling encodings in Python 2 it was awful because suddenly the language runtime was hiding half the data you needed.
- BiteCode_dev 5y agoNowadays, passing bytes to any os function returns bytes objects, not unicode. You'll get string if you pass string objects though, and they will be using utf8 surrogate escaping.
- morelisp 5y agoAnd nowadays, lots of people left the Python ecosystem completely because the 3 upgrade was broken and raising so many UnicodeDecodingErrors for so long. I'm glad it's fixed but it cost too much. (And it had nothing to do with UTF-8. Actually, it's at least partially caused by the CPython developers avoiding UTF-8 for poor reasons.)
- mark-r 5y agoI'd argue that the people correctly handling encodings in Python 2 were vastly outnumbered by the people that weren't, but were getting away with it because the code didn't outright crash. Now in Python 3 it crashes, which is a pain in the short term but better in the long run.
- morelisp 5y agoI don't really think it's better - certainly it was not better until many years after release when they started admitting their big mistakes. But even today, codepoint indexing is too easy and doesn't crash so lots of code is still subtly wrong. Memory usage of a string grows 3x if you add a single emoji. Most libraries are still agnostic about whether they take bytes or str so you still get exceptions thrown with no easy solution. (The growing popularity of type hints is fixing this, but that's not really related to Python 3.)