4 ms·
> We have decided as a team that a change as big as unicode/str/bytes will never happen so abruptly again. When we started Python 3 we thought/hoped that the co
by danso 11y ago
> We have decided as a team that a change as big as unicode/str/bytes will never happen so abruptly again. When we started Python 3 we thought/hoped that the community would do what Python did and do one last feature release supporting Python 2 and then cut over to Python 3 development for feature development while doing bugfix releases only for the Python 2 version.
I'm guessing it's not a coincidence that string encoding was also behind the Great Sadness of Moving From Ruby 1.8 to 1.9. How have other mainstream languages made this jump, if it was needed, and were they able to do it in a non-breaking way?
https://news.ycombinator.com/item?id=1162122 https://news.ycombinator.com/item?id=1162122
- perlgeek 11y agoWell, Perl 6 also introduces a distinction between strings (Unicode) und buffers (octets), but it also introduces loads of other changes, to the point where it's its own language. That said, the Unicode features were a major thing planned for PHP 6, and (afaict) one of the reasons there has never been a PHP 6, but rather they went straight from 5 to 7. I'm not aware of any graceful transitions.
- bdarnell 11y agoC and C++ are so widely used that transitions like this are made not at the language level but at the level of platforms or other communities. Some parts of the C/C++ world made this transition relatively seamlessly, while others got caught in the same traps as Python. The key is UTF-8: UTF-8 is a superset of 7-bit ASCII, so as long as you only convert to/from other encodings at the boundaries of your system, unicode can be introduced to the internal components in a gradual and mostly-compatible way. You only get in trouble when you decide that you need separate "byte string" and "character string" data types (which is generally a mistake: due to the existince of combining characters, graphemes are variable-width even if you're using strings composed of unicode code points, so you don't gain much by using UCS-4 character strings instead of UTF-8 byte strings). My theory is that the python 3 transition would have gone much smoother and still accomplished its goals if they had left the implicit str/bytes conversion in place but just made it use UTF-8 instead of ASCII (although in environments like Windows where UTF-16 is important this may not have worked out as well).
- dietrichepp 11y agoYou are correct that there's no real benefit in UTF-32 over UTF-8, which is why Go and Rust (and others) have worked with UTF-8 in memory just fine. However, the actual encoding of a str object is irrelevant, and it's not the point. The whole point of the str/bytes difference in Python is that you make Python keep track of whether you've done the conversion or not. In Python 2, you can be sloppy, and the programs are buggy as a result! You're putting the cart before the horse here in terms of the Python 3 transition. The str/unicode fix was one of the driving factors for Python 3 to exist in the first place, and if you removed it, then what's the point? Again, look at Go or Rust. Both of them have separate types for strings and bytes, even though they have the same representation in memory, and as a result we don't have that kind of bug in our program.
- bdarnell 11y ago> The str/unicode fix was one of the driving factors for Python 3 to exist in the first place, and if you removed it, then what's the point? The reason python 3 exists is that most python 2 code had latent unicode-related bugs that would only manifest when they were exposed to non-ascii data. The backwards-incompatible barrier between str and bytes was the solution the python 3 team chose for this problem; adopting utf-8 as the standard encoding would have been another solution which I claim would have been more backwards-compatible (essentially moving to the go/rust model, which prove that you don't necessarily need separate byte and character string types for correct unicode handling).
- hetman 11y agoBut not every 8-bit byte string is valid UTF-8 so that could still cause a world of pain.
- e12e 11y agoBut apart from being "Internet compatible", it makes little sense to move from ascii only to utf-8. It only makes sense if you're already trying to deal with stuff that doesn't fit in ascii. Don't get me wrong, I think ascii was a terrible hack (I seem to recall one of the designers called it his biggest mistake/regret - and that they should've gone with some kind of prefix-encoding). But the thing with unicode-everywhere is that it's more beginner friendly: you can have unicode in your variable names, and strings without giving it another thought. $ cat u.py from __future__ import print_function å="æ" print(å) $ python2 u.py File "u.py", line 2 SyntaxError: Non-ASCII character '\xc3' in file u.py on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details $ python3 u.py æ Sure, the explanation of what's wrong is right there in the error-message, but it seems a bit of an unnecessary hurdle for people to get around to write important programs, that ask you to type in your name, and then prints out the name ten times ;-)
- autarch 11y agoPerl 5 introduced Unicode support in 5.6.0 (2000) but it was kind of a mess. It was essentially redesigned in 5.8.0 (2002). At that point it was fairly buggy, but by 5.8.8 (2006) it was in pretty good shape. The most recent versions of Perl 5 (5.22.1 was released a few days ago) have excellent Unicode support. That said, Perl 5 does not have different types for strings and bytes, which is definitely a source of bugs. Since Unicode support is essentially opt-in for a library (you have to explicitly decode/encode data somehow) it's easily possible for a library you use to break your data. Most of the major libraries (database, HTTP, etc.) have long since been patched to respect Unicode in appropriate ways, so the state of Unicode in Perl 5 is good overall.