19 ms·

Why Python 3 Exists

418 points by cocoflunchy 11y ago

Scarbutt 11y ago
Since python3 is not backwards compatible with python2, why didn't the python devs leverage the opportunity for creating a more performant non-GIL runtime for python3?
- chrisseaton 11y ago
  > more performant non-GIL runtime for python3 Because making one of those those is a huge amount of work, may introduce more serious backwards incompatibilities (like C extensions), and not everyone has the knowledge of how to do it so they'd need to either learn from scratch themselves or find people interested in doing it for them.
- svisser 11y ago
  Removing the GIL is a difficult problem: http://python-notes.curiousefficiency.org/en/latest/python3/multicore_python.html http://python-notes.curiousefficiency.org/en/latest/python3/...
  - criddell 11y ago
    I think the question was if you aren't going to be backwards compatible, why not unshackle yourself completely and design a new language without the GIL and make it as pythonic as possible? A scripting language that support multi-threading is possible, right? I think TCL does it.
    - chrisseaton 11y ago
      I guess they were happy with breaking backwards compatibility a little bit, not not as much as simply removing the GIL and not adding back any implicit synchronisation at all.
    - mkesper 11y ago
      I think this question can only be raised in hindsight. I think nobody suspected the unicode transition to be so slow and painful. As there probably won't be such a big transition evermore, I wouldn't hold my breath for it happening.
- js8 11y ago
  Is there a way to do it? My understanding always was that removing GIL means that the code that runs single-threaded (which is like 95% of Python code out there) will get a performance hit.
  - andreasvc 11y ago
    Yes it does mean that and I believe it is unavoidable, because you need to replace the coarse-grained GIL with a fine-grained locking instructions. However, I think the trade off is now worth it, since multi-core CPUs are ubiquitous. Besides, Python itself was never notable for its performance to begin with, so I've always found the argument very odd.
    - js8 11y ago
      It is perfectly avoided by not removing the GIL. :-) Multi-core CPUs may be ubiquitous, but multi-threaded Python code is not, by far. So performance of say, my Python code, would decrease, thank you, I don't need that. It's not that Python would have superb performance, I just don't see the reason to pay for something that's not actually needed. It's also often forgotten that one of the easiest paths to parallelism is to just run multiple processes (and for many problems you only need to split the input data, the processes don't need to even communicate), and this solution naturally uses multiple CPUs without any GIL worries. I think the actual use cases that require GIL removal are very little (something like a server, perhaps), and if you actually need to do that, I think you're better served with some JVM language or Go.
      - andreasvc 11y ago
        What I'm saying is that not only does Python not have superb performance, its performance is actually largely irrelevant. The advantage of Python is in ease of use, capabilities as glue language, readability, etc. So that's why I really don't see why you wouldn't remove the GIL at a reasonable slowdown factor (2x was a figure thrown around IIRC). People always proclaim that you can just use multiple processes for parallelism. It's nice when it suits your (embarrassingly parallel) problem but when you need to share a large amount of data between processes it's a major hassle.
- eric_bullington 11y ago
  Removing the GIL is non-trivial. But I'm surprised that they didn't take advantage of Python 3 to move to a more performant register-based interpreter. Several people expressed interest in building one, and working prototypes were even made, but the core developers didn't seem much interested in performance upgrades. I think their attitude is generally, "if it's performance-critical, write it in C", which was a good approach 15 years ago. However, Python's competition now comes from languages like Go and Rust that have both good performance and are user-friendly and productive (with features such as expressive syntax and a comprehensive standard library).
- lmm 11y ago
  Because the runtime with the GIL is more performant and python users know that it's practically impossible to write correct software using threads.
  - scardine 11y ago
    Yes, this is the point. The GIL matters only if you are using threads with CPU-bound tasks (most of my work is I/O bound). When I have CPU-bound problems that need parallelism and are not yet well implemented in numpy/scipy, I would ratter use AMQP instead of threads.
- morebetterer 11y ago
  It's a shame the GIL won't be removed because it's perceived to be too difficult. It's trivial to remove the GIL - probably a week of mechanical work. Just don't depend on global variables in the interpreter. No global state, no problem. But the Python C API has to be changed to always store interpreter state into a struct, and a pointer to that interpreter state struct has to be passed as the first argument in all C API calls. Not rocket science; this is what Lua does. It's a political decision to keep the GIL, not a technical one. As for preserving C API backwards compatibility, it's a straw man argument - the Python API broke from 2.x to 3.x anyway. There's no such thing as "lesser breakage" - only breakage.
  - laurencerowe 11y ago
    Supporting multiple interpreters in a single process is not what most people mean when they talk about removing the GIL. Objects from one interpreter could not be used safely from another interpreter. This would be handy in a few situations but is essentially not all that different to multiprocessing. What people usually mean when they talk about removing the GIL is having multi-threaded code make use of multiple cpu cores (as it does in Jython.) This would involved splitting the GIL into more fine-grained locks. Unfortunately experiments taking this approach have so far shown significant performance impacts for single threaded code.
    - morebetterer 11y ago
      It would be inefficient to share data structures or interpreter internal state across threads. This would lead to the poor performance you described with mutex contention. It is better to have an interpreter instance per thread, each having their own separate variable pool, and pass messages between the different interpreter instances. This model scales well with multicore systems and is faster than the multi-process equivalent. It's also a simpler model to implement and maintain. Python's current multithreading model is too complicated from an implementation point of view.
- BuckRogers 11y ago
  Because that's difficult to do. But the PyPy team has already done it for Python2 code. Check into PyPy-STM.
danso 11y ago
> We have decided as a team that a change as big as unicode/str/bytes will never happen so abruptly again. When we started Python 3 we thought/hoped that the community would do what Python did and do one last feature release supporting Python 2 and then cut over to Python 3 development for feature development while doing bugfix releases only for the Python 2 version. I'm guessing it's not a coincidence that string encoding was also behind the Great Sadness of Moving From Ruby 1.8 to 1.9. How have other mainstream languages made this jump, if it was needed, and were they able to do it in a non-breaking way? https://news.ycombinator.com/item?id=1162122 https://news.ycombinator.com/item?id=1162122
- perlgeek 11y ago
  Well, Perl 6 also introduces a distinction between strings (Unicode) und buffers (octets), but it also introduces loads of other changes, to the point where it's its own language. That said, the Unicode features were a major thing planned for PHP 6, and (afaict) one of the reasons there has never been a PHP 6, but rather they went straight from 5 to 7. I'm not aware of any graceful transitions.
- bdarnell 11y ago
  C and C++ are so widely used that transitions like this are made not at the language level but at the level of platforms or other communities. Some parts of the C/C++ world made this transition relatively seamlessly, while others got caught in the same traps as Python. The key is UTF-8: UTF-8 is a superset of 7-bit ASCII, so as long as you only convert to/from other encodings at the boundaries of your system, unicode can be introduced to the internal components in a gradual and mostly-compatible way. You only get in trouble when you decide that you need separate "byte string" and "character string" data types (which is generally a mistake: due to the existince of combining characters, graphemes are variable-width even if you're using strings composed of unicode code points, so you don't gain much by using UCS-4 character strings instead of UTF-8 byte strings). My theory is that the python 3 transition would have gone much smoother and still accomplished its goals if they had left the implicit str/bytes conversion in place but just made it use UTF-8 instead of ASCII (although in environments like Windows where UTF-16 is important this may not have worked out as well).
andrewstuart 11y ago
IMO one of the reasons for all the angst is that .encode() and .decode() are so ambiguous and unintuitive which makes them incredibly confusing to use. Which direction are you converting? From what to what? The whole unicode thing is hard enough to understand without Python's encoding and decoding functions adding to the mystery. I still have to refer to the documentation to make sure I'm encoding or decoding as expected. I think there would have been much less of a problem if encode and decode were far more obvious, unambiguous and intuitive to use. Probably without there being two functions. Still a problem of course today.
- cjauvin 11y ago
  I know it's a deeply subjective and personal thing unlikely to help anyone other than me, but I've always successfully relied on the following mnemonic for this particular problem: * unicode -> encode, u -> e (vowels) * str -> decode, s -> d (consonants)
  - deleted 11y ago
    [deleted]
  - andrewstuart 11y ago
    There should be one obvious way to do it.
    - davidism 11y ago
      > Although that way may not be obvious at first unless you're Dutch. There is one obvious way: strings are encoded to bytes, bytes are decoded to strings.
      - jes5199 11y ago
        I think that's backwards
        rspeer 11y ago
onesixtythree 11y ago
From the outside, Python 3 seems like a much better language. I don't have strong views of its object system (I avoid OOP as much as I can) but it seems like the string/bytes handling is much better, and I'm also a fan of map and filter returning generators rather than being hard-coded to a list implementation (stream fusion is a good thing). Also, I fail to see any value in print being a special keyword instead of a regular function (as in 3). What I don't get is: why has Python 3 adoption been so slow? Is it just backward compatibility, or are there deeper problems with it that I'm not aware of?
- harryf 11y ago
  What makes me snarky is the replacing of '%s %s' % ('one', 'two') With '%s %s'.format('one', 'two') The latter is just more annoying to type. Stupid argument I know but I find myself grumbling to myself every time...
  - davidism 11y ago
    It's not replaced, both are perfectly valid. And this addition also exists in Python 2.
  - RussianCow 11y ago
    The second example should be: '{} {}'.format('one', 'two') Either way, the former example still works in Python 3.5, so that syntax hasn't gone away. `format` is preferred, though. This Stack Overflow question has some good answers as to why: http://stackoverflow.com/questions/5082452/python-string-formatting-vs-format http://stackoverflow.com/questions/5082452/python-string-for...
  - dietrichepp 11y ago
    You mean, '{} {}'.format('one', 'two') Your experience may vary, but in my experience, when I switched to .format(), I found a number of bugs in code that used % instead. As mentioned, you can continue using %.
    - 11y ago
mathgenius 11y ago
Ok, fine. Can we have the print statement back?
- untothebreach 11y ago
  Genuinely curious, why do you prefer `print` to be a statement rather than a function? I've heard a lot of criticisms of Py3, but this is the first time I've heard this one.
  - bufordsharkley 11y ago
    I think a function is "better", but the print statement should have been preserved. Print is used either for: 1) Writing to stdout/stderr 2) Debugging the hacky way The print function is better for the former (though more often than not, I use Armin Ronacher's click.echo for compability)[0], but I fastly prefer the print statement for (2). Not dealing with parentheses is always a plus; I can add and remove print statements far more quickly. I find it to be an increase in friction, and I don't see any real downside in leaving it in. [0] http://click.pocoo.org/5/api/#click.echo http://click.pocoo.org/5/api/#click.echo
    - untothebreach 11y ago
      I agree with you on 1), I much prefer `print(..., file=sys.stderr)` to `print >>sys.stderr, ...`. For 2), though, I have always used auto-inserting parens in my vim, so I never really experienced any pain w.r.t. parentheses. I can see how that would be a pain, though.
    - Retra 11y ago
      I prefer the print function for debuging, because I can just do s/print/log.debug if I want to demote debug printing to the logger.
  - orf 11y ago
    I've heard this one many times, but the suggestion is as brainless as it sounds. I'll be surprised if you get anything resembling a decent reason.
gnrme 11y ago
I think this is the primary reason why some scripting languages end up in the education space (as a tool for learning), while others go mainstream and ubiquitous in the commercial space. Breaking stuff between versions is a headache and expense for everyone except the most superficial users. The 'there should be one – and preferably only one – obvious way to do it' rule sounds like another reason. It's like being asked to choose between a perfect general use knife or a Swiss army knife.
- gipp 11y ago
  This would _almost_ be a reasonable post if Python weren't mainstream and ubiquitous in the commercial space.
- icebraining 11y ago
  The 'there should be one – and preferably only one – obvious way to do it' rule sounds like another reason. It's like being asked to choose between a perfect general use knife or a Swiss army knife. You're missing the obvious word. It's not about removing options, it's about having good and clear defaults.
- deadowl 11y ago
  If only they did a release where that was the only change that broke backwards-compatibility with Python 2.
rkrzr 11y ago
IMO the biggest reason to use Python3 is its concurrency support via async + await. Fixing the unicode mess is nice too of course, but you can get most of the benefits in Python2 as well, by simply putting this at the top of all of your source files: from __future__ import unicode_literals Also make sure to decode all data from the outside as early as possible and only encode it again when it goes back to disk or the network etc.
- tudborg 11y ago
  So much this! asyncio was the main selling point for me, but in general, why not follow the language? I never really understood the "rather stay with py2.7" thing. I get it with big old monolithic applications. You don't "just" rewrite those, but _every new python project_ should be done with the latest stable release. Is anyone starting their PHP projects on PHP4? Any new node projects in 0.10? Of course not, that would be moronic.
  - netheril96 11y ago
    Because you don't know what libraries you may depend on in the future when you start a new project. I was a fervent Python 3 supporter, and wrote every of my fresh projects in Python 3 instead of 2, until one day I found I needed LLVM in my project, yet the python port at that time was for 2 only. I mostly avoid writing Python 3 nowadays, because I don't want to rewrite my project or find painstakingly an alternative solution when I could have just imported a module that runs fine under Python 2.
- ak217 11y ago
  For the record, you do not get most of the benefits by doing the future import. The silent Unicode/bytes coercion and lack of standard library support is still there, making it almost impossible to write correct non-ascii-handling software in Python 2.
BuckRogers 11y ago
I chose to port from CPython2 to PyPy4, rather than to CPython3. It just made more sense. I for one see no value in Python3 (unicode has been supported since 2.6). My reasons for migrating to PyPy4 instead of Python3- 1) It was easier than porting to CP3. 2) It gave me a tangible benefit by removing all CPU performance worries once and for all. Added "performance" as a feature for Python. Worth the testing involved. 3) It removed the GIL. If you use PyPy4 STM, which is currently a separate JIT. Which will be at some point merged back into PyPy4. So for me, Python3 can't possibly compete, and likely never will with PyPy4 once you consider the performance and existing code that runs with it. PyPy3 is old, beta, not production-ready, based on 3.2 and Py3 is moving so fast I don't think PyPy3 would be able to keep up if they tried. Python3 is dead to me. There's not enough value for a new language. I'm not worried about library support because Py2 is still bigger than 3 and 2.7 will be supported by 3rd party libraries for a very long time else choose irrelevance (Python3 was released in 2008, and still struggling to justify its existence...). My views on the language changes themselves are stated much better by Mark Lutz[0]. I'm more likely to leave Python entirely for a new platform than I am to migrate to Python3. PyPy is the future of Python. If the PyPy team announces within the next 5 years they're taking the mantle of Python2, that would be the nail in the coffin. All they have to do is sit back and backport whatever features the Python2/PyPy4 community wants into PyPy4 from CPython3 as those guys run off with their experiments bloating their language. I believe it's all desperation, throwing any feature against the wall. Yet doing irreparable harm bloating the language, making the famous "beginner friendly" language the exact opposite. I already consider myself a PyPy4 programmer, so I hope they make it an official language to match the implementation. There's also Pyston to keep an eye on which is also effectively 2.x only at this time. [0]http://learning-python.com/books/python-changes-2014-plus.html http://learning-python.com/books/python-changes-2014-plus.ht...
- cname 11y ago
  There's just a bit of hyperbole in your comment. Most major libraries have been ported to Python 3. I wonder if the opposite of what you're saying will happen--i.e, the libraries that don't support Python 3 will be left behind. Fabric is an example of that for me.
cft 11y ago
We migrated to Go from Python 2, since instead of incompatible Python 3 we needed faster Python 2 replacement.
PythonicAlpha 11y ago
The reason, I still did not port to Python 3: (and yes, Unicode in Py2 is a mess ...) They just broke to many things (unnecessarily!) internally. Particularly they changed many C APIs for enhancement modules, so that all of them had to be ported, before they could be used with Python 3. They did not even consider a portability layer ... why not?? Some (not all) of the bad decisions (like the u"..." strings) they did change afterwards, but than it was a little late. So many modules are still not ported to Python 3 -- so the hurdle is a little to high -- for small to nil benefits! So, the problem (from my side) is not Unicode at all ... just the lack of reasonable support from the deciders side. --- Maybe, some time later, when I have to much spare time.
- stillsut 11y ago
  Agreed. Also this: "So expect Python 4 to not do anything more drastic than to remove maybe deprecated modules from the standard library." But why break all the existing libraries that use those modules, even if there's now "better" ways. In every comparison I've ever seen on performance, robustness, etc python always loses to the other big languages. Except in one area: the availability of user-land submitted packages and extensions. So why break them for a little perf boost?
  - PythonicAlpha 11y ago
    Sounds really a bit weird to me. I hope, that they really think hard, which modules really must be removed. I would only consider security reasons to really remove a library module. Or at least mark it deprecated for a long while until it is used only really rarely.
tzs 11y ago
In the Reddit discussion of this, someone linked to this criticism [1] of Python 3's Unicode handling written by Armin Ronacher, author of the Flask framework. I am not competent to say whether this is spot on or rubbish or somewhere in between [2], but it seemed interesting at least. [1] http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ [2] Almost all of my Python 2 experience is in homework assignments in MOOCs for problems where there was no need to care about whether strings were ASCII, UTF-8, binary, or something else. My Python 3 experience is a handful of small scripts in environments where everything was ASCII.
- lmm 11y ago
  It's wrong. The whole point of cat is to concatenate files. But if you concatenate two files with different encodings, you end up with an unreadable file. So you want cat to error out if one of the files that was passed has a different encoding from the encoding you told it to use, which is exactly what the python cat will do.
  - jlg23 11y ago
    > The whole point of cat is to concatenate files. Yes. > So you want cat to error out if one of the files that was passed has a different encoding. No. I expect it to read bits from stream A until it is exhausted and then read bits from stream B until it is exhausted. All the time just writing the ones and zeros read to the output stream (of bits). And no, a byte does not have to be "8 bit" (http://www.lispworks.com/documentation/HyperSpec/Body/f_by_by.htm http://www.lispworks.com/documentation/HyperSpec/Body/f_by_b...).
    - cosarara97 11y ago
      Do you actually work with systems not using 8 bit bytes or are you just being pedantic?
      - jlg23 11y ago
        Yes, I do. But the point I was trying to make was "it's bits, not bytes" (because "byte" has been defined to mean 8 bits only recently).
henrik_w 11y ago
This is a pretty good explanation of unicode in Python: http://nedbatchelder.com/text/unipain.html http://nedbatchelder.com/text/unipain.html
Animats 11y ago
Unicode worked just fine by Python 2.6. I had a whole system with a web crawler and HTML parsers which did everything in Unicode internally. You had to use "unicode()" instead of "str()" in many places, but that wasn't a serious problem. By Python 2.7, there were types "unicode", "str", and "bytes". That made sense. "str" and "bytes" were still the same thing, for backwards compatibility, but it was clear where things were going. The next step seemed to be a hard break between "str" and "bytes", where "str" would be limited to 0..127 ASCII values. Binary I/O would then return "bytes", which could be decoded into "unicode" or "str" when required. So there was a clear migration path forward. Python 3 dumped in a whole bunch of incompatible changes that had nothing to do with Unicode, which is why there's still more Python 2 running than Python 3. It was Python's Perl 6 moment. From the article: "Obviously it will take decades to see if Python 3 code in the world outstrips Python 2 code in terms of lines of code." Right. Seven years in, Python 2.x still has far more use than Python 3. About a year ago, I converted a moderately large system from Python 2 to Python 3, and it took about a month of pain. Not because of the language changes, but because the third-party packages for Python 3 were so buggy. I should not have been the one to discover that the Python connector for MySQL/MariaDB could not do a "LOAD DATA LOCAL" of a large data set. Clearly, no one had ever used that code in production. One of the big problems with Python and its developers is that the core developers take the position that the quality of third party packages is someone else's problem. Python doesn't even have a third party package repository - PyPI is a link farm of links to packages elsewhere. You can't file a bug report or submit a patch through it. Perl's CPAN is a repository with quality control, bug reporting, and Q/A. Go has good libraries for most server-side tasks, mostly written at Google or used at Google, so you know they've been exercised on lots of data. That "build it and they will convert" attitude and the growth of alternatives to Python is what killed Python 3.
- pwang 11y ago
  > That "build it and they will convert" attitude and the growth of alternatives to Python is what killed Python 3. Well said.
collinmanderson 11y ago
It seems to me if bytes/unicode was the only breaking change we would probably be over the transition by now. There are a lot of other subtle changes that makes the transition harder: comparison changes and keys() being an iterator for example. These are good long term changes, but I wish they weren't bundled in with the bytes/unicode changes.
rdslw 11y ago
I love when people with native english skills write monsters like this: "If people's hopes of coding bug-free code in Python 2 actually panned out then I wouldn't consistently hear from basically every person who ports their project to Python 3 that they found latent bugs in their code regarding encoding and decoding of text and binary data." This should be under penalty ;) Anyone to divide it into few simpler sentences? UPDATE: And another one from our connected sentences loving author: "We assumed that more code would be written in Python 3 than in Python 2 over a long-enough time frame assuming we didn't botch Python 3 as it would last longer than Python 2 and be used more once Python 2.7 was only used for legacy projects and not new ones."
- criley2 11y ago
  Division of this sentence would reduce it's legibility. It's a simple IF <condition> THEN <result>. You can argue that he's overly verbose, but breaking the IF/THEN into multiple sentences reduces their connection and the ability to understand. "If we were actually creating bug free code in Python 2 then porting code to Python 3 would be seamless, which it is not".
- nemmons 11y ago
  Maybe something like: "I consistently hear from basically every person who ports their project to Python 3 that they found latent bugs in their code (regarding encoding/decoding of text and binary data). This would not be the case if people's hopes of coding bug free code had actually panned out." That's a little better, but still not great.
- smegel 11y ago
  If people's hopes of coding bug-free code in Python 2 actually panned out then they would have bug free code thus when they port their code to python 3, the unicode changes would not reveal latent (existing) bugs thus they would not blog about or tell people about said bugs, hence the author of that sentence would not have heard about such bugs moral of the story: Python 2 hides certain kind of unicode related bugs that are not exposed until you port to Python 3.
- teek 11y ago
niels_olson 11y ago
How long is the transition going to take? Serious question. Because I'm rather tired of starting new work and finding some module that drags me back to 2.x.
- ubernostrum 11y ago
  Technically, "forever", since there will be people who never port their code. If you're depending on one of those holdouts, it's time to find a new dependency, because if they haven't ported by now they won't and that's your problem because... in practical terms, the transition is about to be over, since now the Linux distros are all-in on converting to Python 3 for their current or next releases and that will forcibly move the unported libraries in the bin of obsolescence.
diimdeep 11y ago
Str is tip of the iceberg. Python before 2.7 and current Python is completely different language semantically; methods, functions, statements, expressions, Global interpreter lock behavior.. This is sad that this blog post and discussions around it didn't mention anything about it.
- diimdeep 11y ago
  IMHO blog post don't really answer it's own title.
- rcthompson 11y ago
  The article isn't covering all the differences between Python 2 and Python 3. Based on this article as well as other articles I've read in the past, the Unicode issue was the original reason they decided it was necessary to break backward-compatibility, but once that decision was made, there was no reason not to make any further backward-incompatible improvements.
euske 11y ago
I like Python3 personally. It's new and better but a different branch. I'm annoyed by people abbreviating it as "Python" and treating it as a substitute for Python2. In my opinion, the "Python" name should be exclusively used for Python2, and Python3 should've been always used as one word. The whole Python3 situation caused unnecessary confusion to the outside (non-Python) people, which I think could be avoided.
makecheck 11y ago
Since I'm trying to keep a small footprint, I rely on the system version of Python on Mac OS X, which is 2.7.10 now. To use anything newer, I'd have to ask users to install a different interpreter, or bundle a particular version that adds bloat. There's no point. The most I've done is to import a few things from __future__; otherwise, my interest in Python 3 begins when Apple installs it.
echlebek 11y ago
The Go authors have solved this problem thoroughly. When working in Go, I usually never have to think about this. https://blog.golang.org/strings https://blog.golang.org/strings
- eugenekolo2 11y ago
  Go came out in 2009. What's your point? I'd sure hope they'd look at languages older than them.
nulltype 11y ago
So Python 2 did not have super obvious string handling. One of the odd things that they seemingly could have fixed pretty easily is to change the default encoding from 'ascii' to 'utf8'. That would have fixed a bunch of the UnicodeDecodeErrors that were the most obvious problem with strings: http://www.ianbicking.org/illusive-setdefaultencoding.html http://www.ianbicking.org/illusive-setdefaultencoding.html If they had to make Python 3 anyway, I think the main thing they were missing is that they should have added a JIT. That makes upgrading to Python 3 a much easier argument. If the only point of the JIT was to add a selling point to Python 3, that probably would have been worth it.

418 points by cocoflunchy 11y ago

Since python3 is not backwards compatible with python2, why didn't the python devs leverage the opportunity for creating a more performant non-GIL runtime for python3?

chrisseaton 11y ago
> more performant non-GIL runtime for python3 Because making one of those those is a huge amount of work, may introduce more serious backwards incompatibilities (like C extensions), and not everyone has the knowledge of how to do it so they'd need to either learn from scratch themselves or find people interested in doing it for them.
svisser 11y ago
Removing the GIL is a difficult problem: http://python-notes.curiousefficiency.org/en/latest/python3/multicore_python.html http://python-notes.curiousefficiency.org/en/latest/python3/...
- criddell 11y ago
  I think the question was if you aren't going to be backwards compatible, why not unshackle yourself completely and design a new language without the GIL and make it as pythonic as possible? A scripting language that support multi-threading is possible, right? I think TCL does it.
  - chrisseaton 11y ago
    I guess they were happy with breaking backwards compatibility a little bit, not not as much as simply removing the GIL and not adding back any implicit synchronisation at all.
  - mkesper 11y ago
    I think this question can only be raised in hindsight. I think nobody suspected the unicode transition to be so slow and painful. As there probably won't be such a big transition evermore, I wouldn't hold my breath for it happening.
js8 11y ago
Is there a way to do it? My understanding always was that removing GIL means that the code that runs single-threaded (which is like 95% of Python code out there) will get a performance hit.
- andreasvc 11y ago
  Yes it does mean that and I believe it is unavoidable, because you need to replace the coarse-grained GIL with a fine-grained locking instructions. However, I think the trade off is now worth it, since multi-core CPUs are ubiquitous. Besides, Python itself was never notable for its performance to begin with, so I've always found the argument very odd.
  - js8 11y ago
    It is perfectly avoided by not removing the GIL. :-) Multi-core CPUs may be ubiquitous, but multi-threaded Python code is not, by far. So performance of say, my Python code, would decrease, thank you, I don't need that. It's not that Python would have superb performance, I just don't see the reason to pay for something that's not actually needed. It's also often forgotten that one of the easiest paths to parallelism is to just run multiple processes (and for many problems you only need to split the input data, the processes don't need to even communicate), and this solution naturally uses multiple CPUs without any GIL worries. I think the actual use cases that require GIL removal are very little (something like a server, perhaps), and if you actually need to do that, I think you're better served with some JVM language or Go.
    - andreasvc 11y ago
      What I'm saying is that not only does Python not have superb performance, its performance is actually largely irrelevant. The advantage of Python is in ease of use, capabilities as glue language, readability, etc. So that's why I really don't see why you wouldn't remove the GIL at a reasonable slowdown factor (2x was a figure thrown around IIRC). People always proclaim that you can just use multiple processes for parallelism. It's nice when it suits your (embarrassingly parallel) problem but when you need to share a large amount of data between processes it's a major hassle.
eric_bullington 11y ago
Removing the GIL is non-trivial. But I'm surprised that they didn't take advantage of Python 3 to move to a more performant register-based interpreter. Several people expressed interest in building one, and working prototypes were even made, but the core developers didn't seem much interested in performance upgrades. I think their attitude is generally, "if it's performance-critical, write it in C", which was a good approach 15 years ago. However, Python's competition now comes from languages like Go and Rust that have both good performance and are user-friendly and productive (with features such as expressive syntax and a comprehensive standard library).
lmm 11y ago
Because the runtime with the GIL is more performant and python users know that it's practically impossible to write correct software using threads.
- scardine 11y ago
  Yes, this is the point. The GIL matters only if you are using threads with CPU-bound tasks (most of my work is I/O bound). When I have CPU-bound problems that need parallelism and are not yet well implemented in numpy/scipy, I would ratter use AMQP instead of threads.
morebetterer 11y ago
It's a shame the GIL won't be removed because it's perceived to be too difficult. It's trivial to remove the GIL - probably a week of mechanical work. Just don't depend on global variables in the interpreter. No global state, no problem. But the Python C API has to be changed to always store interpreter state into a struct, and a pointer to that interpreter state struct has to be passed as the first argument in all C API calls. Not rocket science; this is what Lua does. It's a political decision to keep the GIL, not a technical one. As for preserving C API backwards compatibility, it's a straw man argument - the Python API broke from 2.x to 3.x anyway. There's no such thing as "lesser breakage" - only breakage.
- laurencerowe 11y ago
  Supporting multiple interpreters in a single process is not what most people mean when they talk about removing the GIL. Objects from one interpreter could not be used safely from another interpreter. This would be handy in a few situations but is essentially not all that different to multiprocessing. What people usually mean when they talk about removing the GIL is having multi-threaded code make use of multiple cpu cores (as it does in Jython.) This would involved splitting the GIL into more fine-grained locks. Unfortunately experiments taking this approach have so far shown significant performance impacts for single threaded code.
  - morebetterer 11y ago
    It would be inefficient to share data structures or interpreter internal state across threads. This would lead to the poor performance you described with mutex contention. It is better to have an interpreter instance per thread, each having their own separate variable pool, and pass messages between the different interpreter instances. This model scales well with multicore systems and is faster than the multi-process equivalent. It's also a simpler model to implement and maintain. Python's current multithreading model is too complicated from an implementation point of view.
BuckRogers 11y ago
Because that's difficult to do. But the PyPy team has already done it for Python2 code. Check into PyPy-STM.

danso 11y ago

> We have decided as a team that a change as big as unicode/str/bytes will never happen so abruptly again. When we started Python 3 we thought/hoped that the community would do what Python did and do one last feature release supporting Python 2 and then cut over to Python 3 development for feature development while doing bugfix releases only for the Python 2 version. I'm guessing it's not a coincidence that string encoding was also behind the Great Sadness of Moving From Ruby 1.8 to 1.9. How have other mainstream languages made this jump, if it was needed, and were they able to do it in a non-breaking way? https://news.ycombinator.com/item?id=1162122 https://news.ycombinator.com/item?id=1162122

perlgeek 11y ago
Well, Perl 6 also introduces a distinction between strings (Unicode) und buffers (octets), but it also introduces loads of other changes, to the point where it's its own language. That said, the Unicode features were a major thing planned for PHP 6, and (afaict) one of the reasons there has never been a PHP 6, but rather they went straight from 5 to 7. I'm not aware of any graceful transitions.
bdarnell 11y ago
C and C++ are so widely used that transitions like this are made not at the language level but at the level of platforms or other communities. Some parts of the C/C++ world made this transition relatively seamlessly, while others got caught in the same traps as Python. The key is UTF-8: UTF-8 is a superset of 7-bit ASCII, so as long as you only convert to/from other encodings at the boundaries of your system, unicode can be introduced to the internal components in a gradual and mostly-compatible way. You only get in trouble when you decide that you need separate "byte string" and "character string" data types (which is generally a mistake: due to the existince of combining characters, graphemes are variable-width even if you're using strings composed of unicode code points, so you don't gain much by using UCS-4 character strings instead of UTF-8 byte strings). My theory is that the python 3 transition would have gone much smoother and still accomplished its goals if they had left the implicit str/bytes conversion in place but just made it use UTF-8 instead of ASCII (although in environments like Windows where UTF-16 is important this may not have worked out as well).

andrewstuart 11y ago

IMO one of the reasons for all the angst is that .encode() and .decode() are so ambiguous and unintuitive which makes them incredibly confusing to use. Which direction are you converting? From what to what? The whole unicode thing is hard enough to understand without Python's encoding and decoding functions adding to the mystery. I still have to refer to the documentation to make sure I'm encoding or decoding as expected. I think there would have been much less of a problem if encode and decode were far more obvious, unambiguous and intuitive to use. Probably without there being two functions. Still a problem of course today.

cjauvin 11y ago
I know it's a deeply subjective and personal thing unlikely to help anyone other than me, but I've always successfully relied on the following mnemonic for this particular problem: * unicode -> encode, u -> e (vowels) * str -> decode, s -> d (consonants)
- deleted 11y ago
  [deleted]
- andrewstuart 11y ago
  There should be one obvious way to do it.
  - davidism 11y ago
    > Although that way may not be obvious at first unless you're Dutch. There is one obvious way: strings are encoded to bytes, bytes are decoded to strings.
    - jes5199 11y ago
      I think that's backwards
      - rspeer 11y ago

onesixtythree 11y ago

From the outside, Python 3 seems like a much better language. I don't have strong views of its object system (I avoid OOP as much as I can) but it seems like the string/bytes handling is much better, and I'm also a fan of map and filter returning generators rather than being hard-coded to a list implementation (stream fusion is a good thing). Also, I fail to see any value in print being a special keyword instead of a regular function (as in 3). What I don't get is: why has Python 3 adoption been so slow? Is it just backward compatibility, or are there deeper problems with it that I'm not aware of?

harryf 11y ago
What makes me snarky is the replacing of '%s %s' % ('one', 'two') With '%s %s'.format('one', 'two') The latter is just more annoying to type. Stupid argument I know but I find myself grumbling to myself every time...
- davidism 11y ago
  It's not replaced, both are perfectly valid. And this addition also exists in Python 2.
- RussianCow 11y ago
  The second example should be: '{} {}'.format('one', 'two') Either way, the former example still works in Python 3.5, so that syntax hasn't gone away. `format` is preferred, though. This Stack Overflow question has some good answers as to why: http://stackoverflow.com/questions/5082452/python-string-formatting-vs-format http://stackoverflow.com/questions/5082452/python-string-for...
- dietrichepp 11y ago
  You mean, '{} {}'.format('one', 'two') Your experience may vary, but in my experience, when I switched to .format(), I found a number of bugs in code that used % instead. As mentioned, you can continue using %.
  - 11y ago

mathgenius 11y ago

Ok, fine. Can we have the print statement back?

untothebreach 11y ago
Genuinely curious, why do you prefer `print` to be a statement rather than a function? I've heard a lot of criticisms of Py3, but this is the first time I've heard this one.
- bufordsharkley 11y ago
  I think a function is "better", but the print statement should have been preserved. Print is used either for: 1) Writing to stdout/stderr 2) Debugging the hacky way The print function is better for the former (though more often than not, I use Armin Ronacher's click.echo for compability)[0], but I fastly prefer the print statement for (2). Not dealing with parentheses is always a plus; I can add and remove print statements far more quickly. I find it to be an increase in friction, and I don't see any real downside in leaving it in. [0] http://click.pocoo.org/5/api/#click.echo http://click.pocoo.org/5/api/#click.echo
  - untothebreach 11y ago
    I agree with you on 1), I much prefer `print(..., file=sys.stderr)` to `print >>sys.stderr, ...`. For 2), though, I have always used auto-inserting parens in my vim, so I never really experienced any pain w.r.t. parentheses. I can see how that would be a pain, though.
  - Retra 11y ago
    I prefer the print function for debuging, because I can just do s/print/log.debug if I want to demote debug printing to the logger.
- orf 11y ago
  I've heard this one many times, but the suggestion is as brainless as it sounds. I'll be surprised if you get anything resembling a decent reason.

gnrme 11y ago

I think this is the primary reason why some scripting languages end up in the education space (as a tool for learning), while others go mainstream and ubiquitous in the commercial space. Breaking stuff between versions is a headache and expense for everyone except the most superficial users. The 'there should be one – and preferably only one – obvious way to do it' rule sounds like another reason. It's like being asked to choose between a perfect general use knife or a Swiss army knife.

gipp 11y ago
This would _almost_ be a reasonable post if Python weren't mainstream and ubiquitous in the commercial space.
icebraining 11y ago
The 'there should be one – and preferably only one – obvious way to do it' rule sounds like another reason. It's like being asked to choose between a perfect general use knife or a Swiss army knife. You're missing the obvious word. It's not about removing options, it's about having good and clear defaults.
deadowl 11y ago
If only they did a release where that was the only change that broke backwards-compatibility with Python 2.

rkrzr 11y ago

IMO the biggest reason to use Python3 is its concurrency support via async + await. Fixing the unicode mess is nice too of course, but you can get most of the benefits in Python2 as well, by simply putting this at the top of all of your source files: from __future__ import unicode_literals Also make sure to decode all data from the outside as early as possible and only encode it again when it goes back to disk or the network etc.

tudborg 11y ago
So much this! asyncio was the main selling point for me, but in general, why not follow the language? I never really understood the "rather stay with py2.7" thing. I get it with big old monolithic applications. You don't "just" rewrite those, but _every new python project_ should be done with the latest stable release. Is anyone starting their PHP projects on PHP4? Any new node projects in 0.10? Of course not, that would be moronic.
- netheril96 11y ago
  Because you don't know what libraries you may depend on in the future when you start a new project. I was a fervent Python 3 supporter, and wrote every of my fresh projects in Python 3 instead of 2, until one day I found I needed LLVM in my project, yet the python port at that time was for 2 only. I mostly avoid writing Python 3 nowadays, because I don't want to rewrite my project or find painstakingly an alternative solution when I could have just imported a module that runs fine under Python 2.
ak217 11y ago
For the record, you do not get most of the benefits by doing the future import. The silent Unicode/bytes coercion and lack of standard library support is still there, making it almost impossible to write correct non-ascii-handling software in Python 2.

BuckRogers 11y ago

I chose to port from CPython2 to PyPy4, rather than to CPython3. It just made more sense. I for one see no value in Python3 (unicode has been supported since 2.6). My reasons for migrating to PyPy4 instead of Python3- 1) It was easier than porting to CP3. 2) It gave me a tangible benefit by removing all CPU performance worries once and for all. Added "performance" as a feature for Python. Worth the testing involved. 3) It removed the GIL. If you use PyPy4 STM, which is currently a separate JIT. Which will be at some point merged back into PyPy4. So for me, Python3 can't possibly compete, and likely never will with PyPy4 once you consider the performance and existing code that runs with it. PyPy3 is old, beta, not production-ready, based on 3.2 and Py3 is moving so fast I don't think PyPy3 would be able to keep up if they tried. Python3 is dead to me. There's not enough value for a new language. I'm not worried about library support because Py2 is still bigger than 3 and 2.7 will be supported by 3rd party libraries for a very long time else choose irrelevance (Python3 was released in 2008, and still struggling to justify its existence...). My views on the language changes themselves are stated much better by Mark Lutz[0]. I'm more likely to leave Python entirely for a new platform than I am to migrate to Python3. PyPy is the future of Python. If the PyPy team announces within the next 5 years they're taking the mantle of Python2, that would be the nail in the coffin. All they have to do is sit back and backport whatever features the Python2/PyPy4 community wants into PyPy4 from CPython3 as those guys run off with their experiments bloating their language. I believe it's all desperation, throwing any feature against the wall. Yet doing irreparable harm bloating the language, making the famous "beginner friendly" language the exact opposite. I already consider myself a PyPy4 programmer, so I hope they make it an official language to match the implementation. There's also Pyston to keep an eye on which is also effectively 2.x only at this time. [0]http://learning-python.com/books/python-changes-2014-plus.html http://learning-python.com/books/python-changes-2014-plus.ht...

cname 11y ago
There's just a bit of hyperbole in your comment. Most major libraries have been ported to Python 3. I wonder if the opposite of what you're saying will happen--i.e, the libraries that don't support Python 3 will be left behind. Fabric is an example of that for me.

cft 11y ago

We migrated to Go from Python 2, since instead of incompatible Python 3 we needed faster Python 2 replacement.

PythonicAlpha 11y ago

The reason, I still did not port to Python 3: (and yes, Unicode in Py2 is a mess ...) They just broke to many things (unnecessarily!) internally. Particularly they changed many C APIs for enhancement modules, so that all of them had to be ported, before they could be used with Python 3. They did not even consider a portability layer ... why not?? Some (not all) of the bad decisions (like the u"..." strings) they did change afterwards, but than it was a little late. So many modules are still not ported to Python 3 -- so the hurdle is a little to high -- for small to nil benefits! So, the problem (from my side) is not Unicode at all ... just the lack of reasonable support from the deciders side. --- Maybe, some time later, when I have to much spare time.

stillsut 11y ago
Agreed. Also this: "So expect Python 4 to not do anything more drastic than to remove maybe deprecated modules from the standard library." But why break all the existing libraries that use those modules, even if there's now "better" ways. In every comparison I've ever seen on performance, robustness, etc python always loses to the other big languages. Except in one area: the availability of user-land submitted packages and extensions. So why break them for a little perf boost?
- PythonicAlpha 11y ago
  Sounds really a bit weird to me. I hope, that they really think hard, which modules really must be removed. I would only consider security reasons to really remove a library module. Or at least mark it deprecated for a long while until it is used only really rarely.

tzs 11y ago

In the Reddit discussion of this, someone linked to this criticism [1] of Python 3's Unicode handling written by Armin Ronacher, author of the Flask framework. I am not competent to say whether this is spot on or rubbish or somewhere in between [2], but it seemed interesting at least. [1] http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ [2] Almost all of my Python 2 experience is in homework assignments in MOOCs for problems where there was no need to care about whether strings were ASCII, UTF-8, binary, or something else. My Python 3 experience is a handful of small scripts in environments where everything was ASCII.

lmm 11y ago
It's wrong. The whole point of cat is to concatenate files. But if you concatenate two files with different encodings, you end up with an unreadable file. So you want cat to error out if one of the files that was passed has a different encoding from the encoding you told it to use, which is exactly what the python cat will do.
- jlg23 11y ago
  > The whole point of cat is to concatenate files. Yes. > So you want cat to error out if one of the files that was passed has a different encoding. No. I expect it to read bits from stream A until it is exhausted and then read bits from stream B until it is exhausted. All the time just writing the ones and zeros read to the output stream (of bits). And no, a byte does not have to be "8 bit" (http://www.lispworks.com/documentation/HyperSpec/Body/f_by_by.htm http://www.lispworks.com/documentation/HyperSpec/Body/f_by_b...).
  - cosarara97 11y ago
    Do you actually work with systems not using 8 bit bytes or are you just being pedantic?
    - jlg23 11y ago
      Yes, I do. But the point I was trying to make was "it's bits, not bytes" (because "byte" has been defined to mean 8 bits only recently).

henrik_w 11y ago

This is a pretty good explanation of unicode in Python: http://nedbatchelder.com/text/unipain.html http://nedbatchelder.com/text/unipain.html

Animats 11y ago

Unicode worked just fine by Python 2.6. I had a whole system with a web crawler and HTML parsers which did everything in Unicode internally. You had to use "unicode()" instead of "str()" in many places, but that wasn't a serious problem. By Python 2.7, there were types "unicode", "str", and "bytes". That made sense. "str" and "bytes" were still the same thing, for backwards compatibility, but it was clear where things were going. The next step seemed to be a hard break between "str" and "bytes", where "str" would be limited to 0..127 ASCII values. Binary I/O would then return "bytes", which could be decoded into "unicode" or "str" when required. So there was a clear migration path forward. Python 3 dumped in a whole bunch of incompatible changes that had nothing to do with Unicode, which is why there's still more Python 2 running than Python 3. It was Python's Perl 6 moment. From the article: "Obviously it will take decades to see if Python 3 code in the world outstrips Python 2 code in terms of lines of code." Right. Seven years in, Python 2.x still has far more use than Python 3. About a year ago, I converted a moderately large system from Python 2 to Python 3, and it took about a month of pain. Not because of the language changes, but because the third-party packages for Python 3 were so buggy. I should not have been the one to discover that the Python connector for MySQL/MariaDB could not do a "LOAD DATA LOCAL" of a large data set. Clearly, no one had ever used that code in production. One of the big problems with Python and its developers is that the core developers take the position that the quality of third party packages is someone else's problem. Python doesn't even have a third party package repository - PyPI is a link farm of links to packages elsewhere. You can't file a bug report or submit a patch through it. Perl's CPAN is a repository with quality control, bug reporting, and Q/A. Go has good libraries for most server-side tasks, mostly written at Google or used at Google, so you know they've been exercised on lots of data. That "build it and they will convert" attitude and the growth of alternatives to Python is what killed Python 3.

pwang 11y ago
> That "build it and they will convert" attitude and the growth of alternatives to Python is what killed Python 3. Well said.

collinmanderson 11y ago

It seems to me if bytes/unicode was the only breaking change we would probably be over the transition by now. There are a lot of other subtle changes that makes the transition harder: comparison changes and keys() being an iterator for example. These are good long term changes, but I wish they weren't bundled in with the bytes/unicode changes.

rdslw 11y ago

I love when people with native english skills write monsters like this: "If people's hopes of coding bug-free code in Python 2 actually panned out then I wouldn't consistently hear from basically every person who ports their project to Python 3 that they found latent bugs in their code regarding encoding and decoding of text and binary data." This should be under penalty ;) Anyone to divide it into few simpler sentences? UPDATE: And another one from our connected sentences loving author: "We assumed that more code would be written in Python 3 than in Python 2 over a long-enough time frame assuming we didn't botch Python 3 as it would last longer than Python 2 and be used more once Python 2.7 was only used for legacy projects and not new ones."

criley2 11y ago
Division of this sentence would reduce it's legibility. It's a simple IF <condition> THEN <result>. You can argue that he's overly verbose, but breaking the IF/THEN into multiple sentences reduces their connection and the ability to understand. "If we were actually creating bug free code in Python 2 then porting code to Python 3 would be seamless, which it is not".
nemmons 11y ago
Maybe something like: "I consistently hear from basically every person who ports their project to Python 3 that they found latent bugs in their code (regarding encoding/decoding of text and binary data). This would not be the case if people's hopes of coding bug free code had actually panned out." That's a little better, but still not great.
smegel 11y ago
If people's hopes of coding bug-free code in Python 2 actually panned out then they would have bug free code thus when they port their code to python 3, the unicode changes would not reveal latent (existing) bugs thus they would not blog about or tell people about said bugs, hence the author of that sentence would not have heard about such bugs moral of the story: Python 2 hides certain kind of unicode related bugs that are not exposed until you port to Python 3.
teek 11y ago

niels_olson 11y ago

How long is the transition going to take? Serious question. Because I'm rather tired of starting new work and finding some module that drags me back to 2.x.

ubernostrum 11y ago
Technically, "forever", since there will be people who never port their code. If you're depending on one of those holdouts, it's time to find a new dependency, because if they haven't ported by now they won't and that's your problem because... in practical terms, the transition is about to be over, since now the Linux distros are all-in on converting to Python 3 for their current or next releases and that will forcibly move the unported libraries in the bin of obsolescence.

diimdeep 11y ago

Str is tip of the iceberg. Python before 2.7 and current Python is completely different language semantically; methods, functions, statements, expressions, Global interpreter lock behavior.. This is sad that this blog post and discussions around it didn't mention anything about it.

diimdeep 11y ago
IMHO blog post don't really answer it's own title.
rcthompson 11y ago
The article isn't covering all the differences between Python 2 and Python 3. Based on this article as well as other articles I've read in the past, the Unicode issue was the original reason they decided it was necessary to break backward-compatibility, but once that decision was made, there was no reason not to make any further backward-incompatible improvements.

euske 11y ago

I like Python3 personally. It's new and better but a different branch. I'm annoyed by people abbreviating it as "Python" and treating it as a substitute for Python2. In my opinion, the "Python" name should be exclusively used for Python2, and Python3 should've been always used as one word. The whole Python3 situation caused unnecessary confusion to the outside (non-Python) people, which I think could be avoided.

makecheck 11y ago

Since I'm trying to keep a small footprint, I rely on the system version of Python on Mac OS X, which is 2.7.10 now. To use anything newer, I'd have to ask users to install a different interpreter, or bundle a particular version that adds bloat. There's no point. The most I've done is to import a few things from __future__; otherwise, my interest in Python 3 begins when Apple installs it.

echlebek 11y ago

The Go authors have solved this problem thoroughly. When working in Go, I usually never have to think about this. https://blog.golang.org/strings https://blog.golang.org/strings

eugenekolo2 11y ago
Go came out in 2009. What's your point? I'd sure hope they'd look at languages older than them.

nulltype 11y ago

So Python 2 did not have super obvious string handling. One of the odd things that they seemingly could have fixed pretty easily is to change the default encoding from 'ascii' to 'utf8'. That would have fixed a bunch of the UnicodeDecodeErrors that were the most obvious problem with strings: http://www.ianbicking.org/illusive-setdefaultencoding.html http://www.ianbicking.org/illusive-setdefaultencoding.html If they had to make Python 3 anyway, I think the main thing they were missing is that they should have added a JIT. That makes upgrading to Python 3 a much easier argument. If the only point of the JIT was to add a selling point to Python 3, that probably would have been worth it.