15 ms·

Python 3 Q&A by a prolific core developer

145 points by hynek 14y ago

toyg 14y ago
Unreadable on iPhone, what's up with its stylesheet? Safari won't even zoom...?
- hynek 14y ago
  Yeah, it seems to be the default RTFD style…try: http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-python-notes.readthedocs.org%2Fen%2Flatest%2Fpython3%2Fquestions_and_answers.html http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-...
deleted 14y ago
[deleted]
thu 14y ago
The following situation happens with other data than just unicode. This seems an argument for static typing in general. The reason this approach is problematic is that it means the traceback for an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x code base almost never points you to the code that is broken. Instead, you have to trace the origins of the data in the failing operation, and try to figure out where the unexpected 8-bit or Unicode code string was introduced. By contrast, Python 3 is designed to fail fast in most situations[...]
- qznc 14y ago
  I agree that static typing would helps for such bugs. However, not "in general". The general problem is that bogus data leads to an irreparable situation. To fix that bug, you have to find out who broke the data. Consider for example, a circle in a linked list. You would need quite an expressive type system to prevent that statically (dependent types might suffice? Theorem provers are the ultimate hammer).
  - zanny 14y ago
    Different implementations for different use cases in different situations :P
  - DannoHung 14y ago
    Uhh, how would you make a linked list with a circle in it with the Haskell List type?
    - dbaupp 14y ago
      a = 1:a
      - ufo 14y ago
        This trick is often called "Tying the Knot", if anyone is interested.
    - qznc 14y ago
      makeCircle rst = rst ++ makeCircle rst The thing is that a circular list is actually an infinite list in Haskell. ;)
      - ufo 14y ago
        This is not a real circular list though, since it does not tie the knot as in dbaupp's comment.
      - DannoHung 14y ago
        If you want to prevent that then you want to take general recursion out of your programming language and if you're gonna do that you might as well make it Turing incomplete. These aren't "circles" per se, they are partially evaluated recursive structures. If you expand them you end up with evaluated list structures that are non-self referential.
- gardentheory 14y ago
  Well the lighter solution and more Pythonic solution is strong typing. If a Unicode string and a byte string could not be combined then the problem would not exist AFAIK. For example "a" + 1 is an error, why is u"a" + "s" not an error?
  - xerula 14y ago
    But b"s" + "a" is an error in Py3.
  - qznc 14y ago
    This just gives a different error message. It does not help to find the error source. The programmer would probably just try to fix it via str(u"a")+"s" or u"a" + unicode("s").
  - masklinn 14y ago
    > For example "a" + 1 is an error, why is u"a" + "s" not an error? "a" + b"s" is an error in Python 3.
    - gardentheory 14y ago
      Sorry, that was my point. Strengthening the types in python3 solves the problem without static typing being needed at all.
      - masklinn 14y ago
        ok. Static types solve it earlier though, especially for little-exercised code paths which could be forgotten or missed in testing.
- slurgfest 14y ago
  It is an argument for failing fast in general. Static typing isn't guaranteed to pick up every mistake at its origin, because far from every mistake is of the "int rather than string" variety. Since you have brought up the general case, the origins of many bugs involve something sophisticated going wrong which would require you to encode much of your program (hopefully not too redundantly) into a Turing-complete type system. Rather than stuffing square pegs into round holes, you could just write the appropriate error handling code (or even just asserts).
- earl 14y ago
  Please: alias hncomment='fold -w 77 -s | sed "s/^/ /" | pbcopy' use: echo "blah blah blah" | hncomment command-v Thus: The reason this approach is problematic is that it means the traceback for an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x code base almost never points you to the code that is broken. Instead, you have to trace the origins of the data in the failing operation, and try to figure out where the unexpected 8-bit or Unicode code string was introduced. By contrast, Python 3 is designed to fail fast in most situations[...]
zanny 14y ago
I think all the outrage is that Python 3 behaves like a new language in how everything needs to be ported (albeit at a magnitude less complexity than going up the language chain or across the playing field, like C to Ruby or C# to Java) but it still incurs the new language cost, while being the new version of the old language. If people just thought of Python 3 as the replacement to broken text model Python, like how XHTML tried to supplant HTML and failed, and then HTML5 supplanted XHTML with some quirks, the same thing happens here. Also, I adore the comments about the GIL. I understand the benefits of using a single language in every circumstance, but (even) a JIT scripting language is not sufficient for all problems. I guess it depends on if a polyglot code base is more complex than handling multithreading and the corresponding memory complexity and gimmicks, and that is developer centric. Still, I would much rather just stick processor bound code in openMP C functions and call those from Python when need be. It seems like the "right" answer to the performance problem, without losing much productivity.
smeg 14y ago
Python3 and Perl6 can both fuck off. I look forward to the day PyPy is considered the real Python. Look at PyPy's homepage (http://pypy.org/ http://pypy.org/), doesn't even mention Unicode as a significant feature. Instead it talks about speed, security, concurrency, and compatibility with the current real Python (2.7.2) - all the things real Python programmers care about and expect the Python developers to focus on. PyPy may be some way off but I want to find the developers and hug them for setting the right vision and trying. I just donated $50 and if you hate Python3 you should too.
- sho_hn 14y ago
  Or if you really like Python 3 (as I do!) you can donate those $50 to Python 3 support in PyPy: http://pypy.org/py3donate.html http://pypy.org/py3donate.html
- Deestan 14y ago
  > all the things real Python programmers care about Guess I'm not a real programmer, then, since I find proper Unicode handling to be a welcome feature, and find the implementation in Python2 is painful.
- Marwy 14y ago
  I think you should take a walk, and then come back and read this ridiculous stuff you wrote.
- lclarkmichalek 14y ago
  Any reason they can "fuck off"? Anyway, you're incorrect about pypy. "Real" python developers will likely not run their code in a sandboxed environment, so PyPy's security advantages are minimal, added to the fact that your sandbox is only as good as its configuration. The concurrency is interesting, but there are numerous other solutions available, and that alone makes it unlikely that the de facto implementation of the language would include one. Claiming that PyPy is better on compatibility than CPython is ridiculous; last time I looked, anything using a C extension had to be rewritten to use ctypes. On speed, yes PyPy is impressive. But in python, anything that is particularly CPU intensive should probably be written as a C extension and PyPy takes a hit when using ctypes iirc. In general, I don't see the point of people trying to optimize generic python code. Most code never gets profiled, and most code isn't a problem. At least, not a problem enough to switch interpreter. Definitely not a problem enough to ignore the massive problem that is python 2.x's unicode implementation.
sutro 14y ago
Unconvinced.
ash 14y ago
Why can't I use Python 2 modules from Python 3 program? Is there any effort to make it possible?
- hynek 14y ago
  There is no such thing as "Python 2" or "Python 3" modules. There are just modules that will go belly up because they are incompatible with one or another. I presume you mean that you can’t access your old Python 2 modules from a new Python 3 installation. That’s just because Python installations usually don’t share their modules (i.e. site-packages). You can try to install them using Python 3 (usually just python3 setup.py install) and see if they work or not.
  - lrem 14y ago
    There's a deeper question here: why doesn't the interpreter contain a compatibility layer for the old code? There's no really good reason for the old modules to go belly up in the new interpreter. At least no other than the man months needed to implement that layer. But then, that probably would still be orders of magnitude less than migrating all the libraries that keep people from migrating...
    - hynek 14y ago
      That’s essentially what __future__ imports are for – things that go further would be simply too much work and Python 3 was about shedding weight, not shipping essentially two interpreters. (while they are very similar from outside, the innards couldn’t be more different)
      - lrem 14y ago
        Cool. So this way porting things to Py3k got easier. That perfectly explains why all the critical libs have been ported fast and no one remembers about Python 2 anymore, doesn't it? Shedding weight is sometimes a step in a good direction. But we need to draw a line at some point. As the history of Python 3 adoption shows, that point maybe was not optimal. Edit: forgot an angle: Also, __future__ is fixing the incompatibility of what people are using with something that they can't use. I'm not sure if it's the right problem to solve.
dmbaggett 14y ago
From the article: Back in reality, though, complaining about the GIL as though its a serious barrier to adoption amongst developers that know what they’re doing often says more about the person doing the complaining than it does about CPython. It was a solid, well-argued piece up to this point. You do yourself and the Python community a disservice by writing off your critics as ignorant. It sounds petulant and childish, and is wrong. There are valid arguments on both sides of the GIL argument, but neither side's advocates are ignorant or bad programmers.
- sho_hn 14y ago
  I have to agree here, despite the two of us rating the importance of the GIL problem quite differently in yesterday's discussion: A better strategy against the GIL FUD is plain old education, not trying to make it a taboo topic. Especially programmers who do know what they're doing won't take too kindly to a community where that becomes prevalent.
  - hynek 14y ago
    I find Nick arguing pretty much to the point and the quote above was ripped from it’s context which is a huge section balancing the arguments against each other. tldr He argues there are better ways for scaling out than threads and removing the GIL would have enormous consequence _throughout_ the whole code base so it’s removal cannot be warranted. Reducing 10 paragraphs and 11 bullet points to “only stupid people use threads” is just poor style.
- slurgfest 14y ago
  Let's be honest: you are upset because you have sometime committed to saying the GIL is a huge deal and you feel insulted by anyone saying it isn't, which is why you are using emotionally loaded words like 'petulant' and 'childish' and 'ignorant' and 'bad'. The GIL is the single biggest target for language-advocacy FUD against Python by advocates of other languages. I would be a rich person if I got a dollar for every time I saw someone trashing Python as a toy language because of the GIL, without significant knowledge of how to use Python. It's just a much huger issue to someone without real Python experience than it is to people with Python experience. Setting that aside, there are a vast number of use cases where someone might try to use threads when actually that's not a good solution. It actually isn't incredibly easy to come up with cases where the GIL is this huge fatal flaw. It's really sad how many times I have seen people ranting about the GIL and totally unaware of multiprocessing, unable to give a technical reason why they can't try processes, unaware of greenlets, wholly unaware that the GIL does not exist in Jython/IronPython, etc. I'm sure that doesn't apply to you, which would put you in the minority. Thus, "often says more about the person doing the complaining". Threads should be used judiciously, because they dramatically increase the complexity of a program and the difficulty of debugging and reasoning about it. Shared-everything is a great way to blow your foot off if you don't specifically need it. So if you throw threads at every problem indiscriminately, then that really is a weak point in your programming. (Note: I am not saying that using threads is always a bad idea) If you are a good programmer then you should already know, and not be offended to hear, that threads are a tool of specific applicability, not a panacea to scale up everything.
seunosewa 14y ago
From the article: "The only downsides of this approach are that it means that CPU bound Python code can’t scale to multiple cores within a single machine using threads, and that IO operations can incur unexpected additional latency in the presence of a CPU bound thread." Response: The only downside? As a Python user suffering from JVM envy, I have to say that that's a SERIOUS downside! Here's why: (1) Python is slow. Almost any real life program in pure Python will have CPU-bound components. (example: BBCode parsing on my forum). (2) Most programs that need to scale won't need to scale beyond a single machine (the most active web forum on my continent runs on a single quad core server) Therefore the need to be able to scale CPU-bound python programs to multiple cores on is a very real need. Even though we accept that removing the GIL is hard, let's not insult real-life Python users by suggesting that their needs are not real.
- hynek 14y ago
  The article doesn’t say multi-CPU scaling isn’t necessary. It says that threads are usually the wrong answer anyway. There are great process based ways to scale out – look no further than Erlang to see it’s true.
  - seunosewa 14y ago
    I use Python for my day job. I've experimented extensively with Java and Scala. I must say this: Threads are AWESOME. Threading is the most flexible model of concurrency because you can can build programs that EFFICIENTLY implement "alternative concurrency models" like message parsing and STM with threads. Threading is supported natively by every OS. And fast. Erlang is hyped as the ideal model for concurrency, but in practice is a niche product that's primarily useful for programs that are almost pure IO - chat servers, routing components like proxy servers and packet switches. The Erlang model does NOT apply to python, anyway, since Python processes are nothing like Erlang processes. Unlike Erlang processes, Python processes are very heavyweight and message passing between them is costly.
    - hynek 14y ago
      If you’re using python, the performance gap between processes with message passing and threads with locking is the last of your problems, believe me. The big difference is that processes are much more robust and testable. The cases where threads are really needed are fringe cases and – while it’s a pity – Python doesn’t seem the right language if you don’t want to go the Jython/IronPython way. The bigger problem is that people are used to go for threads by default although only few are able to write bug-free threaded code. It’s obsolete but prevalent performance wisdom and the fact that threads were really popular in the Java world.
dmbaggett 14y ago
In the last few hours, this has been silently added to the post, and I cannot post a reply on that site: Jumping on the internet to say that “they” (specifically, the people you’re not paying a cent to and who aren’t bothered by the GIL because it only penalises a programming style many of us consider ill advised in the first place) should “just fix it” (despite the serious risk of breaking user code that currently only works due to the coarse grained locking around each bytecode) is also always an option. Generally speaking, such pieces come across as “I have no idea how much engineering effort, now and in the future, is required to make this happen”. Fine, then don't bother. But don't insult us or call foul for our pointing out the downside impact on us of this decision. Your assumption that we are naive rubes who don't know how to code is really, really wrong. I have a very good idea how much engineering effort would be involved in fixing the GIL, and I am well aware that Python has involved many person-millennia of gratis work, and am appreciative of both. However, I still disagree with the Python devs' obviously entrenched position that fixing the GIL isn't worth the effort, and I will continue -- even when shouted down by the likes of you -- to advocate for the GIL's removal or some equivalently good solution. (As I said earlier, I am not opposed to STM solutions, but the current one performs unacceptably without special-purpose hardware.) Why? Because I have a single, selfish interest in this. I depend heavily on Python now, and want the language to be better. I have written many lines of Python 2 code that rely on the threading primitives in the standard library. Perhaps it was foolish of me to expect that the threading model offered by the standard library, modeled on Java's threading primitives, would some day work in the same way as Java threads do in practice. Nonetheless, I am left with a real world problem: my CPU-bound threaded Python code does not scale well to multiple cores. I need the GIL fixed, or to rewrite my Python code, or to migrate to another language that supports the standard model of threading programming that real-world programmers have been using for several decades, and which has built-in support from all major operating systems. Or, sure, wait for STM to be ready for prime time and migrate my thread-based semantics to the new STM-based semantics. The best path right for me right now is migration to Jython or IronPython. But then we are still unsupported orphans, living in the third world of Python 2.X. I guess it comes down to: do you want people to actually use this language to write programs they want to write, or do you want Python to be an advocacy platform for "correct" programming? Python's pragmatism has always appealed to me, so the ivory tower reaction to the practical concerns around the GIL really seem dissonant. (And this is coming from an MIT AI Lab Lisp guy who would rather write everything in Lisp. But Lisp lacks Python's wonderful, high-quality third party libraries and developer-centric pragmatism regarding unit testing, documentation, etc.) I know you are tired of hearing people bitch about the GIL, but, really: people write multithreaded programs. They should work as advertised, using native OS threading primitives and taking advantage of the native OS thread scheduler. Why does Python offer threading primitives if the language is not meant to support, from a practical standpoint, multithreaded programs?
gvalkov 14y ago
It's just one of the little things, but I'm still very conflicted about print as a function. The reasoning is sound, but I just can't seem to get used to it. It feels especially cluttered and clumsy with new-style string templates: print('ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper()) versus: print 'ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper() Despite the more consistent naming (ConfigParser -> configparser), simplified api (iteritems() -> items()) and all other syntactic improvements, I somehow still find 2.x code more enjoyable. Writing small scripts in Python has kind of lost its charm for me. This, of course, is all very subjective and I'll probably grow over it in a few thousand lines of code. I hope you're all less sensitive to the little things that annoy you.
wololo 14y ago
I wish python 3 would make default parameters immutable. It seems to go against the "simple", "explicit" python philosophy, not to mention that (it seems) most people tend to learn about the feature through debugging. description: http://effbot.org/zone/default-values.htm http://effbot.org/zone/default-values.htm
- hynek 14y ago
  Yeah sometimes I wish there were a warning whenever someone defines a function with [] as def arg.
  - wololo 14y ago
    I know pylint and pychecker do this, but not pep8. Opinions on those?

145 points by hynek 14y ago

toyg 14y ago

Unreadable on iPhone, what's up with its stylesheet? Safari won't even zoom...?

hynek 14y ago
Yeah, it seems to be the default RTFD style…try: http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-python-notes.readthedocs.org%2Fen%2Flatest%2Fpython3%2Fquestions_and_answers.html http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-...

deleted 14y ago

[deleted]

thu 14y ago

The following situation happens with other data than just unicode. This seems an argument for static typing in general. The reason this approach is problematic is that it means the traceback for an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x code base almost never points you to the code that is broken. Instead, you have to trace the origins of the data in the failing operation, and try to figure out where the unexpected 8-bit or Unicode code string was introduced. By contrast, Python 3 is designed to fail fast in most situations[...]

qznc 14y ago
I agree that static typing would helps for such bugs. However, not "in general". The general problem is that bogus data leads to an irreparable situation. To fix that bug, you have to find out who broke the data. Consider for example, a circle in a linked list. You would need quite an expressive type system to prevent that statically (dependent types might suffice? Theorem provers are the ultimate hammer).
- zanny 14y ago
  Different implementations for different use cases in different situations :P
- DannoHung 14y ago
  Uhh, how would you make a linked list with a circle in it with the Haskell List type?
  - dbaupp 14y ago
    a = 1:a
    - ufo 14y ago
      This trick is often called "Tying the Knot", if anyone is interested.
  - qznc 14y ago
    makeCircle rst = rst ++ makeCircle rst The thing is that a circular list is actually an infinite list in Haskell. ;)
    - ufo 14y ago
      This is not a real circular list though, since it does not tie the knot as in dbaupp's comment.
    - DannoHung 14y ago
      If you want to prevent that then you want to take general recursion out of your programming language and if you're gonna do that you might as well make it Turing incomplete. These aren't "circles" per se, they are partially evaluated recursive structures. If you expand them you end up with evaluated list structures that are non-self referential.
gardentheory 14y ago
Well the lighter solution and more Pythonic solution is strong typing. If a Unicode string and a byte string could not be combined then the problem would not exist AFAIK. For example "a" + 1 is an error, why is u"a" + "s" not an error?
- xerula 14y ago
  But b"s" + "a" is an error in Py3.
- qznc 14y ago
  This just gives a different error message. It does not help to find the error source. The programmer would probably just try to fix it via str(u"a")+"s" or u"a" + unicode("s").
- masklinn 14y ago
  > For example "a" + 1 is an error, why is u"a" + "s" not an error? "a" + b"s" is an error in Python 3.
  - gardentheory 14y ago
    Sorry, that was my point. Strengthening the types in python3 solves the problem without static typing being needed at all.
    - masklinn 14y ago
      ok. Static types solve it earlier though, especially for little-exercised code paths which could be forgotten or missed in testing.
slurgfest 14y ago
It is an argument for failing fast in general. Static typing isn't guaranteed to pick up every mistake at its origin, because far from every mistake is of the "int rather than string" variety. Since you have brought up the general case, the origins of many bugs involve something sophisticated going wrong which would require you to encode much of your program (hopefully not too redundantly) into a Turing-complete type system. Rather than stuffing square pegs into round holes, you could just write the appropriate error handling code (or even just asserts).
earl 14y ago
Please: alias hncomment='fold -w 77 -s | sed "s/^/ /" | pbcopy' use: echo "blah blah blah" | hncomment command-v Thus: The reason this approach is problematic is that it means the traceback for an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x code base almost never points you to the code that is broken. Instead, you have to trace the origins of the data in the failing operation, and try to figure out where the unexpected 8-bit or Unicode code string was introduced. By contrast, Python 3 is designed to fail fast in most situations[...]

zanny 14y ago

I think all the outrage is that Python 3 behaves like a new language in how everything needs to be ported (albeit at a magnitude less complexity than going up the language chain or across the playing field, like C to Ruby or C# to Java) but it still incurs the new language cost, while being the new version of the old language. If people just thought of Python 3 as the replacement to broken text model Python, like how XHTML tried to supplant HTML and failed, and then HTML5 supplanted XHTML with some quirks, the same thing happens here. Also, I adore the comments about the GIL. I understand the benefits of using a single language in every circumstance, but (even) a JIT scripting language is not sufficient for all problems. I guess it depends on if a polyglot code base is more complex than handling multithreading and the corresponding memory complexity and gimmicks, and that is developer centric. Still, I would much rather just stick processor bound code in openMP C functions and call those from Python when need be. It seems like the "right" answer to the performance problem, without losing much productivity.

smeg 14y ago

Python3 and Perl6 can both fuck off. I look forward to the day PyPy is considered the real Python. Look at PyPy's homepage (http://pypy.org/ http://pypy.org/), doesn't even mention Unicode as a significant feature. Instead it talks about speed, security, concurrency, and compatibility with the current real Python (2.7.2) - all the things real Python programmers care about and expect the Python developers to focus on. PyPy may be some way off but I want to find the developers and hug them for setting the right vision and trying. I just donated $50 and if you hate Python3 you should too.

sho_hn 14y ago
Or if you really like Python 3 (as I do!) you can donate those $50 to Python 3 support in PyPy: http://pypy.org/py3donate.html http://pypy.org/py3donate.html
Deestan 14y ago
> all the things real Python programmers care about Guess I'm not a real programmer, then, since I find proper Unicode handling to be a welcome feature, and find the implementation in Python2 is painful.
Marwy 14y ago
I think you should take a walk, and then come back and read this ridiculous stuff you wrote.
lclarkmichalek 14y ago
Any reason they can "fuck off"? Anyway, you're incorrect about pypy. "Real" python developers will likely not run their code in a sandboxed environment, so PyPy's security advantages are minimal, added to the fact that your sandbox is only as good as its configuration. The concurrency is interesting, but there are numerous other solutions available, and that alone makes it unlikely that the de facto implementation of the language would include one. Claiming that PyPy is better on compatibility than CPython is ridiculous; last time I looked, anything using a C extension had to be rewritten to use ctypes. On speed, yes PyPy is impressive. But in python, anything that is particularly CPU intensive should probably be written as a C extension and PyPy takes a hit when using ctypes iirc. In general, I don't see the point of people trying to optimize generic python code. Most code never gets profiled, and most code isn't a problem. At least, not a problem enough to switch interpreter. Definitely not a problem enough to ignore the massive problem that is python 2.x's unicode implementation.

sutro 14y ago

Unconvinced.

ash 14y ago

Why can't I use Python 2 modules from Python 3 program? Is there any effort to make it possible?

hynek 14y ago
There is no such thing as "Python 2" or "Python 3" modules. There are just modules that will go belly up because they are incompatible with one or another. I presume you mean that you can’t access your old Python 2 modules from a new Python 3 installation. That’s just because Python installations usually don’t share their modules (i.e. site-packages). You can try to install them using Python 3 (usually just python3 setup.py install) and see if they work or not.
- lrem 14y ago
  There's a deeper question here: why doesn't the interpreter contain a compatibility layer for the old code? There's no really good reason for the old modules to go belly up in the new interpreter. At least no other than the man months needed to implement that layer. But then, that probably would still be orders of magnitude less than migrating all the libraries that keep people from migrating...
  - hynek 14y ago
    That’s essentially what __future__ imports are for – things that go further would be simply too much work and Python 3 was about shedding weight, not shipping essentially two interpreters. (while they are very similar from outside, the innards couldn’t be more different)
    - lrem 14y ago
      Cool. So this way porting things to Py3k got easier. That perfectly explains why all the critical libs have been ported fast and no one remembers about Python 2 anymore, doesn't it? Shedding weight is sometimes a step in a good direction. But we need to draw a line at some point. As the history of Python 3 adoption shows, that point maybe was not optimal. Edit: forgot an angle: Also, __future__ is fixing the incompatibility of what people are using with something that they can't use. I'm not sure if it's the right problem to solve.

dmbaggett 14y ago

From the article: Back in reality, though, complaining about the GIL as though its a serious barrier to adoption amongst developers that know what they’re doing often says more about the person doing the complaining than it does about CPython. It was a solid, well-argued piece up to this point. You do yourself and the Python community a disservice by writing off your critics as ignorant. It sounds petulant and childish, and is wrong. There are valid arguments on both sides of the GIL argument, but neither side's advocates are ignorant or bad programmers.

sho_hn 14y ago
I have to agree here, despite the two of us rating the importance of the GIL problem quite differently in yesterday's discussion: A better strategy against the GIL FUD is plain old education, not trying to make it a taboo topic. Especially programmers who do know what they're doing won't take too kindly to a community where that becomes prevalent.
- hynek 14y ago
  I find Nick arguing pretty much to the point and the quote above was ripped from it’s context which is a huge section balancing the arguments against each other. tldr He argues there are better ways for scaling out than threads and removing the GIL would have enormous consequence _throughout_ the whole code base so it’s removal cannot be warranted. Reducing 10 paragraphs and 11 bullet points to “only stupid people use threads” is just poor style.
slurgfest 14y ago
Let's be honest: you are upset because you have sometime committed to saying the GIL is a huge deal and you feel insulted by anyone saying it isn't, which is why you are using emotionally loaded words like 'petulant' and 'childish' and 'ignorant' and 'bad'. The GIL is the single biggest target for language-advocacy FUD against Python by advocates of other languages. I would be a rich person if I got a dollar for every time I saw someone trashing Python as a toy language because of the GIL, without significant knowledge of how to use Python. It's just a much huger issue to someone without real Python experience than it is to people with Python experience. Setting that aside, there are a vast number of use cases where someone might try to use threads when actually that's not a good solution. It actually isn't incredibly easy to come up with cases where the GIL is this huge fatal flaw. It's really sad how many times I have seen people ranting about the GIL and totally unaware of multiprocessing, unable to give a technical reason why they can't try processes, unaware of greenlets, wholly unaware that the GIL does not exist in Jython/IronPython, etc. I'm sure that doesn't apply to you, which would put you in the minority. Thus, "often says more about the person doing the complaining". Threads should be used judiciously, because they dramatically increase the complexity of a program and the difficulty of debugging and reasoning about it. Shared-everything is a great way to blow your foot off if you don't specifically need it. So if you throw threads at every problem indiscriminately, then that really is a weak point in your programming. (Note: I am not saying that using threads is always a bad idea) If you are a good programmer then you should already know, and not be offended to hear, that threads are a tool of specific applicability, not a panacea to scale up everything.

seunosewa 14y ago

From the article: "The only downsides of this approach are that it means that CPU bound Python code can’t scale to multiple cores within a single machine using threads, and that IO operations can incur unexpected additional latency in the presence of a CPU bound thread." Response: The only downside? As a Python user suffering from JVM envy, I have to say that that's a SERIOUS downside! Here's why: (1) Python is slow. Almost any real life program in pure Python will have CPU-bound components. (example: BBCode parsing on my forum). (2) Most programs that need to scale won't need to scale beyond a single machine (the most active web forum on my continent runs on a single quad core server) Therefore the need to be able to scale CPU-bound python programs to multiple cores on is a very real need. Even though we accept that removing the GIL is hard, let's not insult real-life Python users by suggesting that their needs are not real.

hynek 14y ago
The article doesn’t say multi-CPU scaling isn’t necessary. It says that threads are usually the wrong answer anyway. There are great process based ways to scale out – look no further than Erlang to see it’s true.
- seunosewa 14y ago
  I use Python for my day job. I've experimented extensively with Java and Scala. I must say this: Threads are AWESOME. Threading is the most flexible model of concurrency because you can can build programs that EFFICIENTLY implement "alternative concurrency models" like message parsing and STM with threads. Threading is supported natively by every OS. And fast. Erlang is hyped as the ideal model for concurrency, but in practice is a niche product that's primarily useful for programs that are almost pure IO - chat servers, routing components like proxy servers and packet switches. The Erlang model does NOT apply to python, anyway, since Python processes are nothing like Erlang processes. Unlike Erlang processes, Python processes are very heavyweight and message passing between them is costly.
  - hynek 14y ago
    If you’re using python, the performance gap between processes with message passing and threads with locking is the last of your problems, believe me. The big difference is that processes are much more robust and testable. The cases where threads are really needed are fringe cases and – while it’s a pity – Python doesn’t seem the right language if you don’t want to go the Jython/IronPython way. The bigger problem is that people are used to go for threads by default although only few are able to write bug-free threaded code. It’s obsolete but prevalent performance wisdom and the fact that threads were really popular in the Java world.

dmbaggett 14y ago

In the last few hours, this has been silently added to the post, and I cannot post a reply on that site: Jumping on the internet to say that “they” (specifically, the people you’re not paying a cent to and who aren’t bothered by the GIL because it only penalises a programming style many of us consider ill advised in the first place) should “just fix it” (despite the serious risk of breaking user code that currently only works due to the coarse grained locking around each bytecode) is also always an option. Generally speaking, such pieces come across as “I have no idea how much engineering effort, now and in the future, is required to make this happen”. Fine, then don't bother. But don't insult us or call foul for our pointing out the downside impact on us of this decision. Your assumption that we are naive rubes who don't know how to code is really, really wrong. I have a very good idea how much engineering effort would be involved in fixing the GIL, and I am well aware that Python has involved many person-millennia of gratis work, and am appreciative of both. However, I still disagree with the Python devs' obviously entrenched position that fixing the GIL isn't worth the effort, and I will continue -- even when shouted down by the likes of you -- to advocate for the GIL's removal or some equivalently good solution. (As I said earlier, I am not opposed to STM solutions, but the current one performs unacceptably without special-purpose hardware.) Why? Because I have a single, selfish interest in this. I depend heavily on Python now, and want the language to be better. I have written many lines of Python 2 code that rely on the threading primitives in the standard library. Perhaps it was foolish of me to expect that the threading model offered by the standard library, modeled on Java's threading primitives, would some day work in the same way as Java threads do in practice. Nonetheless, I am left with a real world problem: my CPU-bound threaded Python code does not scale well to multiple cores. I need the GIL fixed, or to rewrite my Python code, or to migrate to another language that supports the standard model of threading programming that real-world programmers have been using for several decades, and which has built-in support from all major operating systems. Or, sure, wait for STM to be ready for prime time and migrate my thread-based semantics to the new STM-based semantics. The best path right for me right now is migration to Jython or IronPython. But then we are still unsupported orphans, living in the third world of Python 2.X. I guess it comes down to: do you want people to actually use this language to write programs they want to write, or do you want Python to be an advocacy platform for "correct" programming? Python's pragmatism has always appealed to me, so the ivory tower reaction to the practical concerns around the GIL really seem dissonant. (And this is coming from an MIT AI Lab Lisp guy who would rather write everything in Lisp. But Lisp lacks Python's wonderful, high-quality third party libraries and developer-centric pragmatism regarding unit testing, documentation, etc.) I know you are tired of hearing people bitch about the GIL, but, really: people write multithreaded programs. They should work as advertised, using native OS threading primitives and taking advantage of the native OS thread scheduler. Why does Python offer threading primitives if the language is not meant to support, from a practical standpoint, multithreaded programs?

gvalkov 14y ago

It's just one of the little things, but I'm still very conflicted about print as a function. The reasoning is sound, but I just can't seem to get used to it. It feels especially cluttered and clumsy with new-style string templates: print('ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper()) versus: print 'ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper() Despite the more consistent naming (ConfigParser -> configparser), simplified api (iteritems() -> items()) and all other syntactic improvements, I somehow still find 2.x code more enjoyable. Writing small scripts in Python has kind of lost its charm for me. This, of course, is all very subjective and I'll probably grow over it in a few thousand lines of code. I hope you're all less sensitive to the little things that annoy you.

wololo 14y ago

I wish python 3 would make default parameters immutable. It seems to go against the "simple", "explicit" python philosophy, not to mention that (it seems) most people tend to learn about the feature through debugging. description: http://effbot.org/zone/default-values.htm http://effbot.org/zone/default-values.htm

hynek 14y ago
Yeah sometimes I wish there were a warning whenever someone defines a function with [] as def arg.
- wololo 14y ago
  I know pylint and pychecker do this, but not pep8. Opinions on those?