4 ms·

Hm, this solution seems very cumbersome, inelegant and not like python's "batteries included" approach at all. This means that python will have native threads t

Hm, this solution seems very cumbersome, inelegant and not like python's "batteries included" approach at all. This means that python will have native threads that behhave as expected minus true parallel execution, so you shouldn't use those, even though the interface is fairly simple. Instead, you should learn to use this weird contraption that is neither multiprocessing nor intuitive multithreading and comes with a cumbersome interface. I get that the GIL is a very hard problem to solve, but this solution is so inelegant in my eyes that python would be better off without it. I'd feel better if this was a hidden implementation detail that coukd be improved transparently. Just my two cents.

akvadrako 7y ago
I completely disagree - Python threads are basically "green threads", so they have their place but aren't related to parallelisation. But true multiprocessing is ugly when you have hundreds of cores, which is where CPUs are going. There is no standard UI convention on most OSes to group those processes per app, in terms of signals or stats or whatever. So besides the unproven possibility of removing the GIL, subinterpreters are the best way forward, better than threads or the multiprocessing package.
- icebraining 7y ago
  It's not that Python shouldn't have subinterpreters, but that they should only be added when they have a better API - at least as good multiprocessing.
- chrisseaton 7y ago
  I don’t think they’re green threads - they’re full OS threads. They just have the GIL.
- zbentley 7y ago
  > Python threads are basically "green threads" That's not accurate. > they have their place but aren't related to parallelisation You can parallelize all sorts of things with Python threads--just not some things you'd expect to be able to parallelize, due to the GIL. Waiting on or buffering I/O, calling out to compiled code, doing cryptographic operations--all of those can be parallelized, as (in many cases) they entail releasing the GIL. > But true multiprocessing is ugly when you have hundreds of cores Why? > There is no standard UI convention on most OSes to group those processes per app, in terms of signals or stats or whatever. I have no idea what this means. What do UIs have to do with process groups? Do you know how many processes your Chrome instance is running on the operating system? There are very solid conventions regarding process management, at least on Unix-ish systems: process groups and parent-child relationships are well established and well understood, as is their relationship with signals and signal handling.
  - anaphor 7y ago
    I think he's saying that they allow you to do concurrency, but not parallelism. Those are two different things.
    - zbentley 7y ago
      They do allow you to do parallelism, though. There are just some things they can't parallelize because of the GIL. Green threads also allow for parallelism, if they're scheduled onto more than one executor.
      - anaphor 7y ago
        That's true, behind the scenes they can be running on multiple CPUs, which is the definition of parallelism. I think threads are a bad abstraction for doing parallelism personally, though. Programs designed to run in parallel should be deterministic, unless they need concurrency for some other reason. I think trying to shoehorn parallel programming into Python's threads isn't necessarily the best approach. As far as I'm concerned, if I'm using threads and they happen to get scheduled on to multiple cores, then that's a nice optimization, but isn't necessary for what I use threads for.
    - tellak 7y ago
      If that’s what he’s saying, he’s wrong. Python threads are based on native threads. They are parallel executed, they just don’t do so very well because of the GIL.
ru999gol 7y ago
I have the same opinion about asyncio, its such a bad API that its almost impossible to use correctly. But still, probably better than nothing.
- theelous3 7y ago
  Use curio or trio instead :) async/await are not only for asyncio.
- jnbiche 7y ago
  Coming from the JS world, asyncio feels fine to me. I've noticed a lot of Python programmers who have a strong background in C and Python struggle with asyncio. But in their case, I think it's more them disliking the async programming paradigm rather than the specific asyncio API.
  - ru999gol 7y ago
    personally I have JS experience (browser/node.js), as well as asio in C++/Boost, EventMachine in ruby and Twisted/Tornado in python. I strongly disagree that people who dislike asyncio only do so because they don't like the event loop. That just sounds like a typical quick dismissal of criticism. No, its a objectively awful convoluted API. But if you think you can write correct asyncio applications of any significant complexity, that's great. Doesn't change the fact that a lot of highly experienced developers struggle to do so.
    - jnbiche 7y ago
      > But if you think you can write correct asyncio applications of any significant complexity, that's great. To be fair, I've never written an asyncio application of any significant complexity (at least compared to the async JS application I've written). But nothing jumps out at me as horrific with asyncio. The worst thing is the split between futures and coroutines, but once you realize that there's no real need to use futures any more with asyncio (although you can integrate futures-based libraries if needed), you're good to go. But fair criticism, I've only written small asyncio applications. The only sizable async-Python application I've made was made using Tornado and ZeroMQ async APIs(although even that eventually ran on the asyncio event loop and used an asyncio-based library).
coldtea 7y ago
>This means that python will have native threads that behhave as expected minus true parallel execution, so you shouldn't use those, even though the interface is fairly simple. Python already has exactly that, and has had that for ages. >Instead, you should learn to use this weird contraption that is neither multiprocessing nor intuitive multithreading and comes with a cumbersome interface. It also comes with performance improvements over multiprocess, so there's that. Besides the "cumbersome interface" is irrelevant, as it would be easy to wrap and forget about it, the same way nobody really uses urllib directly.
- ori_b 7y ago
  > Python already has exactly that, and has had that for ages. And the point is that this should be fixed, instead of adding yet another clunky way of doing the same thing. > Besides the "cumbersome interface" is irrelevant, as it would be easy to wrap and forget about it, the same way nobody really uses urllib directly. Then why bother having it, if the only reason for it to exist is so that it can be wrapped? That's just stacking layers of turd.
  - baq 7y ago
    Almost nobody writes assembly nowadays, but I don’t see anyone wanting it gone. This is assembly. Don’t want to wrap it yourself, fine, don’t complain that it exists.
    - cyphar 7y ago
      You could implement requests without urllib. You cannot implement C without assembly. That's the distinction between useful layers of abstraction and pointless ones. If a better interface to subinterpreters is bound to be developed, then that should've been the stdlib interface in the first place.
      - coldtea 7y ago
        >You could implement requests without urllib. Yes, but you couldn't do more fine-grained than requests things, unless you did them from excruciatingly lower level (e.g. sockets).

akvadrako 7y ago

I completely disagree - Python threads are basically "green threads", so they have their place but aren't related to parallelisation. But true multiprocessing is ugly when you have hundreds of cores, which is where CPUs are going. There is no standard UI convention on most OSes to group those processes per app, in terms of signals or stats or whatever. So besides the unproven possibility of removing the GIL, subinterpreters are the best way forward, better than threads or the multiprocessing package.

icebraining 7y ago
It's not that Python shouldn't have subinterpreters, but that they should only be added when they have a better API - at least as good multiprocessing.
chrisseaton 7y ago
I don’t think they’re green threads - they’re full OS threads. They just have the GIL.
zbentley 7y ago
> Python threads are basically "green threads" That's not accurate. > they have their place but aren't related to parallelisation You can parallelize all sorts of things with Python threads--just not some things you'd expect to be able to parallelize, due to the GIL. Waiting on or buffering I/O, calling out to compiled code, doing cryptographic operations--all of those can be parallelized, as (in many cases) they entail releasing the GIL. > But true multiprocessing is ugly when you have hundreds of cores Why? > There is no standard UI convention on most OSes to group those processes per app, in terms of signals or stats or whatever. I have no idea what this means. What do UIs have to do with process groups? Do you know how many processes your Chrome instance is running on the operating system? There are very solid conventions regarding process management, at least on Unix-ish systems: process groups and parent-child relationships are well established and well understood, as is their relationship with signals and signal handling.
- anaphor 7y ago
  I think he's saying that they allow you to do concurrency, but not parallelism. Those are two different things.
  - zbentley 7y ago
    They do allow you to do parallelism, though. There are just some things they can't parallelize because of the GIL. Green threads also allow for parallelism, if they're scheduled onto more than one executor.
    - anaphor 7y ago
      That's true, behind the scenes they can be running on multiple CPUs, which is the definition of parallelism. I think threads are a bad abstraction for doing parallelism personally, though. Programs designed to run in parallel should be deterministic, unless they need concurrency for some other reason. I think trying to shoehorn parallel programming into Python's threads isn't necessarily the best approach. As far as I'm concerned, if I'm using threads and they happen to get scheduled on to multiple cores, then that's a nice optimization, but isn't necessary for what I use threads for.
  - tellak 7y ago
    If that’s what he’s saying, he’s wrong. Python threads are based on native threads. They are parallel executed, they just don’t do so very well because of the GIL.

ru999gol 7y ago

I have the same opinion about asyncio, its such a bad API that its almost impossible to use correctly. But still, probably better than nothing.

theelous3 7y ago
Use curio or trio instead :) async/await are not only for asyncio.
jnbiche 7y ago
Coming from the JS world, asyncio feels fine to me. I've noticed a lot of Python programmers who have a strong background in C and Python struggle with asyncio. But in their case, I think it's more them disliking the async programming paradigm rather than the specific asyncio API.
- ru999gol 7y ago
  personally I have JS experience (browser/node.js), as well as asio in C++/Boost, EventMachine in ruby and Twisted/Tornado in python. I strongly disagree that people who dislike asyncio only do so because they don't like the event loop. That just sounds like a typical quick dismissal of criticism. No, its a objectively awful convoluted API. But if you think you can write correct asyncio applications of any significant complexity, that's great. Doesn't change the fact that a lot of highly experienced developers struggle to do so.
  - jnbiche 7y ago
    > But if you think you can write correct asyncio applications of any significant complexity, that's great. To be fair, I've never written an asyncio application of any significant complexity (at least compared to the async JS application I've written). But nothing jumps out at me as horrific with asyncio. The worst thing is the split between futures and coroutines, but once you realize that there's no real need to use futures any more with asyncio (although you can integrate futures-based libraries if needed), you're good to go. But fair criticism, I've only written small asyncio applications. The only sizable async-Python application I've made was made using Tornado and ZeroMQ async APIs(although even that eventually ran on the asyncio event loop and used an asyncio-based library).

coldtea 7y ago

>This means that python will have native threads that behhave as expected minus true parallel execution, so you shouldn't use those, even though the interface is fairly simple. Python already has exactly that, and has had that for ages. >Instead, you should learn to use this weird contraption that is neither multiprocessing nor intuitive multithreading and comes with a cumbersome interface. It also comes with performance improvements over multiprocess, so there's that. Besides the "cumbersome interface" is irrelevant, as it would be easy to wrap and forget about it, the same way nobody really uses urllib directly.

ori_b 7y ago
> Python already has exactly that, and has had that for ages. And the point is that this should be fixed, instead of adding yet another clunky way of doing the same thing. > Besides the "cumbersome interface" is irrelevant, as it would be easy to wrap and forget about it, the same way nobody really uses urllib directly. Then why bother having it, if the only reason for it to exist is so that it can be wrapped? That's just stacking layers of turd.
- baq 7y ago
  Almost nobody writes assembly nowadays, but I don’t see anyone wanting it gone. This is assembly. Don’t want to wrap it yourself, fine, don’t complain that it exists.
  - cyphar 7y ago
    You could implement requests without urllib. You cannot implement C without assembly. That's the distinction between useful layers of abstraction and pointless ones. If a better interface to subinterpreters is bound to be developed, then that should've been the stdlib interface in the first place.
    - coldtea 7y ago
      >You could implement requests without urllib. Yes, but you couldn't do more fine-grained than requests things, unless you did them from excruciatingly lower level (e.g. sockets).