5 ms·

Stackless Python for Python 3.0 and Python 3.0.1, now available.

28 points by arem 18y ago

ggruschow 18y ago
That's cool and all, but cores are practically growing on trees now. Is there something out there that'll let me efficiently do tons of active objects (coroutines with data that can make sync or async calls?) but scale out across processors and machines (hi EC2, I love you) also? I can do without shared data. As it stands now, unless I had a big need for bazillions of contexts, I wouldn't touch stackless because I've already got to use other mechanisms to get work running on a lot of different cores / machines at once anyway. Since I'm already doing that, it's nicer to stick to the one mechanism and design things to work within it (keep contexts <1,000).
- nihilocrat 18y ago
  The idea is that Stackless allows you to easily get concurrency via coroutines and message passing. My understanding is that, along with some other things, it's basically Python with no C stack and no Global Interpreter Lock. It sounds like you already have a method you're comfortable with, so there's obviously not much reason to look for something else unless you are tearing your hair out with your current solution.
  - bendotc 18y ago
    This ends up needing to be posted in every Stackless discussion: Stackless DOES NOTHING ABOUT THE GIL! Stackless is not about concurrency. It's about implementing coroutines. It's a really easy mistake to make. Lots of people think Stackless threadlets look like Erlang processes, but instead, they look more like regular Python generators (although they're admittedly much cooler). To complicate matters, people sometimes actually do use the fact that you can pickle threadlets, send them across the wire, and resume them on another thread/process/machine to implement some form of concurrency, but this does not mean that Stackless gets around the GIL in any way vanilla CPython can't. To explain more about Stackless, I'll refer you to Jesse Noller's recent post on the subject: http://jessenoller.com/2009/02/23/stackless-you-got-your-coroutines-in-my-subroutines/ http://jessenoller.com/2009/02/23/stackless-you-got-your-cor...
    - neilo 18y ago
      Man as a mostly-outside-admirer of Python, I love its terminology: "pickle threadlets". Delicious!
    - lkjfajfoi 18y ago
      Coroutines are concurrent. They are not parallel.
      - bendotc 18y ago
        That's a distinction without a (widely accepted) difference. Coroutines exist at the same time as one another (you could say they exist in parallel or concurrently), but they do not execute at the same time within a single process (1). I think most people wouldn't call it either parallel or concurrent, but if concurrency means something else to you, that's fine, but don't correct me by your definition. For what it's worth, here's the intro to "Concurrency (computer science)" on Wikipedia: "In computer science, concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other. The computations may be executing on multiple cores in the same chip, preemptively time-shared threads on the same processor, or executed on physically separated processors. A number of mathematical models have been developed for general concurrent computation including Petri nets, process calculi, the synchronous model and the Actor model." (1) Okay, so, they could be operating at the same time if two tasklets are in different threads and at least one is inside C code that has released the GIL. But that's the same as any CPython code.
        lkjfajfoi 18y ago
        Do you agree or disagree with Wikipedia? Tasklets have essentially the same capabilities as "preemptively time-shared threads on the same processor." The fact that the preemptive scheduler operates between VM instructions rather than CPU instructions is irrelevant.
        bendotc 18y ago
        I accept Wikipedia's definition as a fine one and use Wikipedia specifically as something that represents a reasonably mainstream opinion on the matter. VM instructions vs. CPU instructions is not irrelevant when the VM can only execute instructions one-by-one and the CPU can execute instructions on multiple cores at once (ignoring vector operations for simplicity). But what's more is that that distinction between VM and CPU instructions is irrelevant, because in common use (on EVE Online, the people who pretty much keep Stackless going), tasklets are not pre-empted at all, but are instead cooperative, handing off control flow only when they explicitly do so (such as sending or receiving on a channel, sleeping, or terminating). Tasklets can be preempted after a time, and if you're using them that way, then okay, I accept that it's as concurrent as anything on one CPython VM is, but you lose a lot of the benefits of not dealing with locking (as well as a few other things) when you do this, so it's my understanding that it's not the preferred way to use Stackless. So, yes, you can use threadlets preemptively and then I agree that you can kinda call them "concurrent" and be correct, but by correcting people when they say it's not concurrent, you're bringing more heat than light, as in general use, it's not concurrent, and if you use the preemption capabilities in Stackless, then you're not following the Stackless best practices and you're still not going to get any more use out of your CPU's extra cores, which is what most people are looking for in Stackless (and which sadly they won't find).
- dfj225 18y ago
  What about the multiprocessing module, that has been included with Python since v 2.6? http://docs.python.org/library/multiprocessing.html http://docs.python.org/library/multiprocessing.html
  - rbanffy 18y ago
    Won't let programs share a single in-process object the way proper threading support would.
- tdavis 18y ago
  pp (parallel python) supports a pool of machines for handling parallel tasks. She's a bit rough around the edges, but can generally get the job done.
- moe 18y ago
  I hear you, I'd love to have the same thing. Still the big deal about co-routines for me is not mainly raw performance (or core-utilization) but rather the programming model. I've literally slashed the LOC-count in half for some of our projects just by moving to co-routines, that's what I call a productivity boost. Also I find myself enjoying co-routines much more than the tangled mess of callbacks and data-locality issues in the multithreaded world.
  - MaysonL 18y ago
    I still remember with fondness the first coroutines I wrote, almost 35 years ago: it seemed like a really cool hack to connect a producer of bytes to a consumer of them (both already written, just 6 lines of assembler plus a little bit of init code).

28 points by arem 18y ago

That's cool and all, but cores are practically growing on trees now. Is there something out there that'll let me efficiently do tons of active objects (coroutines with data that can make sync or async calls?) but scale out across processors and machines (hi EC2, I love you) also? I can do without shared data. As it stands now, unless I had a big need for bazillions of contexts, I wouldn't touch stackless because I've already got to use other mechanisms to get work running on a lot of different cores / machines at once anyway. Since I'm already doing that, it's nicer to stick to the one mechanism and design things to work within it (keep contexts <1,000).

nihilocrat 18y ago
The idea is that Stackless allows you to easily get concurrency via coroutines and message passing. My understanding is that, along with some other things, it's basically Python with no C stack and no Global Interpreter Lock. It sounds like you already have a method you're comfortable with, so there's obviously not much reason to look for something else unless you are tearing your hair out with your current solution.
- bendotc 18y ago
  This ends up needing to be posted in every Stackless discussion: Stackless DOES NOTHING ABOUT THE GIL! Stackless is not about concurrency. It's about implementing coroutines. It's a really easy mistake to make. Lots of people think Stackless threadlets look like Erlang processes, but instead, they look more like regular Python generators (although they're admittedly much cooler). To complicate matters, people sometimes actually do use the fact that you can pickle threadlets, send them across the wire, and resume them on another thread/process/machine to implement some form of concurrency, but this does not mean that Stackless gets around the GIL in any way vanilla CPython can't. To explain more about Stackless, I'll refer you to Jesse Noller's recent post on the subject: http://jessenoller.com/2009/02/23/stackless-you-got-your-coroutines-in-my-subroutines/ http://jessenoller.com/2009/02/23/stackless-you-got-your-cor...
  - neilo 18y ago
    Man as a mostly-outside-admirer of Python, I love its terminology: "pickle threadlets". Delicious!
  - lkjfajfoi 18y ago
    Coroutines are concurrent. They are not parallel.
    - bendotc 18y ago
      That's a distinction without a (widely accepted) difference. Coroutines exist at the same time as one another (you could say they exist in parallel or concurrently), but they do not execute at the same time within a single process (1). I think most people wouldn't call it either parallel or concurrent, but if concurrency means something else to you, that's fine, but don't correct me by your definition. For what it's worth, here's the intro to "Concurrency (computer science)" on Wikipedia: "In computer science, concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other. The computations may be executing on multiple cores in the same chip, preemptively time-shared threads on the same processor, or executed on physically separated processors. A number of mathematical models have been developed for general concurrent computation including Petri nets, process calculi, the synchronous model and the Actor model." (1) Okay, so, they could be operating at the same time if two tasklets are in different threads and at least one is inside C code that has released the GIL. But that's the same as any CPython code.
      - lkjfajfoi 18y ago
        Do you agree or disagree with Wikipedia? Tasklets have essentially the same capabilities as "preemptively time-shared threads on the same processor." The fact that the preemptive scheduler operates between VM instructions rather than CPU instructions is irrelevant.
        bendotc 18y ago
        I accept Wikipedia's definition as a fine one and use Wikipedia specifically as something that represents a reasonably mainstream opinion on the matter. VM instructions vs. CPU instructions is not irrelevant when the VM can only execute instructions one-by-one and the CPU can execute instructions on multiple cores at once (ignoring vector operations for simplicity). But what's more is that that distinction between VM and CPU instructions is irrelevant, because in common use (on EVE Online, the people who pretty much keep Stackless going), tasklets are not pre-empted at all, but are instead cooperative, handing off control flow only when they explicitly do so (such as sending or receiving on a channel, sleeping, or terminating). Tasklets can be preempted after a time, and if you're using them that way, then okay, I accept that it's as concurrent as anything on one CPython VM is, but you lose a lot of the benefits of not dealing with locking (as well as a few other things) when you do this, so it's my understanding that it's not the preferred way to use Stackless. So, yes, you can use threadlets preemptively and then I agree that you can kinda call them "concurrent" and be correct, but by correcting people when they say it's not concurrent, you're bringing more heat than light, as in general use, it's not concurrent, and if you use the preemption capabilities in Stackless, then you're not following the Stackless best practices and you're still not going to get any more use out of your CPU's extra cores, which is what most people are looking for in Stackless (and which sadly they won't find).
dfj225 18y ago
What about the multiprocessing module, that has been included with Python since v 2.6? http://docs.python.org/library/multiprocessing.html http://docs.python.org/library/multiprocessing.html
- rbanffy 18y ago
  Won't let programs share a single in-process object the way proper threading support would.
tdavis 18y ago
pp (parallel python) supports a pool of machines for handling parallel tasks. She's a bit rough around the edges, but can generally get the job done.
moe 18y ago
I hear you, I'd love to have the same thing. Still the big deal about co-routines for me is not mainly raw performance (or core-utilization) but rather the programming model. I've literally slashed the LOC-count in half for some of our projects just by moving to co-routines, that's what I call a productivity boost. Also I find myself enjoying co-routines much more than the tangled mess of callbacks and data-locality issues in the multithreaded world.
- MaysonL 18y ago
  I still remember with fondness the first coroutines I wrote, almost 35 years ago: it seemed like a really cool hack to connect a producer of bytes to a consumer of them (both already written, just 6 lines of assembler plus a little bit of init code).