10 ms·

A green threading library with true concurrency for Python

30 points by lukes386 13y ago

pekk 13y ago
Since this is a very new project, it is a good time to suggest that you abide by PEP8 (e.g., no 'waitAll') since this would be widely appreciated, and is not easy to fix later on.
- mirman 13y ago
  waitAll gone. Before I put it in any package managers it will get style guided. This is currently on version -1.0.0.
  - derleth 13y ago
    > version -1.0.0. Version negative one? I don't think I've ever seen that before. Usually, the very earliest versions of software are numbered like 0.0.1 or something like that.
    - mirman 13y ago
      - here actually negates the ordering of the numbers in the list. -1.0.0 === 0.0.1.
      - derleth 13y ago
        > - here actually negates the ordering of the numbers in the list. -1.0.0 === 0.0.1. Well, I've certainly never seen that before.
        ceol 13y ago
        It should be written 0.0.1[::-1]
jacob019 13y ago
I don't really get what advantage this gives me beyond using gevent. There's still no parallelism. The readme describes it as "A solution to concurrency," but I already get that using gevent. Is it just to improve communication between green threads?
- mirman 13y ago
  Fewer explicit yields & no monkey patching necessary for IO performing libraries.
  - MetaCosm 13y ago
    As a former python user, I wish this library had existed 18 months ago -- would have helped with some of the nasty cases you can get caught on with gevent.
- jerf 13y ago
  It appears to add some amount of Python-bytecode level preemption to gevent, which allows you to hopefully avoid some of the pathological cases of cooperative scheduling. Said pathological cases are only a matter of scale... if your program becomes large enough, you will hit them, eventually. That said, with no offense intended to mirman, I'd really hesitate before using this for anything serious enough to reach that scale in the first place. Gevent, frankly, visibly pushes Python to the limits (and occasionally a bit beyond), trying to also tack on some preemption on an environment not fundamentally expecting it would scare me another notch.
  - mirman 13y ago
    No offense taken - both of these are obviously visibly delicate. There is a version in the history that used Greenlet instead of gevent which was potentially a bit less delicate, but it required wrapping of the main file and didn't work with time.sleep, and I didn't feel like it was worth writing my own locks, semaphores, mutexes, pipes and whatnot.
  - hogu 13y ago
    what are the pathological cases one runs into with gevent?
fusiongyro 13y ago
> Conpig threads still can only run on one core of a processor. The disillusionment caused by having so many options for non-parallel "concurrency" in Python is, I believe, feeding the high defection rate from Python to Go.
- mirman 13y ago
  The lack of options for parallel (and non-parallel) concurrency is, along with other things, feeding the high defection rate to many other languages.
  - pekk 13y ago
    We've had processes, threads and greenlets for a while now... if anything the problem isn't that there are no options, but too many options that require understanding to choose and apply. Many of the people complaining about this issue don't have a demonstrated problem and could try any simple approach first (if the point is not just to slam Python in favor of something else, from the beginning).
    - MetaCosm 13y ago
      This type of response is why I gave up on Python entirely. Not to pile on you pekk, it isn't your fault, but it is a tone... defensive apologist... "first of all there is no problem, and if there was a problem... it is that Python is too awesome" As someone who has had to ship stuff using multiprocess & gevent to actually meet real world scaling needs -- and integrating them with C code and communicating to a C++ application via ZMQ (inprocess by sharing the object) ... the sad fact is once we started to tackle really hard problems in Python that aren't pre-solved via a nice library all those early advantages fell away and we craved the blessed simplicity of C++ (note: sarcasm, C++ isn't simple, but it was far simpler than the Frankenstein's monster we built).
    - fusiongyro 13y ago
      MetaCosm made the point quite eloquently, but let me juxtapose "too many options that require understanding to choose and apply" with Go, which has exactly one option, which requires no special understanding to choose and apply, and gives one exactly what one wants in basically any situation. I am not really a Go proponent. I'm a Haskell user, personally, and Haskell, like Python, has three or four options that require understanding to choose and apply. The difference there being that in Haskell, each one of them actually gets you real parallelism, no fine print necessary. I bring it up to point out that the situation with Python is not a good example of what you might call "intrinsic complexity" (as you seem to be implying) or the Go solution would not be so much simpler, nor is it really an example of there being many better higher-level abstractions, or more of them would resemble Haskell's many high-level options. It is simply a bad situation that produces many poor kludges, and the mentality that everything is fine is (in my opinion) feeding a substantial defection rate to Go.
mietek 13y ago
> Conpig threads still can only run on one core of a processor. This isn't true concurrency. Scaling to 20 million requests per second over 40 cores on a single machine is true concurrency.
- functional_test 13y ago
  You're confusing concurrency [1] and parallelism [2] -- this is addressed on the bullet point before the one you quoted. [1] http://en.wikipedia.org/wiki/Concurrency_(computer_science) http://en.wikipedia.org/wiki/Concurrency_(computer_science) [2] http://en.wikipedia.org/wiki/Parallel_computing http://en.wikipedia.org/wiki/Parallel_computing
- mercurial 13y ago
  Single-core concurrency is concurrency is concurrency (you have different computations occurring concurrently, even if they are not executing at the same time). What it's not is parallelism.
  - stonemetal 13y ago
    Concurrent means at the same time. You can't have things happening concurrently but not at the same time.
    - mirman 13y ago
      Technically no, you can't have two things happening concurrently and but not finishing in the same time period. But what a "thing" and how long a period is are up for grabs. If we choose period to be anything longer than 1 millisecond, then this library will finish executing both things in that time period. http://stackoverflow.com/questions/1050222/concurrency-vs-parallelism-what-is-the-difference http://stackoverflow.com/questions/1050222/concurrency-vs-pa...
    - silverlake 13y ago
      This may help explain the difference: http://blog.golang.org/concurrency-is-not-parallelism http://blog.golang.org/concurrency-is-not-parallelism
rektide 13y ago
The resounding question in my mind is why this, when there is Stackless Python? What's better about this greenthreading impl? http://www.stackless.com/ http://www.stackless.com/
- mirman 13y ago
  This is a library that can be used to supplement any implementation. If I'd had the option to switch us to stackless easily, and I could guarantee it was as fast, worked with all the libraries, and was as stable, I probably wouldn't have written this. I imagine that there are a lot of people in the same boat, where switching interpreters isn't really an option.
erikb 13y ago
Where's the library? I see about 50 lines of code that don't go far beyond a hello world of Gevent and some strangely written nose tests.

30 points by lukes386 13y ago

pekk 13y ago

Since this is a very new project, it is a good time to suggest that you abide by PEP8 (e.g., no 'waitAll') since this would be widely appreciated, and is not easy to fix later on.

mirman 13y ago
waitAll gone. Before I put it in any package managers it will get style guided. This is currently on version -1.0.0.
- derleth 13y ago
  > version -1.0.0. Version negative one? I don't think I've ever seen that before. Usually, the very earliest versions of software are numbered like 0.0.1 or something like that.
  - mirman 13y ago
    - here actually negates the ordering of the numbers in the list. -1.0.0 === 0.0.1.
    - derleth 13y ago
      > - here actually negates the ordering of the numbers in the list. -1.0.0 === 0.0.1. Well, I've certainly never seen that before.
      - ceol 13y ago
        It should be written 0.0.1[::-1]

jacob019 13y ago

I don't really get what advantage this gives me beyond using gevent. There's still no parallelism. The readme describes it as "A solution to concurrency," but I already get that using gevent. Is it just to improve communication between green threads?

mirman 13y ago
Fewer explicit yields & no monkey patching necessary for IO performing libraries.
- MetaCosm 13y ago
  As a former python user, I wish this library had existed 18 months ago -- would have helped with some of the nasty cases you can get caught on with gevent.
jerf 13y ago
It appears to add some amount of Python-bytecode level preemption to gevent, which allows you to hopefully avoid some of the pathological cases of cooperative scheduling. Said pathological cases are only a matter of scale... if your program becomes large enough, you will hit them, eventually. That said, with no offense intended to mirman, I'd really hesitate before using this for anything serious enough to reach that scale in the first place. Gevent, frankly, visibly pushes Python to the limits (and occasionally a bit beyond), trying to also tack on some preemption on an environment not fundamentally expecting it would scare me another notch.
- mirman 13y ago
  No offense taken - both of these are obviously visibly delicate. There is a version in the history that used Greenlet instead of gevent which was potentially a bit less delicate, but it required wrapping of the main file and didn't work with time.sleep, and I didn't feel like it was worth writing my own locks, semaphores, mutexes, pipes and whatnot.
- hogu 13y ago
  what are the pathological cases one runs into with gevent?

fusiongyro 13y ago

> Conpig threads still can only run on one core of a processor. The disillusionment caused by having so many options for non-parallel "concurrency" in Python is, I believe, feeding the high defection rate from Python to Go.

mirman 13y ago
The lack of options for parallel (and non-parallel) concurrency is, along with other things, feeding the high defection rate to many other languages.
- pekk 13y ago
  We've had processes, threads and greenlets for a while now... if anything the problem isn't that there are no options, but too many options that require understanding to choose and apply. Many of the people complaining about this issue don't have a demonstrated problem and could try any simple approach first (if the point is not just to slam Python in favor of something else, from the beginning).
  - MetaCosm 13y ago
    This type of response is why I gave up on Python entirely. Not to pile on you pekk, it isn't your fault, but it is a tone... defensive apologist... "first of all there is no problem, and if there was a problem... it is that Python is too awesome" As someone who has had to ship stuff using multiprocess & gevent to actually meet real world scaling needs -- and integrating them with C code and communicating to a C++ application via ZMQ (inprocess by sharing the object) ... the sad fact is once we started to tackle really hard problems in Python that aren't pre-solved via a nice library all those early advantages fell away and we craved the blessed simplicity of C++ (note: sarcasm, C++ isn't simple, but it was far simpler than the Frankenstein's monster we built).
  - fusiongyro 13y ago
    MetaCosm made the point quite eloquently, but let me juxtapose "too many options that require understanding to choose and apply" with Go, which has exactly one option, which requires no special understanding to choose and apply, and gives one exactly what one wants in basically any situation. I am not really a Go proponent. I'm a Haskell user, personally, and Haskell, like Python, has three or four options that require understanding to choose and apply. The difference there being that in Haskell, each one of them actually gets you real parallelism, no fine print necessary. I bring it up to point out that the situation with Python is not a good example of what you might call "intrinsic complexity" (as you seem to be implying) or the Go solution would not be so much simpler, nor is it really an example of there being many better higher-level abstractions, or more of them would resemble Haskell's many high-level options. It is simply a bad situation that produces many poor kludges, and the mentality that everything is fine is (in my opinion) feeding a substantial defection rate to Go.

mietek 13y ago

> Conpig threads still can only run on one core of a processor. This isn't true concurrency. Scaling to 20 million requests per second over 40 cores on a single machine is true concurrency.

functional_test 13y ago
You're confusing concurrency [1] and parallelism [2] -- this is addressed on the bullet point before the one you quoted. [1] http://en.wikipedia.org/wiki/Concurrency_(computer_science) http://en.wikipedia.org/wiki/Concurrency_(computer_science) [2] http://en.wikipedia.org/wiki/Parallel_computing http://en.wikipedia.org/wiki/Parallel_computing
mercurial 13y ago
Single-core concurrency is concurrency is concurrency (you have different computations occurring concurrently, even if they are not executing at the same time). What it's not is parallelism.
- stonemetal 13y ago
  Concurrent means at the same time. You can't have things happening concurrently but not at the same time.
  - mirman 13y ago
    Technically no, you can't have two things happening concurrently and but not finishing in the same time period. But what a "thing" and how long a period is are up for grabs. If we choose period to be anything longer than 1 millisecond, then this library will finish executing both things in that time period. http://stackoverflow.com/questions/1050222/concurrency-vs-parallelism-what-is-the-difference http://stackoverflow.com/questions/1050222/concurrency-vs-pa...
  - silverlake 13y ago
    This may help explain the difference: http://blog.golang.org/concurrency-is-not-parallelism http://blog.golang.org/concurrency-is-not-parallelism

rektide 13y ago

The resounding question in my mind is why this, when there is Stackless Python? What's better about this greenthreading impl? http://www.stackless.com/ http://www.stackless.com/

mirman 13y ago
This is a library that can be used to supplement any implementation. If I'd had the option to switch us to stackless easily, and I could guarantee it was as fast, worked with all the libraries, and was as stable, I probably wouldn't have written this. I imagine that there are a lot of people in the same boat, where switching interpreters isn't really an option.

erikb 13y ago

Where's the library? I see about 50 lines of code that don't go far beyond a hello world of Gevent and some strangely written nose tests.