4 ms·

I don't really get what advantage this gives me beyond using gevent. There's still no parallelism. The readme describes it as "A solution to concurrency," but

I don't really get what advantage this gives me beyond using gevent. There's still no parallelism. The readme describes it as "A solution to concurrency," but I already get that using gevent. Is it just to improve communication between green threads?

mirman 13y ago
Fewer explicit yields & no monkey patching necessary for IO performing libraries.
- MetaCosm 13y ago
  As a former python user, I wish this library had existed 18 months ago -- would have helped with some of the nasty cases you can get caught on with gevent.
jerf 13y ago
It appears to add some amount of Python-bytecode level preemption to gevent, which allows you to hopefully avoid some of the pathological cases of cooperative scheduling. Said pathological cases are only a matter of scale... if your program becomes large enough, you will hit them, eventually. That said, with no offense intended to mirman, I'd really hesitate before using this for anything serious enough to reach that scale in the first place. Gevent, frankly, visibly pushes Python to the limits (and occasionally a bit beyond), trying to also tack on some preemption on an environment not fundamentally expecting it would scare me another notch.
- mirman 13y ago
  No offense taken - both of these are obviously visibly delicate. There is a version in the history that used Greenlet instead of gevent which was potentially a bit less delicate, but it required wrapping of the main file and didn't work with time.sleep, and I didn't feel like it was worth writing my own locks, semaphores, mutexes, pipes and whatnot.
- hogu 13y ago
  what are the pathological cases one runs into with gevent?
  - jerf 13y ago
    With a cooperative scheduler, you will, sooner or later, experience some form of starvation. The most obvious is the process that just infinitely loops, but less obvious situations will end up popping up too; calls that you thought were handled by the event loop but turn out to be blocking and add up when you start calling them at scale, strange behavior when you have a set of processes that turn out to yield far less often than you thought and the system starts behaving with much higher latency than it should if you get too many of them, and all kinds of such manifestations. There's also the inability to create things that watch other things; if something does go spinning off into infinity, nothing else gets to run to kill it. You can hack around many of them, but you eventually hit a wall, and the effort of the hack increases rapidly. Note I didn't really use gevent in this reply, this is just about cooperative scheduling. There's a reason why we've all but completely abandoned it at the OS level for things we'd call "computers" (as opposed to "embedded systems" or "microcontrollers", etc). I tend to consider cooperative scheduling an enormous red flag in any system that uses it... and yes, that completely and fully includes the currently-popular runtimes that use it.
    - anonymoushn 13y ago
      It sounds like this sort of problem comes from using a cooperative scheduler to implement concurrency of arbitrary routines rather than control flow. I haven't been in a situation in which it would even be possible for something to yield less often than I expect, because I expect it to run until it yields. Similarly I don't often find that subroutines return too infrequently because I expect them to run until they return. This library is probably nice for the places I would otherwise use threads.
      - jerf 13y ago
        You will eventually, at scale, be wrong about that. To have full and correct knowledge of exactly how long your code takes to run sufficient to do this sort of scheduling correctly, by hand, in advance of running it, is basically equivalent to claiming that you never need to profile code because you already know exactly how long it takes. And it is well known and established to my satisfaction that even absolute, total experts in a field will still often be surprised about what actually comes out of a profiler, even in code strictly in their domains. You may well be right most of the time... but that is all you can hope for.
erikb 13y ago
It basically is 50 lines of code that implement some stuff from gevent... So yes, it's more of a joke I guess.
- mirman 13y ago
  So number of lines of code makes a library a joke? One of the most commonly used Haskell libraries is only 25. http://hackage.haskell.org/packages/archive/forkable-monad/0.1.1/doc/html/src/Control-Concurrent-Forkable.html#ForkableMonad http://hackage.haskell.org/packages/archive/forkable-monad/0... Also, this doesn't implement some stuff from gevent, it implements some stuff over and using gevent.
  - erikb 13y ago
    Yes that's exactly what I'm saying. If external code is less then some hundred lines of code, it would be better suited as a tutorial and people would be better at implementing those lines themselves. And yes, that's only my opinion.
    - mirman 13y ago
      Do you have any reasoning behind that opinion, or is it just unfounded religious zeal? Why should somebody who wants to use threading have to know how to write a preemptive scheduler? If you want a tutorial in addition to the library thats fine, but many good tutorials also release their code as libraries. In open source world, releasing code as a library does not mean having to keep people from understanding the workings of that code. Why should people HAVE to learn how the code works to use it though?

mirman 13y ago

Fewer explicit yields & no monkey patching necessary for IO performing libraries.

MetaCosm 13y ago
As a former python user, I wish this library had existed 18 months ago -- would have helped with some of the nasty cases you can get caught on with gevent.

jerf 13y ago

It appears to add some amount of Python-bytecode level preemption to gevent, which allows you to hopefully avoid some of the pathological cases of cooperative scheduling. Said pathological cases are only a matter of scale... if your program becomes large enough, you will hit them, eventually. That said, with no offense intended to mirman, I'd really hesitate before using this for anything serious enough to reach that scale in the first place. Gevent, frankly, visibly pushes Python to the limits (and occasionally a bit beyond), trying to also tack on some preemption on an environment not fundamentally expecting it would scare me another notch.

mirman 13y ago
No offense taken - both of these are obviously visibly delicate. There is a version in the history that used Greenlet instead of gevent which was potentially a bit less delicate, but it required wrapping of the main file and didn't work with time.sleep, and I didn't feel like it was worth writing my own locks, semaphores, mutexes, pipes and whatnot.
hogu 13y ago
what are the pathological cases one runs into with gevent?
- jerf 13y ago
  With a cooperative scheduler, you will, sooner or later, experience some form of starvation. The most obvious is the process that just infinitely loops, but less obvious situations will end up popping up too; calls that you thought were handled by the event loop but turn out to be blocking and add up when you start calling them at scale, strange behavior when you have a set of processes that turn out to yield far less often than you thought and the system starts behaving with much higher latency than it should if you get too many of them, and all kinds of such manifestations. There's also the inability to create things that watch other things; if something does go spinning off into infinity, nothing else gets to run to kill it. You can hack around many of them, but you eventually hit a wall, and the effort of the hack increases rapidly. Note I didn't really use gevent in this reply, this is just about cooperative scheduling. There's a reason why we've all but completely abandoned it at the OS level for things we'd call "computers" (as opposed to "embedded systems" or "microcontrollers", etc). I tend to consider cooperative scheduling an enormous red flag in any system that uses it... and yes, that completely and fully includes the currently-popular runtimes that use it.
  - anonymoushn 13y ago
    It sounds like this sort of problem comes from using a cooperative scheduler to implement concurrency of arbitrary routines rather than control flow. I haven't been in a situation in which it would even be possible for something to yield less often than I expect, because I expect it to run until it yields. Similarly I don't often find that subroutines return too infrequently because I expect them to run until they return. This library is probably nice for the places I would otherwise use threads.
    - jerf 13y ago
      You will eventually, at scale, be wrong about that. To have full and correct knowledge of exactly how long your code takes to run sufficient to do this sort of scheduling correctly, by hand, in advance of running it, is basically equivalent to claiming that you never need to profile code because you already know exactly how long it takes. And it is well known and established to my satisfaction that even absolute, total experts in a field will still often be surprised about what actually comes out of a profiler, even in code strictly in their domains. You may well be right most of the time... but that is all you can hope for.

erikb 13y ago

It basically is 50 lines of code that implement some stuff from gevent... So yes, it's more of a joke I guess.

mirman 13y ago
So number of lines of code makes a library a joke? One of the most commonly used Haskell libraries is only 25. http://hackage.haskell.org/packages/archive/forkable-monad/0.1.1/doc/html/src/Control-Concurrent-Forkable.html#ForkableMonad http://hackage.haskell.org/packages/archive/forkable-monad/0... Also, this doesn't implement some stuff from gevent, it implements some stuff over and using gevent.
- erikb 13y ago
  Yes that's exactly what I'm saying. If external code is less then some hundred lines of code, it would be better suited as a tutorial and people would be better at implementing those lines themselves. And yes, that's only my opinion.
  - mirman 13y ago
    Do you have any reasoning behind that opinion, or is it just unfounded religious zeal? Why should somebody who wants to use threading have to know how to write a preemptive scheduler? If you want a tutorial in addition to the library thats fine, but many good tutorials also release their code as libraries. In open source world, releasing code as a library does not mean having to keep people from understanding the workings of that code. Why should people HAVE to learn how the code works to use it though?