6 ms·

Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. It's analogous to manual memory management: it

by ef4 15y ago

Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. It's analogous to manual memory management: it's work that's better to outsource to a compiler or runtime, except in specialized circumstances. For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork. A similar idea for Node is node-fibers. Both let you build exception-safe coroutines.

anonymoushn 15y ago
The library referenced in the OP, gevent, uses greenlet rather than stackless to provide this.
j2labs 15y ago
Or just use gevent or eventlet. It's not necessary to use a whole different branch of python. On top of that, gevent outperforms both twisted and tornado because the eventloop is libev, meaning it's written in C instead of pure python.
- gauravk92 15y ago
  Can you find a source for that, I'd love to read about actual performance statistics between anything and the likes ok twisted and such.
  - jeangenie 15y ago
    http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g.html http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g....
    - j2labs 15y ago
      This link is flawed. It points towards eventlet but gevent is significantly better. Not sure why Ted felt otherwise, but I speculate that it's because at the time of writing gevent was considered incomplete compared to eventlet. To be clear: gevent is significantly faster than eventlet and offers the same functionality.
  - j2labs 15y ago
    http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
  - glyph 15y ago
    Twisted's event loop is faster than gevent and tornado. http://cyclone.io/ http://cyclone.io/
    - j2labs 15y ago
      This test is for a pointlessly weak concurrency level of 25. Under serious load the difference is much more gigantic showing that Twisted can't handle the same load as gevent.
  - glyph 15y ago
    Twisted has a benchmark suite and release-to-release tracking of performance information, so that we can be sure optimizations have a positive impact. http://speed.twistedmatrix.com/timeline/ http://speed.twistedmatrix.com/timeline/
- sausagefeet 15y ago
  The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is.
  - j2labs 15y ago
    > The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. It is intended to create an environment similar to preemptive threads, yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. > - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. This is pure speculation. If Python's multicore handling was better (existed?), it's likely gevent would be implemented similar to other systems where you have threads allocated to each core and the concurrency primitives can then be executed on cores according to some algorithm. > - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. While this is true, I don't think it is likely to occur. Gevent's creator agrees: http://www.gevent.org/intro.html#cooperative-multitasking http://www.gevent.org/intro.html#cooperative-multitasking > On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is. You're an Erlang programmer. You have to have that opinion. (sausagefeet and I are friends IRL so I know his biases)
    - sausagefeet 15y ago
      > Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. The weakness of callbacks is the lack of multicore support. It is not pure speculation that gevent simply won't work in a preemptive environment if it wants to utilize multiple cores. The sentence right about this makes my point: > yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Why is that? It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. You can depend on 'my_dict[foo] += 1' to happen atomically. If you want to utilize multiple cores with gevent, as code using it is written now, you can't because all of the code out there depend on operations between I/O events happening atomically. This isn't a secret in the callback/gevent world. On the contrary, it's presented as a selling point. No longer worry about those locking primitives those real threading libraries give you because we are single threaded. (Ocaml/Lwt has the same problem). > While this is true, I don't think it is likely to occur I'll concede it's far less likely. I'm curious how large gevent code bases are in the wild, a large application that one cannot fit in their entire head could very well have a lot of the same race conditions that a preemptively threaded application has. > You're an Erlang programmer. You have to have that opinion Really? I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought.
- glyph 15y ago
  Python outperforms C in many cases. http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-again-string.html http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-a...
  - j2labs 15y ago
    Will be awesome to see more of what we get from pypy. Eventlet will probably gain a lot too, for those with a taste for the implicit eventloop.
voidr 15y ago
> Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. UI programming is callback heavy, but I doubt that any language will support "better" ways for that. So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. Server-side people just need to figure out how to use callbacks properly and everything is gonna be fine. All people need to understand is to nest callbacks properly, nested callbacks are the things that confuse most people. > For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork. You can't just abstract away every programing feature that seems hard and expect everything to be fine, it just doesn't work that way, Java developers are still suffering because Java tries real hard to abstract memory management.
- deleted 15y ago
  [deleted]
- cdavid 15y ago
  I don't think the comparison with UI is so apt. Sure, they both use callbacks, but the granularity is quite different. In UI callbacks, you don't need to insert your own "scheduling points" as you need in explicity asynchronous programming. Mostly, you write functions that reacts to the events generated by the UI, but within those event handler, your own code is mostly synchronous. I wonder why you mean by using callbacks properly ? The critical point mentioned by Guido is error/corner-case handling: the idiomatic callback-based code I have seen becomes difficult very quickly. I have not seen a case where e.g. deferred-based code ala twisted has been simpler than gevent.
- gruseom 15y ago
  So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. I don't think that's right. There is a huge difference. UI callbacks ("onClick" and the like) execute synchronously. But the callbacks in event servers execute at completely different times. Indeed, they're typically nested inside code that has long since terminated. This is what causes the learning curve, not grokking events or functions-as-values, which any half-decent programmer can easily do in the unlikely event that they didn't a long time ago. Indeed, the way we write asynchronous callbacks is cognitively dissonant with everything we've been trained to understand about scoping. The idea of a nested scope that doesn't mean "contained within" or "covered by", but rather "at some unknown time later", is inconsistent enough with how programming languages normally work that it can easily trip you up even after you get good at it. Asynchronous programming is a very different thing than e.g. browser UIs in Javascript. I've programmed those for years. It still took me a long time to get my head around async servers.
chipsy 15y ago
Agreed, roughly. Nobody particularly likes reading callbacky code, but it has a lot of power. What we don't have enough of is "structured alternatives" for our "goto."

by ef4 15y ago

anonymoushn 15y ago

The library referenced in the OP, gevent, uses greenlet rather than stackless to provide this.

j2labs 15y ago

Or just use gevent or eventlet. It's not necessary to use a whole different branch of python. On top of that, gevent outperforms both twisted and tornado because the eventloop is libev, meaning it's written in C instead of pure python.

gauravk92 15y ago
Can you find a source for that, I'd love to read about actual performance statistics between anything and the likes ok twisted and such.
- jeangenie 15y ago
  http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g.html http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g....
  - j2labs 15y ago
    This link is flawed. It points towards eventlet but gevent is significantly better. Not sure why Ted felt otherwise, but I speculate that it's because at the time of writing gevent was considered incomplete compared to eventlet. To be clear: gevent is significantly faster than eventlet and offers the same functionality.
- j2labs 15y ago
  http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
- glyph 15y ago
  Twisted's event loop is faster than gevent and tornado. http://cyclone.io/ http://cyclone.io/
  - j2labs 15y ago
    This test is for a pointlessly weak concurrency level of 25. Under serious load the difference is much more gigantic showing that Twisted can't handle the same load as gevent.
- glyph 15y ago
  Twisted has a benchmark suite and release-to-release tracking of performance information, so that we can be sure optimizations have a positive impact. http://speed.twistedmatrix.com/timeline/ http://speed.twistedmatrix.com/timeline/
sausagefeet 15y ago
The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is.
- j2labs 15y ago
  > The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. It is intended to create an environment similar to preemptive threads, yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. > - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. This is pure speculation. If Python's multicore handling was better (existed?), it's likely gevent would be implemented similar to other systems where you have threads allocated to each core and the concurrency primitives can then be executed on cores according to some algorithm. > - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. While this is true, I don't think it is likely to occur. Gevent's creator agrees: http://www.gevent.org/intro.html#cooperative-multitasking http://www.gevent.org/intro.html#cooperative-multitasking > On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is. You're an Erlang programmer. You have to have that opinion. (sausagefeet and I are friends IRL so I know his biases)
  - sausagefeet 15y ago
    > Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. The weakness of callbacks is the lack of multicore support. It is not pure speculation that gevent simply won't work in a preemptive environment if it wants to utilize multiple cores. The sentence right about this makes my point: > yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Why is that? It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. You can depend on 'my_dict[foo] += 1' to happen atomically. If you want to utilize multiple cores with gevent, as code using it is written now, you can't because all of the code out there depend on operations between I/O events happening atomically. This isn't a secret in the callback/gevent world. On the contrary, it's presented as a selling point. No longer worry about those locking primitives those real threading libraries give you because we are single threaded. (Ocaml/Lwt has the same problem). > While this is true, I don't think it is likely to occur I'll concede it's far less likely. I'm curious how large gevent code bases are in the wild, a large application that one cannot fit in their entire head could very well have a lot of the same race conditions that a preemptively threaded application has. > You're an Erlang programmer. You have to have that opinion Really? I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought.
glyph 15y ago
Python outperforms C in many cases. http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-again-string.html http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-a...
- j2labs 15y ago
  Will be awesome to see more of what we get from pypy. Eventlet will probably gain a lot too, for those with a taste for the implicit eventloop.

voidr 15y ago

> Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. UI programming is callback heavy, but I doubt that any language will support "better" ways for that. So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. Server-side people just need to figure out how to use callbacks properly and everything is gonna be fine. All people need to understand is to nest callbacks properly, nested callbacks are the things that confuse most people. > For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork. You can't just abstract away every programing feature that seems hard and expect everything to be fine, it just doesn't work that way, Java developers are still suffering because Java tries real hard to abstract memory management.

deleted 15y ago
[deleted]
cdavid 15y ago
I don't think the comparison with UI is so apt. Sure, they both use callbacks, but the granularity is quite different. In UI callbacks, you don't need to insert your own "scheduling points" as you need in explicity asynchronous programming. Mostly, you write functions that reacts to the events generated by the UI, but within those event handler, your own code is mostly synchronous. I wonder why you mean by using callbacks properly ? The critical point mentioned by Guido is error/corner-case handling: the idiomatic callback-based code I have seen becomes difficult very quickly. I have not seen a case where e.g. deferred-based code ala twisted has been simpler than gevent.
gruseom 15y ago
So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. I don't think that's right. There is a huge difference. UI callbacks ("onClick" and the like) execute synchronously. But the callbacks in event servers execute at completely different times. Indeed, they're typically nested inside code that has long since terminated. This is what causes the learning curve, not grokking events or functions-as-values, which any half-decent programmer can easily do in the unlikely event that they didn't a long time ago. Indeed, the way we write asynchronous callbacks is cognitively dissonant with everything we've been trained to understand about scoping. The idea of a nested scope that doesn't mean "contained within" or "covered by", but rather "at some unknown time later", is inconsistent enough with how programming languages normally work that it can easily trip you up even after you get good at it. Asynchronous programming is a very different thing than e.g. browser UIs in Javascript. I've programmed those for years. It still took me a long time to get my head around async servers.

chipsy 15y ago

Agreed, roughly. Nobody particularly likes reading callbacky code, but it has a lot of power. What we don't have enough of is "structured alternatives" for our "goto."