8 ms·

A Conversation with Guido about Callbacks

68 points by vgnet 15y ago

ef4 15y ago
Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. It's analogous to manual memory management: it's work that's better to outsource to a compiler or runtime, except in specialized circumstances. For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork. A similar idea for Node is node-fibers. Both let you build exception-safe coroutines.
- anonymoushn 15y ago
  The library referenced in the OP, gevent, uses greenlet rather than stackless to provide this.
- j2labs 15y ago
  Or just use gevent or eventlet. It's not necessary to use a whole different branch of python. On top of that, gevent outperforms both twisted and tornado because the eventloop is libev, meaning it's written in C instead of pure python.
  - gauravk92 15y ago
    Can you find a source for that, I'd love to read about actual performance statistics between anything and the likes ok twisted and such.
    - jeangenie 15y ago
      http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g.html http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g....
      - j2labs 15y ago
        This link is flawed. It points towards eventlet but gevent is significantly better. Not sure why Ted felt otherwise, but I speculate that it's because at the time of writing gevent was considered incomplete compared to eventlet. To be clear: gevent is significantly faster than eventlet and offers the same functionality.
    - j2labs 15y ago
      http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
    - glyph 15y ago
      Twisted's event loop is faster than gevent and tornado. http://cyclone.io/ http://cyclone.io/
      - j2labs 15y ago
        This test is for a pointlessly weak concurrency level of 25. Under serious load the difference is much more gigantic showing that Twisted can't handle the same load as gevent.
    - glyph 15y ago
      Twisted has a benchmark suite and release-to-release tracking of performance information, so that we can be sure optimizations have a positive impact. http://speed.twistedmatrix.com/timeline/ http://speed.twistedmatrix.com/timeline/
  - sausagefeet 15y ago
    The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is.
    - j2labs 15y ago
      > The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. It is intended to create an environment similar to preemptive threads, yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. > - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. This is pure speculation. If Python's multicore handling was better (existed?), it's likely gevent would be implemented similar to other systems where you have threads allocated to each core and the concurrency primitives can then be executed on cores according to some algorithm. > - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. While this is true, I don't think it is likely to occur. Gevent's creator agrees: http://www.gevent.org/intro.html#cooperative-multitasking http://www.gevent.org/intro.html#cooperative-multitasking > On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is. You're an Erlang programmer. You have to have that opinion. (sausagefeet and I are friends IRL so I know his biases)
      - sausagefeet 15y ago
        > Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. The weakness of callbacks is the lack of multicore support. It is not pure speculation that gevent simply won't work in a preemptive environment if it wants to utilize multiple cores. The sentence right about this makes my point: > yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Why is that? It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. You can depend on 'my_dict[foo] += 1' to happen atomically. If you want to utilize multiple cores with gevent, as code using it is written now, you can't because all of the code out there depend on operations between I/O events happening atomically. This isn't a secret in the callback/gevent world. On the contrary, it's presented as a selling point. No longer worry about those locking primitives those real threading libraries give you because we are single threaded. (Ocaml/Lwt has the same problem). > While this is true, I don't think it is likely to occur I'll concede it's far less likely. I'm curious how large gevent code bases are in the wild, a large application that one cannot fit in their entire head could very well have a lot of the same race conditions that a preemptively threaded application has. > You're an Erlang programmer. You have to have that opinion Really? I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought.
  - glyph 15y ago
    Python outperforms C in many cases. http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-again-string.html http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-a...
    - j2labs 15y ago
      Will be awesome to see more of what we get from pypy. Eventlet will probably gain a lot too, for those with a taste for the implicit eventloop.
- voidr 15y ago
  > Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. UI programming is callback heavy, but I doubt that any language will support "better" ways for that. So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. Server-side people just need to figure out how to use callbacks properly and everything is gonna be fine. All people need to understand is to nest callbacks properly, nested callbacks are the things that confuse most people. > For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork. You can't just abstract away every programing feature that seems hard and expect everything to be fine, it just doesn't work that way, Java developers are still suffering because Java tries real hard to abstract memory management.
  - deleted 15y ago
    [deleted]
  - cdavid 15y ago
    I don't think the comparison with UI is so apt. Sure, they both use callbacks, but the granularity is quite different. In UI callbacks, you don't need to insert your own "scheduling points" as you need in explicity asynchronous programming. Mostly, you write functions that reacts to the events generated by the UI, but within those event handler, your own code is mostly synchronous. I wonder why you mean by using callbacks properly ? The critical point mentioned by Guido is error/corner-case handling: the idiomatic callback-based code I have seen becomes difficult very quickly. I have not seen a case where e.g. deferred-based code ala twisted has been simpler than gevent.
  - gruseom 15y ago
    So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. I don't think that's right. There is a huge difference. UI callbacks ("onClick" and the like) execute synchronously. But the callbacks in event servers execute at completely different times. Indeed, they're typically nested inside code that has long since terminated. This is what causes the learning curve, not grokking events or functions-as-values, which any half-decent programmer can easily do in the unlikely event that they didn't a long time ago. Indeed, the way we write asynchronous callbacks is cognitively dissonant with everything we've been trained to understand about scoping. The idea of a nested scope that doesn't mean "contained within" or "covered by", but rather "at some unknown time later", is inconsistent enough with how programming languages normally work that it can easily trip you up even after you get good at it. Asynchronous programming is a very different thing than e.g. browser UIs in Javascript. I've programmed those for years. It still took me a long time to get my head around async servers.
- chipsy 15y ago
  Agreed, roughly. Nobody particularly likes reading callbacky code, but it has a lot of power. What we don't have enough of is "structured alternatives" for our "goto."
jQueryIsAwesome 15y ago
Slightly on-topic, a post of mine about how to have zero nested callbacks in Javascript: http://javascriptisawesome.blogspot.com/2012/03/zero-nested-callbacks.html http://javascriptisawesome.blogspot.com/2012/03/zero-nested-...
sausagefeet 15y ago
Seems like one has to be very out of touch with modern approaches to concurrency to like callbacks. Sounds flamey, but I'm just not sure how one can look at Go, or Erlang, or even Ocaml/Lwt with syntax extensions, and think "yes, callbacks are superior to this".
- lobster_johnson 15y ago
  Surely Erlang uses callbacks heavily, in the sense that you pass functions around. Supervisors, for example, depend heavily on callbacks to receive notifications when something happens to the processes it supervises.
  - sausagefeet 15y ago
    You're confusing things. Callbacks are not the unit of concurrency in Erlang, a process is, which is a sequence of steps. You implement callbacks for behaviors in Erlang, but that is an interface system, not a concurrency unit.
    - lobster_johnson 15y ago
      Well, I was referring to the "interface system", which is a pretty significant detail when you are comparing two programming approaches. The fact that Erlang has a richer set of concurrency semantics — that, among other things, let you quite easily manage a graph of dependencies so that your callback logic becomes much simpler — doesn't change the fact that it relies on callbacks (something "calls back" a function in order to supply information asynchronously) as a fundamental unit of programming. I don't know gevent in detail, but I suspect that it's possible to create an Erlang-style process/supervisor-style framework on top of it, albeit with less elegance and more syntax than Erlang. That would change the underlying concurrency mechanism, but the interface — callbacks — would remain very similar.
      - sausagefeet 15y ago
        A callback in Twisted/Tornado/Node is a handler for the result of some asynchronous action happening. "Download this webpage and call ME with the results". A callback in Erlang is some functionality in a service a process provides. You call a callback to perform some task then it gives it back to you, possibly synchronously in the case of gen_server:call. Using a function in the Twisted/Node style would not make sense since a process is the fundamental unit of concurrency in Erlang so message passing is the only way to communicate back and forth. It would simply make no sense to ask a service in Erlang to "call THIS function when you're done" in most cases. The function would run in the wrong process and this isn't how you construct Erlang code. The semantics are completely different. A callback in Erlang is much closer to a webservices, for example, where you ask for some work to be done and it gives you the result. As far as I know, Gevent does not use the 'callback' terminology for anything. If you want to redefine "callback" to be "any method to communicate between two piece of code" then you're free to, but we aren't talking about the same thing then and nobody else will know what you're talking about.
aufreak3 15y ago
Given how much callback based stuff exists in the Javascript world -- GUIs, Node.js and some recent W3C specs like the FileSystem API -- I thought it ought to be possible to write a simple sequencing function that can take care of this in the JS world. The key to a useful sequencing function is that it should let you customize the sequencing of the actions and recovery techniques pretty easily. Here is my attempt at it - https://gist.github.com/2192413 https://gist.github.com/2192413 Thoughts?
bluesnowmonkey 15y ago
I'm waiting for the day when everybody realizes that callbacks and preemptive threading both suck, relative to automatic CPS transformations. http://matt.might.net/articles/cps-conversion/ http://matt.might.net/articles/cps-conversion/

68 points by vgnet 15y ago

ef4 15y ago

Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. It's analogous to manual memory management: it's work that's better to outsource to a compiler or runtime, except in specialized circumstances. For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork. A similar idea for Node is node-fibers. Both let you build exception-safe coroutines.

anonymoushn 15y ago
The library referenced in the OP, gevent, uses greenlet rather than stackless to provide this.
j2labs 15y ago
Or just use gevent or eventlet. It's not necessary to use a whole different branch of python. On top of that, gevent outperforms both twisted and tornado because the eventloop is libev, meaning it's written in C instead of pure python.
- gauravk92 15y ago
  Can you find a source for that, I'd love to read about actual performance statistics between anything and the likes ok twisted and such.
  - jeangenie 15y ago
    http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g.html http://teddziuba.com/2010/02/eventlet-asynchronous-io-for-g....
    - j2labs 15y ago
      This link is flawed. It points towards eventlet but gevent is significantly better. Not sure why Ted felt otherwise, but I speculate that it's because at the time of writing gevent was considered incomplete compared to eventlet. To be clear: gevent is significantly faster than eventlet and offers the same functionality.
  - j2labs 15y ago
    http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
  - glyph 15y ago
    Twisted's event loop is faster than gevent and tornado. http://cyclone.io/ http://cyclone.io/
    - j2labs 15y ago
      This test is for a pointlessly weak concurrency level of 25. Under serious load the difference is much more gigantic showing that Twisted can't handle the same load as gevent.
  - glyph 15y ago
    Twisted has a benchmark suite and release-to-release tracking of performance information, so that we can be sure optimizations have a positive impact. http://speed.twistedmatrix.com/timeline/ http://speed.twistedmatrix.com/timeline/
- sausagefeet 15y ago
  The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is.
  - j2labs 15y ago
    > The main problem I see with gevent/eventlet is it has many of the weaknesses of callback and preemptive threaded code with none of the strengths. It is intended to create an environment similar to preemptive threads, yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. > - It can't use multiple cores, even if Python got the ability in the future, gevent code simply wouldn't work as intended. This is pure speculation. If Python's multicore handling was better (existed?), it's likely gevent would be implemented similar to other systems where you have threads allocated to each core and the concurrency primitives can then be executed on cores according to some algorithm. > - While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. While this is true, I don't think it is likely to occur. Gevent's creator agrees: http://www.gevent.org/intro.html#cooperative-multitasking http://www.gevent.org/intro.html#cooperative-multitasking > On the net, gevent may be better than Twisted or Tornado, but I think that is saying more about how much those suck than how good gevent is. You're an Erlang programmer. You have to have that opinion. (sausagefeet and I are friends IRL so I know his biases)
    - sausagefeet 15y ago
      > Regarding the weaknesses of callbacks, I disagree and would like to see evidence supporting your case. The weakness of callbacks is the lack of multicore support. It is not pure speculation that gevent simply won't work in a preemptive environment if it wants to utilize multiple cores. The sentence right about this makes my point: > yet due to it being a networking library it is unlikely gevent users will have to use the semaphore and locking primitives provided with gevent. Why is that? It's because gevent context switches on I/O events, not preemptively, so you have some control over when context switches happen. You can depend on 'my_dict[foo] += 1' to happen atomically. If you want to utilize multiple cores with gevent, as code using it is written now, you can't because all of the code out there depend on operations between I/O events happening atomically. This isn't a secret in the callback/gevent world. On the contrary, it's presented as a selling point. No longer worry about those locking primitives those real threading libraries give you because we are single threaded. (Ocaml/Lwt has the same problem). > While this is true, I don't think it is likely to occur I'll concede it's far less likely. I'm curious how large gevent code bases are in the wild, a large application that one cannot fit in their entire head could very well have a lot of the same race conditions that a preemptively threaded application has. > You're an Erlang programmer. You have to have that opinion Really? I didn't realize when I started using Erlang I signed a contract saying that. I also write Ocaml, so I guess any criticism I levy against Java is a product of that too, not rational thought.
- glyph 15y ago
  Python outperforms C in many cases. http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-again-string.html http://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-a...
  - j2labs 15y ago
    Will be awesome to see more of what we get from pypy. Eventlet will probably gain a lot too, for those with a taste for the implicit eventloop.
voidr 15y ago
> Callback-heavy programming is what you get when your language doesn't support better ways of controlling context. UI programming is callback heavy, but I doubt that any language will support "better" ways for that. So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. Server-side people just need to figure out how to use callbacks properly and everything is gonna be fine. All people need to understand is to nest callbacks properly, nested callbacks are the things that confuse most people. > For Python, you can get the benefits of asynchronous IO without callback hell if you use the Stackless fork. You can't just abstract away every programing feature that seems hard and expect everything to be fine, it just doesn't work that way, Java developers are still suffering because Java tries real hard to abstract memory management.
- deleted 15y ago
  [deleted]
- cdavid 15y ago
  I don't think the comparison with UI is so apt. Sure, they both use callbacks, but the granularity is quite different. In UI callbacks, you don't need to insert your own "scheduling points" as you need in explicity asynchronous programming. Mostly, you write functions that reacts to the events generated by the UI, but within those event handler, your own code is mostly synchronous. I wonder why you mean by using callbacks properly ? The critical point mentioned by Guido is error/corner-case handling: the idiomatic callback-based code I have seen becomes difficult very quickly. I have not seen a case where e.g. deferred-based code ala twisted has been simpler than gevent.
- gruseom 15y ago
  So basically UI programmers are used to callbacks, others not that much, and now that event-loop servers are becoming popular, we see confusion from people not familiar with events. I don't think that's right. There is a huge difference. UI callbacks ("onClick" and the like) execute synchronously. But the callbacks in event servers execute at completely different times. Indeed, they're typically nested inside code that has long since terminated. This is what causes the learning curve, not grokking events or functions-as-values, which any half-decent programmer can easily do in the unlikely event that they didn't a long time ago. Indeed, the way we write asynchronous callbacks is cognitively dissonant with everything we've been trained to understand about scoping. The idea of a nested scope that doesn't mean "contained within" or "covered by", but rather "at some unknown time later", is inconsistent enough with how programming languages normally work that it can easily trip you up even after you get good at it. Asynchronous programming is a very different thing than e.g. browser UIs in Javascript. I've programmed those for years. It still took me a long time to get my head around async servers.
chipsy 15y ago
Agreed, roughly. Nobody particularly likes reading callbacky code, but it has a lot of power. What we don't have enough of is "structured alternatives" for our "goto."

jQueryIsAwesome 15y ago

Slightly on-topic, a post of mine about how to have zero nested callbacks in Javascript: http://javascriptisawesome.blogspot.com/2012/03/zero-nested-callbacks.html http://javascriptisawesome.blogspot.com/2012/03/zero-nested-...

sausagefeet 15y ago

Seems like one has to be very out of touch with modern approaches to concurrency to like callbacks. Sounds flamey, but I'm just not sure how one can look at Go, or Erlang, or even Ocaml/Lwt with syntax extensions, and think "yes, callbacks are superior to this".

lobster_johnson 15y ago
Surely Erlang uses callbacks heavily, in the sense that you pass functions around. Supervisors, for example, depend heavily on callbacks to receive notifications when something happens to the processes it supervises.
- sausagefeet 15y ago
  You're confusing things. Callbacks are not the unit of concurrency in Erlang, a process is, which is a sequence of steps. You implement callbacks for behaviors in Erlang, but that is an interface system, not a concurrency unit.
  - lobster_johnson 15y ago
    Well, I was referring to the "interface system", which is a pretty significant detail when you are comparing two programming approaches. The fact that Erlang has a richer set of concurrency semantics — that, among other things, let you quite easily manage a graph of dependencies so that your callback logic becomes much simpler — doesn't change the fact that it relies on callbacks (something "calls back" a function in order to supply information asynchronously) as a fundamental unit of programming. I don't know gevent in detail, but I suspect that it's possible to create an Erlang-style process/supervisor-style framework on top of it, albeit with less elegance and more syntax than Erlang. That would change the underlying concurrency mechanism, but the interface — callbacks — would remain very similar.
    - sausagefeet 15y ago
      A callback in Twisted/Tornado/Node is a handler for the result of some asynchronous action happening. "Download this webpage and call ME with the results". A callback in Erlang is some functionality in a service a process provides. You call a callback to perform some task then it gives it back to you, possibly synchronously in the case of gen_server:call. Using a function in the Twisted/Node style would not make sense since a process is the fundamental unit of concurrency in Erlang so message passing is the only way to communicate back and forth. It would simply make no sense to ask a service in Erlang to "call THIS function when you're done" in most cases. The function would run in the wrong process and this isn't how you construct Erlang code. The semantics are completely different. A callback in Erlang is much closer to a webservices, for example, where you ask for some work to be done and it gives you the result. As far as I know, Gevent does not use the 'callback' terminology for anything. If you want to redefine "callback" to be "any method to communicate between two piece of code" then you're free to, but we aren't talking about the same thing then and nobody else will know what you're talking about.

aufreak3 15y ago

Given how much callback based stuff exists in the Javascript world -- GUIs, Node.js and some recent W3C specs like the FileSystem API -- I thought it ought to be possible to write a simple sequencing function that can take care of this in the JS world. The key to a useful sequencing function is that it should let you customize the sequencing of the actions and recovery techniques pretty easily. Here is my attempt at it - https://gist.github.com/2192413 https://gist.github.com/2192413 Thoughts?

bluesnowmonkey 15y ago

I'm waiting for the day when everybody realizes that callbacks and preemptive threading both suck, relative to automatic CPS transformations. http://matt.might.net/articles/cps-conversion/ http://matt.might.net/articles/cps-conversion/