15 ms·

Guido van Rossum Deconstructing Twisted's Deferreds

225 points by thezilch 13y ago

pkinsky 13y ago
I'm probably missing something, but isn't Idea 4: Chaining Deferreds just flatMap over Futures? Scala may have spoiled me somewhat.
- zeckalpha 13y ago
  And errbacks are really just the Maybe monad.
  - seanmcdirmid 13y ago
    None of these future/aync systems are very novel, but there is demand for them. Eventually most of them will fail and a few of the good ones will survive. And ya, they will resemble functional programming language X and Y in some of their features, because they aren't meant to be novel.
  - nostrademons 13y ago
    Errbacks are the Error monad. Failure is the Maybe monad (or errbacks with callback=None).
- noelwelsh 13y ago
  Yes, the Twisted Deferred is a not-quite-a-monad. The API would be simpler if it was, but I guess the original developers didn't have that insight at the time they were writing it.
- dons 13y ago
  Calling monadic bind "flatmap" makes me cry inside.
  - DanWaterworth 13y ago
    Indeed, it's like snatching defeat from the jaws of victory. On a related note, Guido van Rossum gives monads 5/5 stars... sort of.
aidos 13y ago
I've only looked at Twisted briefly before but I feel like I learnt a lot from this run-through. Guido writes so well; he is able to thoughtfully deconstruct ideas with a critical eye while still praising the bits he likes. I think we could all learn something from the way he has approached this. It's definitely a model I will aspire to follow while appraising unfamiliar code.
Pxtl 13y ago
The bit with passing the result along in the chain of deferrals reminds me a bit of some of the fun stuff you can do with Events in C#.
- Locke1689 13y ago
  Yes, although that's considered bad practice with async. I previously used Twisted to implement a speculative model of new Internet architectures for a graduate class when I was in my distributed systems phase in college. One thing that Guido mentions is that Twisted Deferred computations generally rely on adding callbacks, such that you say add_this, add_that, etc. and all of these callbacks are meant to be executed asynchronously, but sequentially. My feeling about Twisted was that Deferred's are almost too easy -- to the point where I found myself returning a deferred to another function, adding more callbacks, passing it to someone else to add more callbacks, etc. At the end of the computation I felt like I had inadvertently brought back a lot of the pain of callbacks -- it's often difficult to figure out what gets called when and you have to look at a lot of places in the code to figure it out. In contrast, my recent experience with async is that it encourages having asynchronous code that looks synchronous. That alone feels like a benefit to me, because it encourages the same grouping that one would normally do in their code, which just seems harder to lose control of. C# async is certainly capable of the same "spaghetti" nature of Deferred (or callbacks in general, actually), but my experience has been that the structure encourages the opposite.
parennoob 13y ago
I don't understand any of the ideas alluded to therein, but the tone of the discourse in there and the thread that led to this (https://groups.google.com/forum/#!msg/python-tulip/EgpBV5-sIQ4/hcTQwMmKFOUJ https://groups.google.com/forum/#!msg/python-tulip/EgpBV5-sI...) is very respectful, and people apologize if their tone was too snarky or contemptuous. I like this a lot about python, and it makes me excited and eager to perhaps contribute a little to its development some day (I'm currently at the level where I have a decent idea about the OOP aspects of python, and know some tricks that I think are nifty, such as overloading __setattr__ and the like. So quite a long way to go, but hopefully I will get there some day).
- takluyver 13y ago
  You probably don't have as far to go as you think - the standard library is largely written in straightforward Python. The limiting factor is more finding a bug that no-one else is already fixing.
  - chrismonsanto 13y ago
    Could fix this one I submitted 3 years ago: http://bugs.python.org/issue9226 http://bugs.python.org/issue9226
- d0mine 13y ago
  Most code don't need __setattr__ level of magic https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy (search for "Levels of Magic")
thezilch 13y ago
If you happen to not make it through the thread to the following link or would prefer to join Guido and others discuss further edits to the post, before he makes it into a blog or a contribution to Twisted docs, the following is a Google Doc with several comment threads about quotes throughout the article: https://docs.google.com/document/d/10WOZgLQaYNpOrag-eTbUm-JUCCfdyfravZ4qSOQPg1M/edit https://docs.google.com/document/d/10WOZgLQaYNpOrag-eTbUm-JU...
eagsalazar2 13y ago
Looks a lot like promises. Cancellation is missing from Q.js anyway. One thing that always surprises me though is how people rightly are disgusted by callbacks but think promises/deferreds/futures/etc are nice. They are nicer but doing a lot of async stuff stills sucks big time even using these tools. You will write way more code and it will be way tricker than an otherwise comparable synchronous app. Obviously sometimes you have no choice but I think one should always strive to minimize the async parts of your app as much as possible.
- sethammons 13y ago
  Or maximize the async parts of your app as much as possible if you are wanting to build a scalable, distributed system and don't want any blocking calls :)
- coldtea 13y ago
  >One thing that always surprises me though is how people rightly are disgusted by callbacks but think promises/deferreds/futures/etc are nice. They are nicer but doing a lot of async stuff stills sucks big time even using these tools. You will write way more code and it will be way tricker than an otherwise comparable synchronous app. Well, nicer is all that matters...
- spartango 13y ago
  Promises, Deferreds, and Futures are all names for the same general idea, that which uses a monad to stand in for the result of operation that may or may not have been completed. There are implementations of this concept for many languages, from Java's futures to C# Tasks to Twisted Deferred to various JS Promise/Future implementations. One thing that I find amusing in the callbacks+/-futures discussion is Google Guava's ListenableFutures (http://code.google.com/p/guava-libraries/wiki/ListenableFutureExplained http://code.google.com/p/guava-libraries/wiki/ListenableFutu...): The JDK ships with Futures under java.util.concurrent, and they do what you would expect. What Guava adds is the ability to use Futures in the "pure" sense (get/check result) and with callbacks or delegates. The neat thing here is that you can use whichever paradigm is appropriate for the design of different modules of your system, and safely mix and match as needed. Guava isn't the answer to all the questions in the Futures world, but I do think it brings a neat idea to the table here.
hcarvalhoalves 13y ago
OT: Isn't passing an object around that defines a series of callbacks to be applied essentially a... monad?
- jerf 13y ago
  No. There's too many differences in all sorts of directions for this to be a useful way of approaching them.
  - hcarvalhoalves 13y ago
    Right, I'm stretching. For a moment I had a glimpse of it being the same design pattern, since Deferreds are composable.
- noelwelsh 13y ago
  Yes, futures/promises/whatever-you-want-to-call-them are monads in Scala and Haskell. The interface to these Deferreds are not, but it would be quite simple to make them so.
- voyou 13y ago
  Yes: here's an interesting discussion of promises (in JavaScript) in terms of monads: http://blog.jcoglan.com/2011/03/11/promises-are-the-monad-of-asynchronous-programming/ http://blog.jcoglan.com/2011/03/11/promises-are-the-monad-of...
YZF 13y ago
I made lots of micro-edits while thinking about this, let me try and combine to something more cohesive... Also of note is that I use Twisted in my day to day. In common Twisted programming a deferred can not fire before you add the callback since the only way a deferred can fire (for typical async work!) is through the proactor (which Twisted calls the reactor) and you haven't yet returned control to the proactor. The bug comment seems to imply a more fundamental issue but if Twisted had such a fundamental bug it just wouldn't work. I think Guido's point is that you can still potentially add callbacks to a Deferred that has fired which opens an opportunity for bugs. He has a point there but you don't usually use Deferred that way. Generally when you add callbacks you know the Deferred hasn't fired yet. I think Twisted isn't that bad but there are a few things I dislike about it: It's harder than I'm used to to combine synchronous and asynchronous. Twisted is "contagious". Because the Twisted application "wants" to be single threaded, everything goes through the reactor and therefore everything has to be pure async. You can deviate from that but it has some amount of built in inflexibility. If I compare with C++ Boost::Asio there's much better multi-threading support in Asio with its worker threads (which is important for scaling across cores) and you can do more interesting combinations of async/sync IMO. Twisted's @inlinecallbacks decorator allows you to write linear looking code you it also encourages a too-serialized way of doing things. The power of asynchronous is to do things in parallel but in Twisted you often either end up with very hard to follow "pure" asynch code or less efficient "chained" asynch code. At least that's my experience. I personally prefer having callbacks to the Deferred returns because very often the only thing you do with a Deferred is add callbacks and it makes your code messier, more clutter/boilerplate. Returning Deferred allows for the "generator" style @inlinecallbacks which actually can be an anti-pattern from a performance perspective and also many people find it confusing to read... Explicitly chaining callbacks is what I prefer. To elaborate on callback vs. Deferred you can relatively easily build more complex structures of any kind you wish over the callback when you need it. In a sense it's the most basic way of doing things. I like basic :-) Related to combining async and sync styles you end up seeing things like isinstance(ret, Deferred). There's also maybedeferred for dealing with async vs. sync. Still not great.
- rdtsc 13y ago
  > it's like Twisted is contagious. I can't quite put my finger on it. I can. Twisted is viral (not in a good way always). Once you start using Twisted you are doomed and stuck in a parallel reality of Twisted-only libraries forever. I have used Twisted professionally for 4+ years, this is not something I just read on a blog. You first say "Oh, I know look at the cute example of parsing a line protocol with it, let my try that. That was easy, I should adopt it". And then it got you. You want to fetch data from a particular database. That's easy, github has a project in Python that does it. Oh but you need it to return a deferred now. You can't simply make a connection and block because it blocks the reactor. This library is no good, you either have to pray someone who knows Twisted wrote a Twisted version of it, or you have to write your own. As you pointed out, you can use code to combine threaded and deferred code but it is not fun or easy. Make sure you think twice before adopting it. BTW the best way to learn it is to use it. Don't read about it. Reading about it won't help much. It is one of those subjects, like riding a bike if you don't know how. Once you get it you get it and it becomes easy. What is not easy is library fragmentation. You have been down-voted and perhaps I'll be too for my negativity. But I am worried about infatuation with Twisted and Promises in recent (last year and on) Python design emails and moving them more to be part of the standard library. I think this is a not healthy move for Python and a step forward. I'll speak my mind and if you feel this is too negative, just downvote, I'll understand, but I think it has to be said. They are copying what they are seeing in other language hint: node.js (Javascript). I feel there is a bit of jealousy, Python is maturing, its last exciting thing was 3.0. PyPy has been relegated to the magic academic research corner (unfairly I think). But, node.js and Javascript on the server is not successful because it has a callback style of concurrency. There are other reasons. So copying that aspect won't make Python more appealing. Heck, Python had an awesome callback library for ages -- Twisted, when Javascript was still use to make buttons flash and Java applets were the hot thing. Did it take off? Somewhat, but not really. It was famously hard to grok. The freshest breath of air so far in practical Python use on the server is using greenlet. But that also has been pushed to sort of "ah it is dirty, we don't like, how will you ever know if your code switches co-routines". Well it is not perfect but it is most practical and performant to date. I wish that became part of core Python. I wish they copied more from Go, Rust or Erlang rather than a callback style concurrency with some extra sauce on top. A Anyway, ending my rant here, and hoping I didn't upset too many people already.
jasonmoo 13y ago
I think the entire go spec would fit into the same number of pages as Guido's explanation of deferreds. zing!
- jasonmoo 13y ago
  Omg you guys have no sense of humor.
- iopq 13y ago
  I can write what's new and exciting about Go on the back of a napkin.
thristian 13y ago
I first came across Twisted Python and Deferreds years ago, and it was my first introduction to the world of asynchronous programming. It was a little bit difficult to wrap my head around the intricate braid of callbacks and errbacks, but once I realised they mapped directly onto sequential statements and try/except in synchronous code, I got a lot done. In the intervening years, I've seen a lot of projects re-implementing asynchronous programming (A perfectly valid idea, Twisted certainly has enough backwards-compatibility warts to warrant an occasional rebuilding), but the early attempts used bare callbacks for everything, which I found frustrating because I felt Deferreds were clearly a better system. Later attempts introduced "Futures" or "Promises" which were slightly better than bare callbacks but did silly things like pass the same initial result to every registered callback instead of being composable. These almost-but-not-quite-entirely-unlike-Deferreds were even more infuriating, in a "somebody is wrong on the Internet" kind of way. I'm really glad to see Guido picking the best ideas out of Twisted's Deferreds and trying to make them accessible to a professional-developer audience. If your language/runtime doesn't support some kind of co-routines, then your only hope for dealing with asynchronous code to tame callbacks, and Deferreds are the best model I've come across for doing that. (Guido mentions he won't be using Deferreds or Deferred-analogues in his Tulip async framework, for undisclosed reasons. I'm betting those reasons are that modern Python does have workable co-routines, so callbacks and callback-taming aren't necessary.)
- lazyfunctor 13y ago
  I am new to this. Can you give me any pointers to differences between promises and deferreds. I have some idea about promises in JS. http://promises-aplus.github.io/promises-spec/ http://promises-aplus.github.io/promises-spec/ How is twisted's deferred different?
  - thristian 13y ago
    Just the first difference that caught my eye: section 3.1 of the document you linked to says that once fulfilled or rejected, a Promise must not change to any other state, and its value may not change. In Guido's exposition of Deferreds, under the heading "Idea 2: Pass results from callback to callback", he says: The most interesting bits are the last two lines: the result of each callback is passed to the next. This is different from how things work in concurrent.futures and Tulip, where the result (once set) is fixed as an attribute of the Future. Here the result can be modified by each callback. This enables a new pattern when one function returning a Deferred calls another one and transforms its result, and this is what earns this idea three stars.
annnnd 13y ago
This bit sums my experience with Twisted nicely: "...[the available docs] are either aimed at absolute beginners or at experienced Twisted users." I was highly frustrated trying to find any useful docs on Twisted. This is where I dumped Twisted for Tornado (and I'm quite happy about it too ;). Looking back Twisted seems over-engineered too, but that is strictly my personal opinion and could be due to lack of understanding.
scrambam 13y ago
What's the point? Python is slow and soon to be dead. Go is just like Python and backed by Google. Node.js isn't as good as Go, but everyone already knows how to use it.
rkangel 13y ago
What interested me is the discussion about naming (re-use of callback, "errback", "Failure" etc.). That sort of discussion quite often smacks of bike-shedding, but Guido's points here made me release how clear and simple a lot of the naming in Python is, and how much that adds to readability and ease of understanding of code. Clearly a lot of thought along the lines of "this should be a verb" has gone into the standard library and choice of language keywords, and this has spread to code built on top. Without realising it, this is one of the things that has drawn me to Python. I'll stop taking it for granted now, pay attention to how it's done, and maybe my own code will benefit.
dustingetz 13y ago
Here is a much better explanation, in Javascript: http://domenic.me/2012/10/14/youre-missing-the-point-of-promises/ http://domenic.me/2012/10/14/youre-missing-the-point-of-prom...
robert-zaremba 13y ago
Interesting thing is that when using geenlet + proper event framework (gevent / eventlet) you don't have all this issues about different callback styles / methods / objects, and what is more important, IO libraries support (like DB drivers). I know there are ways to use twisted drivers in tornado, or use `deferToThread` - but they doesn't seems as pythonic as monkey patching + python drivers (like with eventlet + <your pure python DB driver>) It's worth nothing that PyPy has a great support for greenlets with his JIT.

225 points by thezilch 13y ago

I'm probably missing something, but isn't Idea 4: Chaining Deferreds just flatMap over Futures? Scala may have spoiled me somewhat.

zeckalpha 13y ago
And errbacks are really just the Maybe monad.
- seanmcdirmid 13y ago
  None of these future/aync systems are very novel, but there is demand for them. Eventually most of them will fail and a few of the good ones will survive. And ya, they will resemble functional programming language X and Y in some of their features, because they aren't meant to be novel.
- nostrademons 13y ago
  Errbacks are the Error monad. Failure is the Maybe monad (or errbacks with callback=None).
noelwelsh 13y ago
Yes, the Twisted Deferred is a not-quite-a-monad. The API would be simpler if it was, but I guess the original developers didn't have that insight at the time they were writing it.
dons 13y ago
Calling monadic bind "flatmap" makes me cry inside.
- DanWaterworth 13y ago
  Indeed, it's like snatching defeat from the jaws of victory. On a related note, Guido van Rossum gives monads 5/5 stars... sort of.

aidos 13y ago

I've only looked at Twisted briefly before but I feel like I learnt a lot from this run-through. Guido writes so well; he is able to thoughtfully deconstruct ideas with a critical eye while still praising the bits he likes. I think we could all learn something from the way he has approached this. It's definitely a model I will aspire to follow while appraising unfamiliar code.

Pxtl 13y ago

The bit with passing the result along in the chain of deferrals reminds me a bit of some of the fun stuff you can do with Events in C#.

Locke1689 13y ago
Yes, although that's considered bad practice with async. I previously used Twisted to implement a speculative model of new Internet architectures for a graduate class when I was in my distributed systems phase in college. One thing that Guido mentions is that Twisted Deferred computations generally rely on adding callbacks, such that you say add_this, add_that, etc. and all of these callbacks are meant to be executed asynchronously, but sequentially. My feeling about Twisted was that Deferred's are almost too easy -- to the point where I found myself returning a deferred to another function, adding more callbacks, passing it to someone else to add more callbacks, etc. At the end of the computation I felt like I had inadvertently brought back a lot of the pain of callbacks -- it's often difficult to figure out what gets called when and you have to look at a lot of places in the code to figure it out. In contrast, my recent experience with async is that it encourages having asynchronous code that looks synchronous. That alone feels like a benefit to me, because it encourages the same grouping that one would normally do in their code, which just seems harder to lose control of. C# async is certainly capable of the same "spaghetti" nature of Deferred (or callbacks in general, actually), but my experience has been that the structure encourages the opposite.

parennoob 13y ago

I don't understand any of the ideas alluded to therein, but the tone of the discourse in there and the thread that led to this (https://groups.google.com/forum/#!msg/python-tulip/EgpBV5-sIQ4/hcTQwMmKFOUJ https://groups.google.com/forum/#!msg/python-tulip/EgpBV5-sI...) is very respectful, and people apologize if their tone was too snarky or contemptuous. I like this a lot about python, and it makes me excited and eager to perhaps contribute a little to its development some day (I'm currently at the level where I have a decent idea about the OOP aspects of python, and know some tricks that I think are nifty, such as overloading __setattr__ and the like. So quite a long way to go, but hopefully I will get there some day).

takluyver 13y ago
You probably don't have as far to go as you think - the standard library is largely written in straightforward Python. The limiting factor is more finding a bug that no-one else is already fixing.
- chrismonsanto 13y ago
  Could fix this one I submitted 3 years ago: http://bugs.python.org/issue9226 http://bugs.python.org/issue9226
d0mine 13y ago
Most code don't need __setattr__ level of magic https://github.com/lihaoyi/macropy https://github.com/lihaoyi/macropy (search for "Levels of Magic")

thezilch 13y ago

If you happen to not make it through the thread to the following link or would prefer to join Guido and others discuss further edits to the post, before he makes it into a blog or a contribution to Twisted docs, the following is a Google Doc with several comment threads about quotes throughout the article: https://docs.google.com/document/d/10WOZgLQaYNpOrag-eTbUm-JUCCfdyfravZ4qSOQPg1M/edit https://docs.google.com/document/d/10WOZgLQaYNpOrag-eTbUm-JU...

eagsalazar2 13y ago

Looks a lot like promises. Cancellation is missing from Q.js anyway. One thing that always surprises me though is how people rightly are disgusted by callbacks but think promises/deferreds/futures/etc are nice. They are nicer but doing a lot of async stuff stills sucks big time even using these tools. You will write way more code and it will be way tricker than an otherwise comparable synchronous app. Obviously sometimes you have no choice but I think one should always strive to minimize the async parts of your app as much as possible.

sethammons 13y ago
Or maximize the async parts of your app as much as possible if you are wanting to build a scalable, distributed system and don't want any blocking calls :)
coldtea 13y ago
>One thing that always surprises me though is how people rightly are disgusted by callbacks but think promises/deferreds/futures/etc are nice. They are nicer but doing a lot of async stuff stills sucks big time even using these tools. You will write way more code and it will be way tricker than an otherwise comparable synchronous app. Well, nicer is all that matters...
spartango 13y ago
Promises, Deferreds, and Futures are all names for the same general idea, that which uses a monad to stand in for the result of operation that may or may not have been completed. There are implementations of this concept for many languages, from Java's futures to C# Tasks to Twisted Deferred to various JS Promise/Future implementations. One thing that I find amusing in the callbacks+/-futures discussion is Google Guava's ListenableFutures (http://code.google.com/p/guava-libraries/wiki/ListenableFutureExplained http://code.google.com/p/guava-libraries/wiki/ListenableFutu...): The JDK ships with Futures under java.util.concurrent, and they do what you would expect. What Guava adds is the ability to use Futures in the "pure" sense (get/check result) and with callbacks or delegates. The neat thing here is that you can use whichever paradigm is appropriate for the design of different modules of your system, and safely mix and match as needed. Guava isn't the answer to all the questions in the Futures world, but I do think it brings a neat idea to the table here.

hcarvalhoalves 13y ago

OT: Isn't passing an object around that defines a series of callbacks to be applied essentially a... monad?

jerf 13y ago
No. There's too many differences in all sorts of directions for this to be a useful way of approaching them.
- hcarvalhoalves 13y ago
  Right, I'm stretching. For a moment I had a glimpse of it being the same design pattern, since Deferreds are composable.
noelwelsh 13y ago
Yes, futures/promises/whatever-you-want-to-call-them are monads in Scala and Haskell. The interface to these Deferreds are not, but it would be quite simple to make them so.
voyou 13y ago
Yes: here's an interesting discussion of promises (in JavaScript) in terms of monads: http://blog.jcoglan.com/2011/03/11/promises-are-the-monad-of-asynchronous-programming/ http://blog.jcoglan.com/2011/03/11/promises-are-the-monad-of...

YZF 13y ago

I made lots of micro-edits while thinking about this, let me try and combine to something more cohesive... Also of note is that I use Twisted in my day to day. In common Twisted programming a deferred can not fire before you add the callback since the only way a deferred can fire (for typical async work!) is through the proactor (which Twisted calls the reactor) and you haven't yet returned control to the proactor. The bug comment seems to imply a more fundamental issue but if Twisted had such a fundamental bug it just wouldn't work. I think Guido's point is that you can still potentially add callbacks to a Deferred that has fired which opens an opportunity for bugs. He has a point there but you don't usually use Deferred that way. Generally when you add callbacks you know the Deferred hasn't fired yet. I think Twisted isn't that bad but there are a few things I dislike about it: It's harder than I'm used to to combine synchronous and asynchronous. Twisted is "contagious". Because the Twisted application "wants" to be single threaded, everything goes through the reactor and therefore everything has to be pure async. You can deviate from that but it has some amount of built in inflexibility. If I compare with C++ Boost::Asio there's much better multi-threading support in Asio with its worker threads (which is important for scaling across cores) and you can do more interesting combinations of async/sync IMO. Twisted's @inlinecallbacks decorator allows you to write linear looking code you it also encourages a too-serialized way of doing things. The power of asynchronous is to do things in parallel but in Twisted you often either end up with very hard to follow "pure" asynch code or less efficient "chained" asynch code. At least that's my experience. I personally prefer having callbacks to the Deferred returns because very often the only thing you do with a Deferred is add callbacks and it makes your code messier, more clutter/boilerplate. Returning Deferred allows for the "generator" style @inlinecallbacks which actually can be an anti-pattern from a performance perspective and also many people find it confusing to read... Explicitly chaining callbacks is what I prefer. To elaborate on callback vs. Deferred you can relatively easily build more complex structures of any kind you wish over the callback when you need it. In a sense it's the most basic way of doing things. I like basic :-) Related to combining async and sync styles you end up seeing things like isinstance(ret, Deferred). There's also maybedeferred for dealing with async vs. sync. Still not great.

rdtsc 13y ago
> it's like Twisted is contagious. I can't quite put my finger on it. I can. Twisted is viral (not in a good way always). Once you start using Twisted you are doomed and stuck in a parallel reality of Twisted-only libraries forever. I have used Twisted professionally for 4+ years, this is not something I just read on a blog. You first say "Oh, I know look at the cute example of parsing a line protocol with it, let my try that. That was easy, I should adopt it". And then it got you. You want to fetch data from a particular database. That's easy, github has a project in Python that does it. Oh but you need it to return a deferred now. You can't simply make a connection and block because it blocks the reactor. This library is no good, you either have to pray someone who knows Twisted wrote a Twisted version of it, or you have to write your own. As you pointed out, you can use code to combine threaded and deferred code but it is not fun or easy. Make sure you think twice before adopting it. BTW the best way to learn it is to use it. Don't read about it. Reading about it won't help much. It is one of those subjects, like riding a bike if you don't know how. Once you get it you get it and it becomes easy. What is not easy is library fragmentation. You have been down-voted and perhaps I'll be too for my negativity. But I am worried about infatuation with Twisted and Promises in recent (last year and on) Python design emails and moving them more to be part of the standard library. I think this is a not healthy move for Python and a step forward. I'll speak my mind and if you feel this is too negative, just downvote, I'll understand, but I think it has to be said. They are copying what they are seeing in other language hint: node.js (Javascript). I feel there is a bit of jealousy, Python is maturing, its last exciting thing was 3.0. PyPy has been relegated to the magic academic research corner (unfairly I think). But, node.js and Javascript on the server is not successful because it has a callback style of concurrency. There are other reasons. So copying that aspect won't make Python more appealing. Heck, Python had an awesome callback library for ages -- Twisted, when Javascript was still use to make buttons flash and Java applets were the hot thing. Did it take off? Somewhat, but not really. It was famously hard to grok. The freshest breath of air so far in practical Python use on the server is using greenlet. But that also has been pushed to sort of "ah it is dirty, we don't like, how will you ever know if your code switches co-routines". Well it is not perfect but it is most practical and performant to date. I wish that became part of core Python. I wish they copied more from Go, Rust or Erlang rather than a callback style concurrency with some extra sauce on top. A Anyway, ending my rant here, and hoping I didn't upset too many people already.

jasonmoo 13y ago

I think the entire go spec would fit into the same number of pages as Guido's explanation of deferreds. zing!

jasonmoo 13y ago
Omg you guys have no sense of humor.
iopq 13y ago
I can write what's new and exciting about Go on the back of a napkin.

thristian 13y ago

I first came across Twisted Python and Deferreds years ago, and it was my first introduction to the world of asynchronous programming. It was a little bit difficult to wrap my head around the intricate braid of callbacks and errbacks, but once I realised they mapped directly onto sequential statements and try/except in synchronous code, I got a lot done. In the intervening years, I've seen a lot of projects re-implementing asynchronous programming (A perfectly valid idea, Twisted certainly has enough backwards-compatibility warts to warrant an occasional rebuilding), but the early attempts used bare callbacks for everything, which I found frustrating because I felt Deferreds were clearly a better system. Later attempts introduced "Futures" or "Promises" which were slightly better than bare callbacks but did silly things like pass the same initial result to every registered callback instead of being composable. These almost-but-not-quite-entirely-unlike-Deferreds were even more infuriating, in a "somebody is wrong on the Internet" kind of way. I'm really glad to see Guido picking the best ideas out of Twisted's Deferreds and trying to make them accessible to a professional-developer audience. If your language/runtime doesn't support some kind of co-routines, then your only hope for dealing with asynchronous code to tame callbacks, and Deferreds are the best model I've come across for doing that. (Guido mentions he won't be using Deferreds or Deferred-analogues in his Tulip async framework, for undisclosed reasons. I'm betting those reasons are that modern Python does have workable co-routines, so callbacks and callback-taming aren't necessary.)

lazyfunctor 13y ago
I am new to this. Can you give me any pointers to differences between promises and deferreds. I have some idea about promises in JS. http://promises-aplus.github.io/promises-spec/ http://promises-aplus.github.io/promises-spec/ How is twisted's deferred different?
- thristian 13y ago
  Just the first difference that caught my eye: section 3.1 of the document you linked to says that once fulfilled or rejected, a Promise must not change to any other state, and its value may not change. In Guido's exposition of Deferreds, under the heading "Idea 2: Pass results from callback to callback", he says: The most interesting bits are the last two lines: the result of each callback is passed to the next. This is different from how things work in concurrent.futures and Tulip, where the result (once set) is fixed as an attribute of the Future. Here the result can be modified by each callback. This enables a new pattern when one function returning a Deferred calls another one and transforms its result, and this is what earns this idea three stars.

annnnd 13y ago

This bit sums my experience with Twisted nicely: "...[the available docs] are either aimed at absolute beginners or at experienced Twisted users." I was highly frustrated trying to find any useful docs on Twisted. This is where I dumped Twisted for Tornado (and I'm quite happy about it too ;). Looking back Twisted seems over-engineered too, but that is strictly my personal opinion and could be due to lack of understanding.

scrambam 13y ago

What's the point? Python is slow and soon to be dead. Go is just like Python and backed by Google. Node.js isn't as good as Go, but everyone already knows how to use it.

rkangel 13y ago

What interested me is the discussion about naming (re-use of callback, "errback", "Failure" etc.). That sort of discussion quite often smacks of bike-shedding, but Guido's points here made me release how clear and simple a lot of the naming in Python is, and how much that adds to readability and ease of understanding of code. Clearly a lot of thought along the lines of "this should be a verb" has gone into the standard library and choice of language keywords, and this has spread to code built on top. Without realising it, this is one of the things that has drawn me to Python. I'll stop taking it for granted now, pay attention to how it's done, and maybe my own code will benefit.

dustingetz 13y ago

Here is a much better explanation, in Javascript: http://domenic.me/2012/10/14/youre-missing-the-point-of-promises/ http://domenic.me/2012/10/14/youre-missing-the-point-of-prom...

robert-zaremba 13y ago

Interesting thing is that when using geenlet + proper event framework (gevent / eventlet) you don't have all this issues about different callback styles / methods / objects, and what is more important, IO libraries support (like DB drivers). I know there are ways to use twisted drivers in tornado, or use `deferToThread` - but they doesn't seems as pythonic as monkey patching + python drivers (like with eventlet + <your pure python DB driver>) It's worth nothing that PyPy has a great support for greenlets with his JIT.