9 ms·

Node.js Flow – Callback Hell vs. Async vs. Highland Streams

91 points by ddispaltro 12y ago

andrea_sdl 12y ago
I started using node.js 1 year ago for a personal project and I must say that the callback hell is a real issue to be solved. Personally I think the highland approach is too verbose, while the bluebird promises is the one I prefer for both readability and functionality. Does anyone has used them in big production systems? Are you happy with them?
- pkorzeniewski 12y ago
  I've been using async for quite long time now and it's absolutely sufficient - never had any problems, it has all the control flow types you'll need and in my opinion it's very readable. It's also the most mature amongst them, so yes, it's definitiely ready for big production systems ;)
- STRML 12y ago
  I use bluebird in all of my production systems, not only because I believe it creates the most readable code, but also because it can do things that callback flow libraries and async simply cannot do. It's fast to boot and is great for error handling. I highly recommend it.
- spion 12y ago
  We use bluebird. With BLUEBIRD_DEBUG=1 we get amazing long stack traces that span across events, and they got even better (cleaner, more accurate) post bluebird 2.8. If willing to take a performance hit (they will remain faster than caolan's async or native promises though), they're also usable in production.
  - andrea_sdl 12y ago
    sounds like a really good feature. I think I'll try bluebird on my personal project to rewrite part of the code. thanks
- Lazare 12y ago
  I like the idea of highland, but I think as a practical matter bluebird promises are the best solution for production. I've used them a fair bit in production, and I'd say yes, I'm quite happy with them. Edit: I've used async too, but I feel that async does not scale well in terms of complexity. When you start needing solid error handling while dealing with multiple async processes with complex mixtures of parallel and sequential...bluebird works, async just ends up as a mess. In my experience, anyhow. Either lib works great for simple uses cases though. :)
  - andrea_sdl 12y ago
    Thanks for the reply, this was just what I wanted to understand
- davidw 12y ago
  One approach is to use an underlying scheduler so that even while (1) { ... } kinds of things don't lock everything up. This is the approach that Erlang (and therefore Elixir) takes.
aeosynth 12y ago
You can also use a transpiler such as 6to5[0] which supports ES7 async/await. [0]: http://6to5.org/ http://6to5.org/
- drapper 12y ago
  ...and you can also have something like that using Bluebird.coroutine() (https://github.com/petkaantonov/bluebird/blob/master/API.md#... https://github.com/petkaantonov/bluebird/blob/master/API.md#...) or co (https://github.com/tj/co https://github.com/tj/co, see also https://github.com/koajs/koa https://github.com/koajs/koa)
  - phpnode 12y ago
    6to5 actually supports compiling async/await to bluebird coroutines
- soaringmonchi 12y ago
  Definitely my preferred approach. Note that you need to run 6to5 with the -e flag to try this.
  - thejameskyle 12y ago
    I think you mean the --experimental flag. -e is for evaluate.
- jonpaul 12y ago
  Where does it say that it supports async/await?
  - drapper 12y ago
    http://6to5.org/docs/usage/transformers/#async-to-generator http://6to5.org/docs/usage/transformers/#async-to-generator
- davedx 12y ago
  Whaaa... they're putting experimental ES7 features in a transpiler called 6to5? Or am I missing something?
deleted 12y ago
[deleted]
vladev 12y ago
I think RxJs[1] can help here as well. [1]: https://github.com/Reactive-Extensions/RxJS https://github.com/Reactive-Extensions/RxJS
jamesrom 12y ago
.then(Promise.promisifyAll(process1)) .then(Promise.promisifyAll(process2)) .then(Promise.promisifyAll(process3)) Shouldn't it be just Promise.promisify[1]. Not that it matters, promisifyAll does the same thing, but also looks for child functions to promisify. [1] https://github.com/petkaantonov/bluebird/blob/master/API.md#promisepromisifyfunction-nodefunction--dynamic-receiver---function https://github.com/petkaantonov/bluebird/blob/master/API.md#...
- eiriklv 12y ago
  Blog owner here: Yes, you're correct, will fix it asap :-)
- eiriklv 12y ago
  fixed :-)
tuananh 12y ago
using generators is much more elegant.
- olegp 12y ago
  And fibers even more so, in my opinion.
  - tuananh 12y ago
    I didn't know about fibers. I'll take a look later today.
  - yuchi 12y ago
    I hate how they can introduce collateral damage to other flows. They’re many times more powerful than generator+yield/async+await, but I love how yield/await is explicitly telling where an async behaviour will be flattened into the routine.
    - olegp 12y ago
      That's a bit of a made up problem. Having used fibers for a few years now for some heavy lifting, I've never had issues with not knowing where async behavior occurs. No different from goroutines really.
BonoboBoner 12y ago
Interesting comparison. I have never used Node for anything bigger than small personal stuff, because non-blocking code was never a requirement. I still dont see the value in it in scenarios like in the blog post, as all he is doing is a bunch of functions and once all of them are done, send something back to the client. How does the client or the dev benefit from the non-blocking approach here? At the end of the day he is describing a sequence of actions the depend on each other. Why not block then? I really dont want to sound arrogant, but I am just curious what the benefits are that I dont know of.
- ZitchDog 12y ago
  Because node is single threaded, so blocking for this user will block all requests for all users.
  - BonoboBoner 12y ago
    That explains it, thanks. To me this makes the focus on non-blocking code in PlayFramework2.0 a bit more strange to me, since the JVM isnt single threaded.
    - yowmamasita 12y ago
      it's because native threads are expensive
      - BonoboBoner 12y ago
        They are again? In this talk they mentioned them to be cheap in 2008
- olalonde 12y ago
  Originally, for performance reasons. Async code lets you handle multiple concurrent connections without having to rely on multiple threads or processes (which are more resource hungry)[0]. Nowadays, there's also the fact that if you want to use Javascript on the backend, Node.js is the de facto standard. You don't necessarily have to do async I/O but you'd be fighting against the whole ecosystem by not doing so. [0] http://en.wikipedia.org/wiki/Preemption_(computing)#Preemptive_multitasking http://en.wikipedia.org/wiki/Preemption_(computing)#Preempti...
samuell 12y ago
No mention about NoFlo, the Node.js Flow-based programming library, designed to tackle exactly this problem? [1] [1] http://noflojs.org http://noflojs.org
- marrs 12y ago
  Difficult to evaluate it when I couldn't see a single line of code on that website. It looks to me that it's targeting quite a different problem.
yoshuaw 12y ago
Noticeably missing from that list is regular Streams2, with possibly Through2 / From2 to make it easier to work with.
- eiriklv 12y ago
  There has surfaced quite a few additional approaches now, and I'll try to incorporate all of them into the post.
pmontra 12y ago
Unfortunately all those .flatMap(_.wrapCallback(one)) .flatMap(_.wrapCallback(two)) ... or .then(Promise.promisifyAll(one)) .then(Promise.promisifyAll(two)) ... are noise making the signal difficult to read. It should come down to nothing more than one | two plus the error handling clause. That's not JavaScript but you got the idea. Syntax is important. I still have bad dreams about arrays() hell in an app written by a customer with CakePHP 1 (a Rails clone with an approach that didn't suit PHP well). I'll skip over the $this-> mess. I'm not much into asynchronous languages. Is there any examples around with a built in and slim syntax for concatenating callbacks? Thanks.
- tel 12y ago
  In Haskell it would be do one two because it's pretty well understood there that syntax can be a big differentiator. What's more is that this syntax adopts for the binding aspect of flatMap. So if two is dependent on one... do output <- one two output which is a shortcut for one >>= two which might be Promise.promisifyAll(one) .then(function (output) { return Promise.promisifyAll(two(output)) }) in Javascript
  - jerf 12y ago
    Haskell has the advantage that what you've proposed is an exact conversion of the "flatMap" code, meaning that anything else that you can do with flatMap can be used with do notation, which is a lot of things beyond just tying two bits of IO code together. But in this specific case of just tying two bit of execution together, I'd observe that Erlang is One(), Two() and Go would be One() Two() and that's all regardless of whether One and Two have 0 accesses to IO or do IO dozens of times, and One and Two may also freely call other functions that do arbitrary IO with no further ceremony. The "solution" Erlang and Go offer to callback hell is simply not to have it, and pretty much no matter how you slice it, not having the problem in the first place beats any solution the JS community may or may not settle on.
    - tel 12y ago
nawitus 12y ago
I don't like that highland uses the "_" variable in their examples. The same variable name is already used by lo-dash and underscore.
doublerebel 12y ago
Highland, promises, async.js are all hacks. Best to use a language construct like await/yield, as in ES6, tamejs, and Iced CoffeeScript. These days noone should experience callback hell, it is a well-solved problem.
- SeanDav 12y ago
  Not saying you are wrong (I don't know enough to disagree) but if asynchronous program is solved in node/javascript, why is there the need for all these post on callbacks, streams, promises, fibres etc? Can you expand more on your comment and/or point to some references?
  - doublerebel 12y ago
    "Callback hell" is very effective linkbait for a title, there is not a need for the post. Promises are an effective implementation of control flow for ES5 JS, but don't really improve program structure. They are most useful "under-the-hood", as TameJS/Iced-CS creates when compiling. The overarching solution is to be "flat" by using small, composable functions. Lazy.js and Highland.js force the user to do this but require learning a specific syntax. Streams make this technique easier to reason about (easier say, than Transducers or Generators), but the goal is the same: operate on one object at a time with one purpose, and return a result in a standard manner so it's easy to consume with the next function. This is good practice, but leaning on one of these libraries is an excuse to force the practice. Chained callbacks is JS looking from the inside out; looking from the outside in, we see this practice is solved by many other languages [1] -- not just serial calls but parallel also! C#, Scala, Python, Java, Dart, Spiderjs and more have implementations of Futures [2]. Fibers implement Futures but require a native module for V8 -- since they are not portable outside Node they don't have great adoption and aren't used in almost any node.js modules: only ~100 out of the endless ecosystem [3]. Fibers, Async.js, and Highland also aren't syntactically an improvement over Promises -- the point of programming, which devs will get tunnel vision and forget -- is to leverage the language to do more work for less keystrokes/manhours [4]. The compiler should create control flow for my code, not the other way around. There should be no reason to learn a complex API just to get serial/parallel calls, when a simple keyword addition to the language will suffice. Not to mention, await/defer is so readable that new devs can start working with async code right away, without needing to research. This is not the case with promises/fibers/asyncjs/highland or whatever flavor-of-the-month a shop has picked for control flow. [1] https://github.com/KjellSchubert/promise-future-task https://github.com/KjellSchubert/promise-future-task [2] http://en.wikipedia.org/wiki/Futures_and_promises#List_of_implementations http://en.wikipedia.org/wiki/Futures_and_promises#List_of_im... [3] https://www.npmjs.com/package/fibers https://www.npmjs.com/package/fibers [4] http://blog.teamleadnet.com/2013/08/async-await-in-net-is-not-for.html http://blog.teamleadnet.com/2013/08/async-await-in-net-is-no...
api 12y ago
vs. using a real language? Webbies seriously have Stockholm syndrome with this language.
robinricard 12y ago
Combining Promises+ES6 Generators seems the next way to go. It is already used successfully in many Firefox tests and the syntax just feels great. For the Node.js ecosystem, you should take a look at IO.js [1], Koa [2] and co [3] that are able to handle nice async syntaxes. [1]: https://iojs.org/ https://iojs.org/ [2]: http://koajs.com/ http://koajs.com/ [3]: https://github.com/tj/co https://github.com/tj/co
deleted 12y ago
[deleted]

91 points by ddispaltro 12y ago

I started using node.js 1 year ago for a personal project and I must say that the callback hell is a real issue to be solved. Personally I think the highland approach is too verbose, while the bluebird promises is the one I prefer for both readability and functionality. Does anyone has used them in big production systems? Are you happy with them?

pkorzeniewski 12y ago
I've been using async for quite long time now and it's absolutely sufficient - never had any problems, it has all the control flow types you'll need and in my opinion it's very readable. It's also the most mature amongst them, so yes, it's definitiely ready for big production systems ;)
STRML 12y ago
I use bluebird in all of my production systems, not only because I believe it creates the most readable code, but also because it can do things that callback flow libraries and async simply cannot do. It's fast to boot and is great for error handling. I highly recommend it.
spion 12y ago
We use bluebird. With BLUEBIRD_DEBUG=1 we get amazing long stack traces that span across events, and they got even better (cleaner, more accurate) post bluebird 2.8. If willing to take a performance hit (they will remain faster than caolan's async or native promises though), they're also usable in production.
- andrea_sdl 12y ago
  sounds like a really good feature. I think I'll try bluebird on my personal project to rewrite part of the code. thanks
Lazare 12y ago
I like the idea of highland, but I think as a practical matter bluebird promises are the best solution for production. I've used them a fair bit in production, and I'd say yes, I'm quite happy with them. Edit: I've used async too, but I feel that async does not scale well in terms of complexity. When you start needing solid error handling while dealing with multiple async processes with complex mixtures of parallel and sequential...bluebird works, async just ends up as a mess. In my experience, anyhow. Either lib works great for simple uses cases though. :)
- andrea_sdl 12y ago
  Thanks for the reply, this was just what I wanted to understand
davidw 12y ago
One approach is to use an underlying scheduler so that even while (1) { ... } kinds of things don't lock everything up. This is the approach that Erlang (and therefore Elixir) takes.

aeosynth 12y ago

You can also use a transpiler such as 6to5[0] which supports ES7 async/await. [0]: http://6to5.org/ http://6to5.org/

drapper 12y ago
...and you can also have something like that using Bluebird.coroutine() (https://github.com/petkaantonov/bluebird/blob/master/API.md#... https://github.com/petkaantonov/bluebird/blob/master/API.md#...) or co (https://github.com/tj/co https://github.com/tj/co, see also https://github.com/koajs/koa https://github.com/koajs/koa)
- phpnode 12y ago
  6to5 actually supports compiling async/await to bluebird coroutines
soaringmonchi 12y ago
Definitely my preferred approach. Note that you need to run 6to5 with the -e flag to try this.
- thejameskyle 12y ago
  I think you mean the --experimental flag. -e is for evaluate.
jonpaul 12y ago
Where does it say that it supports async/await?
- drapper 12y ago
  http://6to5.org/docs/usage/transformers/#async-to-generator http://6to5.org/docs/usage/transformers/#async-to-generator
davedx 12y ago
Whaaa... they're putting experimental ES7 features in a transpiler called 6to5? Or am I missing something?

deleted 12y ago

[deleted]

vladev 12y ago

I think RxJs[1] can help here as well. [1]: https://github.com/Reactive-Extensions/RxJS https://github.com/Reactive-Extensions/RxJS

jamesrom 12y ago

.then(Promise.promisifyAll(process1)) .then(Promise.promisifyAll(process2)) .then(Promise.promisifyAll(process3)) Shouldn't it be just Promise.promisify[1]. Not that it matters, promisifyAll does the same thing, but also looks for child functions to promisify. [1] https://github.com/petkaantonov/bluebird/blob/master/API.md#promisepromisifyfunction-nodefunction--dynamic-receiver---function https://github.com/petkaantonov/bluebird/blob/master/API.md#...

eiriklv 12y ago
Blog owner here: Yes, you're correct, will fix it asap :-)
eiriklv 12y ago
fixed :-)

tuananh 12y ago

using generators is much more elegant.

olegp 12y ago
And fibers even more so, in my opinion.
- tuananh 12y ago
  I didn't know about fibers. I'll take a look later today.
- yuchi 12y ago
  I hate how they can introduce collateral damage to other flows. They’re many times more powerful than generator+yield/async+await, but I love how yield/await is explicitly telling where an async behaviour will be flattened into the routine.
  - olegp 12y ago
    That's a bit of a made up problem. Having used fibers for a few years now for some heavy lifting, I've never had issues with not knowing where async behavior occurs. No different from goroutines really.

BonoboBoner 12y ago

Interesting comparison. I have never used Node for anything bigger than small personal stuff, because non-blocking code was never a requirement. I still dont see the value in it in scenarios like in the blog post, as all he is doing is a bunch of functions and once all of them are done, send something back to the client. How does the client or the dev benefit from the non-blocking approach here? At the end of the day he is describing a sequence of actions the depend on each other. Why not block then? I really dont want to sound arrogant, but I am just curious what the benefits are that I dont know of.

ZitchDog 12y ago
Because node is single threaded, so blocking for this user will block all requests for all users.
- BonoboBoner 12y ago
  That explains it, thanks. To me this makes the focus on non-blocking code in PlayFramework2.0 a bit more strange to me, since the JVM isnt single threaded.
  - yowmamasita 12y ago
    it's because native threads are expensive
    - BonoboBoner 12y ago
      They are again? In this talk they mentioned them to be cheap in 2008
olalonde 12y ago
Originally, for performance reasons. Async code lets you handle multiple concurrent connections without having to rely on multiple threads or processes (which are more resource hungry)[0]. Nowadays, there's also the fact that if you want to use Javascript on the backend, Node.js is the de facto standard. You don't necessarily have to do async I/O but you'd be fighting against the whole ecosystem by not doing so. [0] http://en.wikipedia.org/wiki/Preemption_(computing)#Preemptive_multitasking http://en.wikipedia.org/wiki/Preemption_(computing)#Preempti...

samuell 12y ago

No mention about NoFlo, the Node.js Flow-based programming library, designed to tackle exactly this problem? [1] [1] http://noflojs.org http://noflojs.org

marrs 12y ago
Difficult to evaluate it when I couldn't see a single line of code on that website. It looks to me that it's targeting quite a different problem.

yoshuaw 12y ago

Noticeably missing from that list is regular Streams2, with possibly Through2 / From2 to make it easier to work with.

eiriklv 12y ago
There has surfaced quite a few additional approaches now, and I'll try to incorporate all of them into the post.

pmontra 12y ago

Unfortunately all those .flatMap(_.wrapCallback(one)) .flatMap(_.wrapCallback(two)) ... or .then(Promise.promisifyAll(one)) .then(Promise.promisifyAll(two)) ... are noise making the signal difficult to read. It should come down to nothing more than one | two plus the error handling clause. That's not JavaScript but you got the idea. Syntax is important. I still have bad dreams about arrays() hell in an app written by a customer with CakePHP 1 (a Rails clone with an approach that didn't suit PHP well). I'll skip over the $this-> mess. I'm not much into asynchronous languages. Is there any examples around with a built in and slim syntax for concatenating callbacks? Thanks.

tel 12y ago
In Haskell it would be do one two because it's pretty well understood there that syntax can be a big differentiator. What's more is that this syntax adopts for the binding aspect of flatMap. So if two is dependent on one... do output <- one two output which is a shortcut for one >>= two which might be Promise.promisifyAll(one) .then(function (output) { return Promise.promisifyAll(two(output)) }) in Javascript
- jerf 12y ago
  Haskell has the advantage that what you've proposed is an exact conversion of the "flatMap" code, meaning that anything else that you can do with flatMap can be used with do notation, which is a lot of things beyond just tying two bits of IO code together. But in this specific case of just tying two bit of execution together, I'd observe that Erlang is One(), Two() and Go would be One() Two() and that's all regardless of whether One and Two have 0 accesses to IO or do IO dozens of times, and One and Two may also freely call other functions that do arbitrary IO with no further ceremony. The "solution" Erlang and Go offer to callback hell is simply not to have it, and pretty much no matter how you slice it, not having the problem in the first place beats any solution the JS community may or may not settle on.
  - tel 12y ago

nawitus 12y ago

I don't like that highland uses the "_" variable in their examples. The same variable name is already used by lo-dash and underscore.

doublerebel 12y ago

Highland, promises, async.js are all hacks. Best to use a language construct like await/yield, as in ES6, tamejs, and Iced CoffeeScript. These days noone should experience callback hell, it is a well-solved problem.

SeanDav 12y ago
Not saying you are wrong (I don't know enough to disagree) but if asynchronous program is solved in node/javascript, why is there the need for all these post on callbacks, streams, promises, fibres etc? Can you expand more on your comment and/or point to some references?
- doublerebel 12y ago
  "Callback hell" is very effective linkbait for a title, there is not a need for the post. Promises are an effective implementation of control flow for ES5 JS, but don't really improve program structure. They are most useful "under-the-hood", as TameJS/Iced-CS creates when compiling. The overarching solution is to be "flat" by using small, composable functions. Lazy.js and Highland.js force the user to do this but require learning a specific syntax. Streams make this technique easier to reason about (easier say, than Transducers or Generators), but the goal is the same: operate on one object at a time with one purpose, and return a result in a standard manner so it's easy to consume with the next function. This is good practice, but leaning on one of these libraries is an excuse to force the practice. Chained callbacks is JS looking from the inside out; looking from the outside in, we see this practice is solved by many other languages [1] -- not just serial calls but parallel also! C#, Scala, Python, Java, Dart, Spiderjs and more have implementations of Futures [2]. Fibers implement Futures but require a native module for V8 -- since they are not portable outside Node they don't have great adoption and aren't used in almost any node.js modules: only ~100 out of the endless ecosystem [3]. Fibers, Async.js, and Highland also aren't syntactically an improvement over Promises -- the point of programming, which devs will get tunnel vision and forget -- is to leverage the language to do more work for less keystrokes/manhours [4]. The compiler should create control flow for my code, not the other way around. There should be no reason to learn a complex API just to get serial/parallel calls, when a simple keyword addition to the language will suffice. Not to mention, await/defer is so readable that new devs can start working with async code right away, without needing to research. This is not the case with promises/fibers/asyncjs/highland or whatever flavor-of-the-month a shop has picked for control flow. [1] https://github.com/KjellSchubert/promise-future-task https://github.com/KjellSchubert/promise-future-task [2] http://en.wikipedia.org/wiki/Futures_and_promises#List_of_implementations http://en.wikipedia.org/wiki/Futures_and_promises#List_of_im... [3] https://www.npmjs.com/package/fibers https://www.npmjs.com/package/fibers [4] http://blog.teamleadnet.com/2013/08/async-await-in-net-is-not-for.html http://blog.teamleadnet.com/2013/08/async-await-in-net-is-no...

api 12y ago

vs. using a real language? Webbies seriously have Stockholm syndrome with this language.

robinricard 12y ago

Combining Promises+ES6 Generators seems the next way to go. It is already used successfully in many Firefox tests and the syntax just feels great. For the Node.js ecosystem, you should take a look at IO.js [1], Koa [2] and co [3] that are able to handle nice async syntaxes. [1]: https://iojs.org/ https://iojs.org/ [2]: http://koajs.com/ http://koajs.com/ [3]: https://github.com/tj/co https://github.com/tj/co

deleted 12y ago

[deleted]