7 ms·

Managing Node.js Callback Hell with Promises, Generators and Other Approaches

• shubhra51 13y ago
  Promises have always been debatable. The new Generators is probably what I am looking for.
  • guiomie 13y ago
    Why?
  • onestone 13y ago
    Generators and Promises play very well together.
• shtylman 13y ago
  The generator example has no error handling (at least not from what I can see). Once the "callback hell" example was re-written with named functions, it is not much different than the generator example and depending on how you are testing, easier to test because it separates out IO from logic (Which you could still do in the generator example). One thing callback style code has begun to make me think about is the nesting/IO complexity of the functions a I write. When you nest, the IO/event boundary is visible in the code (by the indentation). As you continue to nest, you can start to get a clear idea of the amount of IO being done and where a particular function may be doing too much. Anyhow, most of this is preference anyway :)
  • sync 13y ago
    A simple try/catch around the whole thing would take care of error handling for the generator example.
    • inglor 13y ago
      only until you have a syntax error :)
• tgriesser 13y ago
  Disappointing that most promise articles/tutorials reference Q as the preferred promise library, rather than bluebird - https://github.com/petkaantonov/bluebird https://github.com/petkaantonov/bluebird Bluebird is not only so fast that it's almost on par with callbacks[1], but also innovates on the ability to filter on catching different error types[2], support for generators[3], context binding[4] among other things. [1]: https://github.com/petkaantonov/bluebird/blob/master/benchmark/stats/latest.md https://github.com/petkaantonov/bluebird/blob/master/benchma... [2]: https://github.com/petkaantonov/bluebird/blob/master/API.md#catchfunction-errorclassfunction-predicate-function-handler---promise https://github.com/petkaantonov/bluebird/blob/master/API.md#... [3]: https://github.com/petkaantonov/bluebird/blob/master/API.md#generators https://github.com/petkaantonov/bluebird/blob/master/API.md#... [4]: https://github.com/petkaantonov/bluebird/blob/master/API.md#promisebinddynamic-thisarg---promise https://github.com/petkaantonov/bluebird/blob/master/API.md#...
  • jbeja 13y ago
    That library looks pretty cool and is fairly popular , i will give it a try.
  • phpnode 13y ago
    yes! I recently switched from Q to bluebird and it's been great, and really is exceptionally fast. Also the source code is a work of art - extremely highly optimised.
  • inglor 13y ago
    Bluebird is definitely the way to go, we recently introduced it to an existing 100K LoC code base and we couldn't be happier. The stack traces and error management is amazing and everything is very fast.
  • angersock 13y ago
    We just started using Q for our all--aaaargh, motherfucker. Great to see progress being made, I guess. :(
    • tlrobinson 13y ago
• pak 13y ago
  As you read all these many lines of code to simply find the largest file in a directory, recall that in any language with friendlier blocking/non-blocking semantics, it could be as simple as (e.g. in Ruby) Dir.glob('*').select{|f| File.file?(f) }.max_by{|f| File.size(f) } Plus, the error semantics of this code are sane: if something goes wrong I get an exception I can rescue. If there were no files in the directory, I get nil. Statting a bunch of files is rarely so costly that parallelism is warranted, unless it is impressed upon you as in Node. The cost in readability and dreaming up generators and promises to solve such a mundane problem is likely much worse. Before you charge headlong into building your next webapp with Node, consider this article food for thought.
  • rmgraham 13y ago
    It's definitely a trade-off. What Node gives you is async when you need it at the cost of giving you async even when you don't. The pain and constraints introduced by this forced use of async sure has spawned a lot of wheels of all sizes, colours, and compositions.
    • sync 13y ago
      There's always `fs.readdirSync` etc. node isn't forcing async on you, especially when it comes to file handling.
    • jerf 13y ago
      Ruby is the wrong comparison. Write that same code in Go, suitably translated, and it would be fully "async". I hit Go just because it's the most Algol-ish of the sane languages. There's several other options where you just write the code and the compiler and runtime do the async for you.
  • ggreer 13y ago
    It was a short, simple example to illustrate how yield will improve Node. In real life, you'd probably just do something like: fs.readdirSync(dir).map(function (f) { return { name: f, stat: fs.statSync(path.join(dir, f)) } }).sort(function (a, b) { return b.stat.size - a.stat.size; }); Not going to win any code golf tournaments, but that's JavaScript for you. It was never designed to do this sort of thing. Its strengths lie elsewhere.
• DonPellegrino 13y ago
  I'd like to mention my favorite tool to deal with Callback Hell: Streamline, a JavaScript (and CoffeeScript) language extension that can compile to either 1) callback spaghetti 2) Node-fibers or 3) generator code. It basically writes the mess for you. It lets you focus on the higher order concepts and provides configurable parallel array operations as well as many other convenient features. The callback "error parameter" is thrown as an exception to be caught with a basic try/catch. I highly recommend that any Node and JS enthusiast give it a try. https://github.com/sage/streamlinejs https://github.com/sage/streamlinejs
• rdtsc 13y ago
  Looking at it, in general, ignoring that I know about Node internals, you know what is the simplest way to solve this "problem"? It is something like this: --- callback(find_largest(get_stats(read_files(dir))) --- 1 line. If an error happens, an exception should be raised. The fact that there are multiple such requests happening or that some of those operations have to wait for a select or epoll loop to return, should be handled by an underlying framework. The fact that even with promises you have to write 27 lines of code to do that, should be making you worried.
  • deleted 13y ago
    [deleted]
  • inglor 13y ago
    Please stop spreading FUD. With promises (Bluebird) this sounds like very little code in practice, assuming promisifyAll has been called the FS module: - One line to read all files in a directory - Map them with .map to a .stat call on them, returning array of promises. - Reduce them to the largest element. Something like: var fs = Promise.promisifyAll(require("fs")); var largest = fs.readdirAsync("/your/file/path").map(function(file){ return fs.statAsync(file); }).reduce(function(x,y){ return (x.size > y.size) ? x : y; }); With arrow syntax: var largest = fs.readdirAsync("/your/file/path"). map((file) => fs.statAsync(file)). reduce((x,y) => (x.size > y.size) ? x : y); Alternatively, with Generators using Bluebird promises var largest = Promise.coroutine(function *largest(dirname){ var files = yield fs.readdirAsync(dirname); var sizes = yield files.map(name => fs.statAsync(name)); return files.reduce((x,y) => x.size > y.size ? x : y); }); If one error occurs, the promise rejects. VERY far from 27 lines of code. Not to mention you can of course compose these in one line if you extract them to methods like in your code: var largest = fs.readdirAsync("/your/file/path").map(fs.statAsync).reduce(largest);
    • rdtsc 13y ago
      > Please stop spreading FUD. Was that addressed to strongloop.com? They are the ones that posted an example with promises using 27+/-whitespace lines of code and the initial problem algorithm only had 4 lines. Sounds like you should write a blog post and not them about using promises. I don't use Node.js on daily basis and am not familiar with promises so just commented as an outsider based on the article. Your last one line looks fantastic and makes it clear what is going on. Any reason you can think of why that wasn't the "obvious" solution to the original posters of the article?
• pekk 13y ago
  When we had these things in other languages like Python, node's approach was better. Now that node has adopted them, node is even better still, I guess.
  • davidw 13y ago
    Some other camps ( http://www.erlang.org/ http://www.erlang.org/ ) still strongly prefer their own approach, and I can't say I blame them.
• gfodor 13y ago
  This is satire, right?
• inglor 13y ago
  Why would one use Q promises in Node in 2014? They're very very very slow. Especially when there are alternatives like Bluebird promises that are almost as fast as callbacks.
• ilaksh 13y ago
  Here is how I would do it using ToffeeScript (https://github.com/jiangmiao/toffee-script https://github.com/jiangmiao/toffee-script) fs = require 'fs' async = require 'async' path = require 'path' module.exports = (dir, cb) -> err, files = fs.readdir! dir paths = (path.join dir, file for file in files) er, stats = async.map! paths, fs.stat largest = 0 for stat, i in stats if stat.isFile and stat.size > largest fname = files[i] largest = stat.size cb fname I use ToffeeScript for everything now, since it makes things much easier to read.
• campbellmorgan 13y ago
  an important omission here is the async.auto function which is a phenomenal relatively new addition that takes a amd approach to asynchronously running multiple tasks at the same time: https://github.com/caolan/async#auto https://github.com/caolan/async#auto if you haven't seen it already
• porphyry3 13y ago
  There is another nice feature of ES6 that helps a lot with callbacks and promises: proxies. Check out this article I wrote on Flippin Awesome (http://flippinawesome.org/2013/11/18/taming-asynchronous-javascript-programming-with-ecmascript6/ http://flippinawesome.org/2013/11/18/taming-asynchronous-jav...). The following is some sample code to show how code can be simplified with ES6 proxies, making async functions chainable (example in coffeescript): getUser = (user) -> _("https://npmjs.org/~#{user}") .get() .extractLinks() .filter( -> /package/.test(arguments[0]) ) .map( -> "https://npmjs.org#{arguments[0]}" ) .log() getUser("foo") where `get` is THE jQuery get for Ajax requests returning a promise and is chained with the rest of the functions in a pipeline fashion. I've found that these techniques can work very well for form validation, as [you can check here](http://www.vittoriozaccaria.net/chained/demo.html http://www.vittoriozaccaria.net/chained/demo.html) — Firefox required. Edit: I wrote in markdown but it seems that only verbatim is supported. Edit2: some clarification
• programminggeek 13y ago
  I remember doing a deep dive into node and evented JS a couple years ago and I realized that I couldn't recommend it to my team because the code would become such a huge mess on an large system that whatever performance you gain, you lose in maintainability in a large team environment. The best I could find at the time is IcedCoffeescript, which sort of replaces the whole existing coffee script tooling, so now you are dealing with custom tooling to install for each team member as well. I find it sad that years later it looks like what IcedCoffeescript did is still the cleanest looking solution to the problem. It feels like the evented model is maybe great for smaller things, but horrible for building and reasoning about in larger systems on larger teams.
• pron 13y ago
  The problem with all of these solutions (callbacks, promises, etc.) is that they make you adopt a certain programming style not because it's the most appropriate from a design perspective, but rather to work around scheduling limitations. Lightweight threads first remove the problem, and then let you choose the most appropriate programming style for your domain. See https://news.ycombinator.com/item?id=7179144 https://news.ycombinator.com/item?id=7179144
• ksherlock 13y ago
  All that work and the findLargest routine doesn't seem to benefit whatsoever from fs.readdir or fs.stat being asynchronous.