11 ms·

Wait.for: Sequential programming for Node.js (beta)

52 points by lucio 13y ago

Simple, straightforward abstraction over Fibers. By using wait.for, you can call any nodejs standard async function in sequential/Sync mode, waiting for result data, without blocking node's event loop (thanks to fibers) I'm looking for real-world node apps, to test wait.for

casual_slacker 13y ago
Cool idea. You should mention upfront that you're using Fibers under the hood, since it has important performance implications. Also, your Wait.for function only takes functions right now; but if you check for a .then function you can accept promises too.
- lucio 13y ago
  Thanks. I've put a link to node-fibers, 2nd line of readme. ¿are Fiber's performance problems solvable? ¿what if fibers were integrated in node core?
  - AdrianRossouw 13y ago
    fibers will never be integrated into node core. for what it's worth, I also refuse to use or depend on any code that makes use of node-fibers. I know i'm far from the only person who feels that way too.
deleted 13y ago
[deleted]
- lucio 13y ago
  A fiber is a particularly lightweight thread of execution. basically coroutines. http://en.wikipedia.org/wiki/Fiber_(computer_science) http://en.wikipedia.org/wiki/Fiber_(computer_science) in node.js, fibers are... http://sstur.com/pres-fibers http://sstur.com/pres-fibers
EGreg 13y ago
I don't get it, console.log(wait.for(fs.readfile,'/etc/passwd')); won't that execute console.log immediately, whereas fs.readfile will return the data later? "I don't think it means what you think it means" http://www.youtube.com/watch?v=G2y8Sx4B2Sk http://www.youtube.com/watch?v=G2y8Sx4B2Sk
- lucio 13y ago
  wait.for, waits for fs.readfile to callback, and returns "data" from callback(err,data) wait.for(fs.readfile,'/etc/passwd') == fs.readfileSync('/etc/passwd') wait.for allows you to call any async function as if it were a sync function, without blocking node's event loop. It works inside a fiber. Check the examples.
  - unnuun 13y ago
    Right, but how the heck is this achieved? Could you show a line or two of example pseudo code that demonstrates the core of this functionality?
    - tg3 13y ago
      It uses Fibers (https://github.com/laverdet/node-fibers https://github.com/laverdet/node-fibers). I don't know much about it, but you have to wrap all your code in a call to `Fiber()`, which is written in C++.
      - lucio 13y ago
        Fiber() is under the hood, and I hope ES6 generators -when stable- will provide similar tools, so wait.for could be based on ES6 generators. You don't have to wrap all your code in a call to `Fiber()` full classic node-server sample: var http=require('http'), wait=require('wait.for'); http.createServer(function(req, res) { wait.launchFiber(handleRequest,req, res); //run in a Fiber, keep node spinning }).listen(8080); //in a fiber function handleRequest(req, res) { try { res.writeHead(200, {'Content-Type': 'text/html'}); res.end( "<h1>Demo page</h1>" + "<p>"+ wait.for(fs.readFile,'/etc/somefile') +"</p>" + "<p>"+ wait.for(fs.readFile,'/etc/otherfile') +"</p>"); } catch(err) { res.end('error '+e.message); } }
chrisohara 13y ago
Why not use harmony generators? Generators are provided by V8, don't require an external fibers extension (i.e. hack), and will be available in the next node stable via `node --harmony-generators`. Here's an example for those interested: https://github.com/visionmedia/co https://github.com/visionmedia/co
- lucio 13y ago
  node-fibers is under the hood, and works on the actual node stable. I hope ES6 generators -when stable- will provide similar tools, so wait.for will be based on ES6 generators.
tspike 13y ago
Is this different from async.series? If so, in what way?
- lucio 13y ago
  - syntax - wait.for it's simpler - wait.for uses try/catch I don't know async.series deeply, but, ¿how do you use results from the first function on the second or third? wait.for: try{ data = wait.for(function1); result = wait.for(function2, data); } catch(err){ // error handling }; async.series: async.series([ function1, function2 ], // optional callback function(err, results){ // results is now equal to ['one', 'two'] }); async has a lot of functionality. You can use wait.for and async at the same time. They supply different functionality. Example: results = wait.for(async.map,['file1','file2','file3'], fs.stat);
  - htilford 13y ago
    > ¿how do you use results from the first function on the second or third? waterfall async.waterfall([ function (cb) { . . . cb(stuff); }, function (stuff, cb) { . . .
zongitsrinzler 13y ago
Correct me if I'm wrong but isn't a big part of async programming that it forces you to think about error handling. Working as a Java developer I have seen very lousy error handling mostly because try/catch lets you wrap everything and kill errors easily. Most of the node code I have seen handles errors much better thanks to it's async nature.
- gruseom 13y ago
  If you do it well, you can get very precise error handling, because errors are detected in the context where they first occur. However, to do it well you have to put a try/catch around every asynchronous operation, which makes nested callbacks even harder to read than they already are. Consequently people write code that doesn't handle all possible errors, and when one falls between the cracks, it gets caught only by the top-level default handler (assuming you've declared one), at which point the context is lost. The other problem that comes up is the difficulty of getting a backtrace beyond the current callback. It can be hard to troubleshoot an error in a callback when you can't look back at how the async call got started. In such cases what you really want is a list of backtraces that capture all the relevant history. I seem to recall that people have come up with solutions for this, but can't remember what they are. Maybe someone else can comment?
  - zongitsrinzler 13y ago
    I think this is a good point, sometimes you don't need too precise error handling, then try/catch is very good.
  - tg3 13y ago
    Most Node.js I've seen doesn't make extensive use of Javascript's built-in exception handling, and instead relies on the convention of the first parameter of a callback being an Error object if one occurred (so no try/catch, which doesn't catch async errors anyway). The stack trace is a pain for those operations, and any error that occurs as part of "normal" javascript operation (e.g. ReferenceError) has to be handled like a normal exception, which makes the style inconsistent and therefore a bit harder to manage. I think that the latest or upcoming version of Node is presenting a solution to this in the form of domains, but I'm not too familiar with it.
olegp 13y ago
If all you want to do is make existing async methods synchronous, it's easier to modify the function prototoype like this: https://github.com/olegp/mongo-sync/blob/master/lib/mongo-sync.js#L12 https://github.com/olegp/mongo-sync/blob/master/lib/mongo-sy... or just use Futures that come with fibers. However, in practice you want a higher level of abstraction, since multiple async calls typically end up mapping to one sync call. This applies to streams in particular where you need to wait until they become readable. I've written Common Node (https://github.com/olegp/common-node https://github.com/olegp/common-node) to do just this and made it CommonJS and RingoJS compatible, which means the code written for Common Node runs on both V8 and the JVM. We've been using it in production at https://starthq.com https://starthq.com for a while with great success. Due to some optimizations we put in place, Common Node actually performs faster than Node in some benchmarks.
ilaksh 13y ago
Please also take a look at ToffeeScript which is a brilliant and amazingly relatively unknown solution. https://github.com/jiangmiao/toffee-script https://github.com/jiangmiao/toffee-script
- SeanDav 13y ago
  Not dissing your comment, but I am not a believer in any "compile to" language because ultimately when things go wrong you are forced to deal with the underlying language in any case. Rather become an expert in the underlying language and learn to deal with its idiosyncrasies.
  - ilaksh 13y ago
    I'm not suggesting that people shouldn't learn JavaScript. Anyway all languages compile are abstractions of some lower level (usually multiple levels).
FooBarWidget 13y ago
> What if... Fibers and WaitFor were part of node core? > then you can deprecate almost half the functions at: http://nodejs.org/api/fs.html http://nodejs.org/api/fs.html (clue: the Sync versions) Not quite. I use the Sync versions not to avoid callbacks, but to ensure that nothing else happens while I'm doing the operation. For example, take a look at the Database.reload function from this project (a NoSQL database written in Node): https://github.com/phusion/zangetsu/blob/master/lib/zangetsu/database.js#L103 https://github.com/phusion/zangetsu/blob/master/lib/zangetsu.... Reloading the database contents involves traversing multiple directories and reading multiple files. While reloading the database contents, I do not want the database to be modified. With asynchrony, the amount of different states that I have to be aware of rises so quickly that it's much, MUCH easier to use sync calls and tell the user "reloading is not concurrent, deal with it; at least it's correct". Wait.for does not solve this problem. It seems people believe that you can't run into concurrency problems or race conditions because Node is single threaded. This is false. I even had to write my own locks (https://github.com/phusion/zangetsu/blob/master/lib/zangetsu/database.js#L509 https://github.com/phusion/zangetsu/blob/master/lib/zangetsu...) because Node doesn't provide one.
scriby 13y ago
https://github.com/scriby/asyncblock https://github.com/scriby/asyncblock I'll just leave this here. It's been my pet project for the last couple years, and is a fully featured version of this concept. There's also https://github.com/0ctave/node-sync https://github.com/0ctave/node-sync, which is the same concept.
ibizaman 13y ago
Wouldn't it be easier to just not use Node.js then ? :p

52 points by lucio 13y ago

casual_slacker 13y ago

Cool idea. You should mention upfront that you're using Fibers under the hood, since it has important performance implications. Also, your Wait.for function only takes functions right now; but if you check for a .then function you can accept promises too.

lucio 13y ago
Thanks. I've put a link to node-fibers, 2nd line of readme. ¿are Fiber's performance problems solvable? ¿what if fibers were integrated in node core?
- AdrianRossouw 13y ago
  fibers will never be integrated into node core. for what it's worth, I also refuse to use or depend on any code that makes use of node-fibers. I know i'm far from the only person who feels that way too.

deleted 13y ago

[deleted]

lucio 13y ago
A fiber is a particularly lightweight thread of execution. basically coroutines. http://en.wikipedia.org/wiki/Fiber_(computer_science) http://en.wikipedia.org/wiki/Fiber_(computer_science) in node.js, fibers are... http://sstur.com/pres-fibers http://sstur.com/pres-fibers

EGreg 13y ago

I don't get it, console.log(wait.for(fs.readfile,'/etc/passwd')); won't that execute console.log immediately, whereas fs.readfile will return the data later? "I don't think it means what you think it means" http://www.youtube.com/watch?v=G2y8Sx4B2Sk http://www.youtube.com/watch?v=G2y8Sx4B2Sk

lucio 13y ago
wait.for, waits for fs.readfile to callback, and returns "data" from callback(err,data) wait.for(fs.readfile,'/etc/passwd') == fs.readfileSync('/etc/passwd') wait.for allows you to call any async function as if it were a sync function, without blocking node's event loop. It works inside a fiber. Check the examples.
- unnuun 13y ago
  Right, but how the heck is this achieved? Could you show a line or two of example pseudo code that demonstrates the core of this functionality?
  - tg3 13y ago
    It uses Fibers (https://github.com/laverdet/node-fibers https://github.com/laverdet/node-fibers). I don't know much about it, but you have to wrap all your code in a call to `Fiber()`, which is written in C++.
    - lucio 13y ago
      Fiber() is under the hood, and I hope ES6 generators -when stable- will provide similar tools, so wait.for could be based on ES6 generators. You don't have to wrap all your code in a call to `Fiber()` full classic node-server sample: var http=require('http'), wait=require('wait.for'); http.createServer(function(req, res) { wait.launchFiber(handleRequest,req, res); //run in a Fiber, keep node spinning }).listen(8080); //in a fiber function handleRequest(req, res) { try { res.writeHead(200, {'Content-Type': 'text/html'}); res.end( "<h1>Demo page</h1>" + "<p>"+ wait.for(fs.readFile,'/etc/somefile') +"</p>" + "<p>"+ wait.for(fs.readFile,'/etc/otherfile') +"</p>"); } catch(err) { res.end('error '+e.message); } }

chrisohara 13y ago

Why not use harmony generators? Generators are provided by V8, don't require an external fibers extension (i.e. hack), and will be available in the next node stable via `node --harmony-generators`. Here's an example for those interested: https://github.com/visionmedia/co https://github.com/visionmedia/co

lucio 13y ago
node-fibers is under the hood, and works on the actual node stable. I hope ES6 generators -when stable- will provide similar tools, so wait.for will be based on ES6 generators.

tspike 13y ago

Is this different from async.series? If so, in what way?

lucio 13y ago
- syntax - wait.for it's simpler - wait.for uses try/catch I don't know async.series deeply, but, ¿how do you use results from the first function on the second or third? wait.for: try{ data = wait.for(function1); result = wait.for(function2, data); } catch(err){ // error handling }; async.series: async.series([ function1, function2 ], // optional callback function(err, results){ // results is now equal to ['one', 'two'] }); async has a lot of functionality. You can use wait.for and async at the same time. They supply different functionality. Example: results = wait.for(async.map,['file1','file2','file3'], fs.stat);
- htilford 13y ago
  > ¿how do you use results from the first function on the second or third? waterfall async.waterfall([ function (cb) { . . . cb(stuff); }, function (stuff, cb) { . . .

zongitsrinzler 13y ago

Correct me if I'm wrong but isn't a big part of async programming that it forces you to think about error handling. Working as a Java developer I have seen very lousy error handling mostly because try/catch lets you wrap everything and kill errors easily. Most of the node code I have seen handles errors much better thanks to it's async nature.

gruseom 13y ago
If you do it well, you can get very precise error handling, because errors are detected in the context where they first occur. However, to do it well you have to put a try/catch around every asynchronous operation, which makes nested callbacks even harder to read than they already are. Consequently people write code that doesn't handle all possible errors, and when one falls between the cracks, it gets caught only by the top-level default handler (assuming you've declared one), at which point the context is lost. The other problem that comes up is the difficulty of getting a backtrace beyond the current callback. It can be hard to troubleshoot an error in a callback when you can't look back at how the async call got started. In such cases what you really want is a list of backtraces that capture all the relevant history. I seem to recall that people have come up with solutions for this, but can't remember what they are. Maybe someone else can comment?
- zongitsrinzler 13y ago
  I think this is a good point, sometimes you don't need too precise error handling, then try/catch is very good.
- tg3 13y ago
  Most Node.js I've seen doesn't make extensive use of Javascript's built-in exception handling, and instead relies on the convention of the first parameter of a callback being an Error object if one occurred (so no try/catch, which doesn't catch async errors anyway). The stack trace is a pain for those operations, and any error that occurs as part of "normal" javascript operation (e.g. ReferenceError) has to be handled like a normal exception, which makes the style inconsistent and therefore a bit harder to manage. I think that the latest or upcoming version of Node is presenting a solution to this in the form of domains, but I'm not too familiar with it.

olegp 13y ago

If all you want to do is make existing async methods synchronous, it's easier to modify the function prototoype like this: https://github.com/olegp/mongo-sync/blob/master/lib/mongo-sync.js#L12 https://github.com/olegp/mongo-sync/blob/master/lib/mongo-sy... or just use Futures that come with fibers. However, in practice you want a higher level of abstraction, since multiple async calls typically end up mapping to one sync call. This applies to streams in particular where you need to wait until they become readable. I've written Common Node (https://github.com/olegp/common-node https://github.com/olegp/common-node) to do just this and made it CommonJS and RingoJS compatible, which means the code written for Common Node runs on both V8 and the JVM. We've been using it in production at https://starthq.com https://starthq.com for a while with great success. Due to some optimizations we put in place, Common Node actually performs faster than Node in some benchmarks.

ilaksh 13y ago

Please also take a look at ToffeeScript which is a brilliant and amazingly relatively unknown solution. https://github.com/jiangmiao/toffee-script https://github.com/jiangmiao/toffee-script

SeanDav 13y ago
Not dissing your comment, but I am not a believer in any "compile to" language because ultimately when things go wrong you are forced to deal with the underlying language in any case. Rather become an expert in the underlying language and learn to deal with its idiosyncrasies.
- ilaksh 13y ago
  I'm not suggesting that people shouldn't learn JavaScript. Anyway all languages compile are abstractions of some lower level (usually multiple levels).

FooBarWidget 13y ago

> What if... Fibers and WaitFor were part of node core? > then you can deprecate almost half the functions at: http://nodejs.org/api/fs.html http://nodejs.org/api/fs.html (clue: the Sync versions) Not quite. I use the Sync versions not to avoid callbacks, but to ensure that nothing else happens while I'm doing the operation. For example, take a look at the Database.reload function from this project (a NoSQL database written in Node): https://github.com/phusion/zangetsu/blob/master/lib/zangetsu/database.js#L103 https://github.com/phusion/zangetsu/blob/master/lib/zangetsu.... Reloading the database contents involves traversing multiple directories and reading multiple files. While reloading the database contents, I do not want the database to be modified. With asynchrony, the amount of different states that I have to be aware of rises so quickly that it's much, MUCH easier to use sync calls and tell the user "reloading is not concurrent, deal with it; at least it's correct". Wait.for does not solve this problem. It seems people believe that you can't run into concurrency problems or race conditions because Node is single threaded. This is false. I even had to write my own locks (https://github.com/phusion/zangetsu/blob/master/lib/zangetsu/database.js#L509 https://github.com/phusion/zangetsu/blob/master/lib/zangetsu...) because Node doesn't provide one.

scriby 13y ago

https://github.com/scriby/asyncblock https://github.com/scriby/asyncblock I'll just leave this here. It's been my pet project for the last couple years, and is a fully featured version of this concept. There's also https://github.com/0ctave/node-sync https://github.com/0ctave/node-sync, which is the same concept.

ibizaman 13y ago

Wouldn't it be easier to just not use Node.js then ? :p