7 ms·

Tame.JS: Flow-control by the makers of OkCupid.com

247 points by petar 15y ago

aston 15y ago
I've been waiting for this sort of thing for the longest. The first time I saw the callback passed to a callback of a callback's callback style in Node.js, I wondered why the code couldn't look as nice as the async C++ I'd written at OkCupid. From my lips to Max & Chris's minds... (telepathy!) How long 'till the port to CoffeeScript syntax?
- malgorithms 15y ago
  It's funny, when Max started programming Tame.Js a couple weeks ago, I exchanged emails with a developer who tried to make something similar for CoffeeScript who couldn't convince others it was needed. He said the common response was that if you need something like Tame for async you're not "doing it right." (Obviously we disagree.)
  - deleted 15y ago
    [deleted]
  - jashkenas 15y ago
    I'd be more than happy to explore the addition of Tame.js-style CPS to the CoffeeScript compiler -- but there's a lot of prior work there already: https://github.com/jashkenas/coffee-script/issues/241 https://github.com/jashkenas/coffee-script/issues/241 https://github.com/jashkenas/coffee-script/issues/287 https://github.com/jashkenas/coffee-script/issues/287 https://github.com/jashkenas/coffee-script/issues/350 https://github.com/jashkenas/coffee-script/issues/350 Edit: Things look a little less promising after running a simple test. This input JavaScript: while (i--) { twait { fs.readFile("one"); fs.readFile("two"); } } Gets compiled into this resulting "tamed" JavaScript: var tame = require('tamejs').runtime; var __tame_fn_0 = function (__tame_k) { var __tame_k_implicit = {}; var __tame_fn_1 = function (__tame_k) { if (i --) { var __tame_fn_2 = function (__tame_k) { var __tame_ev = new tame.Event (__tame_k); var __tame_fn_3 = function (__tame_k) { fs .readFile ( "one" ) ; fs .readFile ( "two" ) ; tame.callChain([__tame_k]); }; __tame_fn_3(tame.end); __tame_ev.trigger(); }; tame.callChain([__tame_fn_2, __tame_fn_1, __tame_k]); } else { tame.callChain([__tame_k]); } }; __tame_k_implicit.k_break = __tame_k; __tame_k_implicit.k_continue = function() { __tame_fn_1(__tame_k); }; tame.callChain([__tame_fn_1, __tame_k]); }; __tame_fn_0 (tame.end); ... not so nice to work with or debug. The general conclusion of that series of tickets was that the code generation required to make this CPS transformation work with all edge cases is a bit too hairy to be worth it on balance. Depending on how much sequential async you're doing, YMMV.
    - jjm 15y ago
      Progress was made but looks like it's stalled atm. Since Tame has an endpoint maybe easier to just suck it in? Edit: That compiled script looks a wee scary. I need to be able to fully dive into a debugger with clarity and can't imagine if there were tens or hundreds of lines of this.
    - cpr 15y ago
      As jjm says, with Tame as a concrete, proven solution, it'd sure be great to have CS adopt something like it. Turning the problem "inside out" like this may be the best approach. I.e., it has well-defined semantics and "only" involves JS rewriting, which is CS's forte.
      - po 15y ago
        The branch which attempted the defer keyword also had working code. The input was clean and the edge cases of what to do with return, try/catch, etc had been thought through. The stopping issue was twofold: Coffeescript cares about the readability of both the compiler input and output and… Coffeescript cares about not inserting dynamic lookups and calls into the program which can add undefined performance impact. Perhaps a fork of coffeescript that is targeted for writers of async NodeJS code would be able to safely trade those off. I think jashkenas believes that Javascript will benefit from many little languages springing up to solve specific problems.
    - maxtaco 15y ago
      It's on the list of todos to preserve the input line-numbering in the output code. This would mean some changes to the code emitter, and also to the code emitted to preserve the ordering of statements. In tame/C++, debugging either compiler or runtime errors is straight-ahead, as the developer doesn't need to examine the mangled code. Now if only JavaScript had the equivalent of cpp's #line directive....
      - jashkenas 15y ago
        We're getting there ... maybe this summer. https://bugs.webkit.org/show_bug.cgi?id=63940 https://bugs.webkit.org/show_bug.cgi?id=63940 https://bugzilla.mozilla.org/show_bug.cgi?id=618650 https://bugzilla.mozilla.org/show_bug.cgi?id=618650
  - sawyer 15y ago
    I think adding Tame style CPS to CoffeeScript would be amazing; it looks like an incredibly clean way to write the async code necessary for complex Node apps. From my understanding of the prior work the issue with adding defer or <- to CS was that it required too much overhead to get right in all cases. Does TameJS' approach improve that overhead in any way, or is this essentially the same work that's already been explored for CS, broken out into a dedicated compiler?
    - tjholowaychuk 15y ago
      adding something like this at the grammar level is a massive hack... we can just use coros
  - Cushman 15y ago
    Not surprised. A rough transliteration of `huntMen` for example into CoffeeScript using normal node callback syntax is under 20 lines and relatively easy to understand: huntMen =(buffy)-> soulmates =(buffy, cb, mates=[])-> getMatches buffy, 10, (userids)-> for u in userids do (u)-> getThumbnail u, (thumb)-> isPicAVampire thumb, (is_vamp)-> unless is_vamp getPersonality u, (personality)-> getLastTalked u, match, (last_talked)-> soulmates.push userid: u, thumb, personality, last_talked if soulmates.length >= 10 cb(mates) else soulmates(buffy, cb, mates) else soulmates(buffy, cb, mates) soulmates buffy, (soulmates)-> #Do whatever you need to with soulmates Obviously it's still more of a hassle to deal with than a more featurey async library, but there isn't quite the panic of trying to do the same thing in JS. Edit: For comparison, the Tame.JS style transliterates to ~15 lines of CS— but presumably that would compile to dozens more lines of JavaScript. In exchange, it is lying to you about what the program is actually doing, as opposed to the callback syntax which obscures nothing. So yeah, not surprised if you don't get too much traction with examples like that.
    - malgorithms 15y ago
      I believe this code is painfully unreadable - and that's despite the fact that CoffeeScript is very elegant and easy to read. (It's not you, it's async.) Further, maybe i'm misreading this, but it appears getPersonality and getLastTalked are fired in serial. Can someone who knows CS well fix that and reply? Thanks!
      - Cushman 15y ago
        Ah, that's a good catch. Missed that. Calling those in parallel is legitimately a hassle to roll yourself, something like: ... isPicAVampire thumb, (is_vamp)-> unless is_vamp mate = userid: u, thumb: thumb finish =-> return unless mate.personality? and mate.last_talked? soulmates.push mate if soulmates.length >= 10 cb(mates) else soulmates(buffy, cb, mates) getPersonality u, (p)->finish mate.personality = p getLastTalked u, match, (lt)->finish mate.last_talked = lt Anyway, I'll grant you that code is pretty harsh. It's much nicer with syntax highlighting and wide tabs, but still, fair cop. The flip side is that that code compiles into JavaScript that does exactly what it says.
pmjordan 15y ago
The problem this solves is a serious one, in my experience, even though I find their choice of syntax rather curious. Given that JavaScript 1.7 introduces the yield keyword, it would make sense to add support for that to V8 and implement the infrastructure for concurrent asynchronous I/O around that as a library. The concurrency aspect is, after all, orthogonal to the blocking vs. callback situation, and can easily be done even when using callbacks, with a single callback function called upon completion of all concurrent I/O. I believe the Dojo framework provides such a utility, and I wrote my own simplistic stand-alone mini-library for exactly this a while back. [0] I've run into the problem of endless chained callbacks in C, where it's much worse due to the lack of nested functions, let alone closures or garbage collection.[1] I ended up using the switch block "coroutine" hack [2] for the worst cases, along with dynamically allocated "context" structs to hold "local" variables. A proper macro system would have helped transform blocking code into CPS form. I tried to integrate a SC [3] pass into our build, which could have done it, but ran into all sorts of practical/yak shaving problems, so I ended up with the C preprocessor macro/switch solution for now. In user space, explicit stack-switching with something like swapcontext() is probably preferable, if you can get away with it, but in the kernel this is rather problematic. [0] https://github.com/pmj/MultiAsync-js https://github.com/pmj/MultiAsync-js The reason I wrote my own was because I originally needed it in Rhino, the JVM-based JS implementation, and I couldn't find anything similar that worked there. [1] Yes, there are garbage collectors that work with C, but to my knowledge, none of them can be used in kernel modules. In any case, the other 2 issues are worse and aren't solveable within the language via libraries. [2] http://www.linuxhowtos.org/C_C++/coroutines.htm http://www.linuxhowtos.org/C_C++/coroutines.htm [3] http://super.para.media.kyoto-u.ac.jp/~tasuku/sc/index-e.html http://super.para.media.kyoto-u.ac.jp/~tasuku/sc/index-e.htm...
- deleted 15y ago
  [deleted]
- maxtaco 15y ago
  My first thought for implementing tame.js was with yield, but V8 doesn't currently support it (though it's reserved as a "future" keyword). A direct use of yield (without anything like twait) would probably make node code look more like Python/Twisted code, which while better than the status quo, still can get unmanageable in my experience. Agreed that twait conflates concurrency with blocking/callbacks, but it my experience, it's a natural and useful combination.
starwed 15y ago
There's a bug in huntMen. if (! is_vamp) should instead be if ( is_vamp)... :P
statictype 15y ago
Ever since I first heard of CoffeeScript, I'd been hoping that features like this would make it into the language. It's not realistic to wait for javascript interpreters in the browser to catch up, but this would be a perfect addition for a compiler like CoffeeScript.
frankdenbow 15y ago
Slightly Unrelated: There was a site recently on hn that was a listing of various js libraries, like this one, on one page. What was it?
- matthiaswh 15y ago
  There are two that fit your description and are really useful: http://www.everyjs.com/ http://www.everyjs.com/ http://microjs.com/ http://microjs.com/
  - frankdenbow 15y ago
    thanks! microjs was what i was thinking of
va1en0k 15y ago
That reminds me of monads in Haskell (they can solve this problem (and other as well) matemagically). I've seen somewhere a proposal to add them into Javascript, but I doubt the idea will be loved by public. (By "add them to JS" I mean some syntactic sugar, not a library)
- pmjordan 15y ago
  Proposal? JavaScript 1.7 introduced generators circa 2006, which I do believe lets you implement something like this (in addition to a bunch of other cool things). https://developer.mozilla.org/en/New_in_JavaScript_1.7#Generators https://developer.mozilla.org/en/New_in_JavaScript_1.7#Gener... Unfortunately, V8 doesn't support generators, as far as I can tell.
  - va1en0k 15y ago
    sorry? generators? well, I like generators, but what's your point?
    - pmjordan 15y ago
      yield; Will block execution until the generator is resumed and return control flow to the caller. I haven't actually tried it, but it should be pretty straightforward to wrap a generator function in such a way that it's driven (resumed) by callbacks from async I/O calls.
tjholowaychuk 15y ago
why not just use node-fibers?
- maxtaco 15y ago
  A key advantage of fibers of course is that they preserve exception semantics, whereas tame can't in all cases. I'm not too interested in reviving the stalemated thread v. event religious war. I prefer explicitly-managed events, but if others prefer a more thread-like API, by all means....
  - jrockway 15y ago
    In the end, both are the same thing; managing action-specific state in an application-defined per-action data structure. With OS threads, you let the OS manage the state instead. This can be inefficient with a large number of actions, because most of the state kept has nothing to do with the application itself; it's OS bookkeeping overhead.
- sandstrom 15y ago
  I remember reading something about fibers from Asana (maybe thats the same as node-fibers): http://asana.com/2010/10/adding-fibers-to-v8-efficiency-clarity-in-ssjs/ http://asana.com/2010/10/adding-fibers-to-v8-efficiency-clar...
  - tlrobinson 15y ago
    Same idea, not the same implementation: https://github.com/laverdet/node-fibers https://github.com/laverdet/node-fibers
CrabDude 15y ago
This should do the trick, though I would add a fair amount of this code could be refactored to look prettier with the new control-flow, but is left unaltered for the sake of comparison: var step = require('step'); function huntMen(buffy) { var soulmates = []; step(function() { var group = this.group(); while (soulmates.length < 10) { // Self-executing for closure (function() { var cb = group(); step(function() { // Get 10 candidates for Buffy getMatches(buffy, 10, this); }, function(err, userids) { // don't check userids.length on each loop iteration for (var i = 0, l = userids.length; u = userids[i], i < l; i++) { // Self-executing for closure, or use userids.forEach (function() { step(function() { // get their pic from our pic server getThumbnail (u, this); }, function(err, thumb) { thumb2 = thumb; // ask our pic analyzer to review isPicAVampire(thumb, this); }, function (err, is_vamp) { if (! is_vamp) { // get 2 more pieces of info getPersonality(u, this.parallel()); getLastTalked (u, match, this.parallel()); } else cb(); }, function (err, personality, last_talked) { soulmates.push({ "userid" : match, // match does not exist in example code "thumb" : thumb2, "last_talked" : last_talked, "personality" : personality }); cb(); }); })(); } }); })(); } }, function(err) { // // Our function can now continue to do // whatever it wants with // soulmates... // }); }
Pewpewarrows 15y ago
This seems neat, but after reading through it twice I can't seem to understand how this provides any advantage over just using Deferreds. Someone care to enlighten me?
diamondhead 15y ago
For those who looking for the JavaScript way of the examples; https://gist.github.com/1090228 https://gist.github.com/1090228
lzm 15y ago
The next version of Visual Studio will have something similar: http://msdn.microsoft.com/en-us/vstudio/async.aspx http://msdn.microsoft.com/en-us/vstudio/async.aspx
geuis 15y ago
In what significant ways is this different from deferreds/promises/futures?
- voidfiles 15y ago
  I was wondering the same thing. If the spec evolves tools to handle this stuff, thats one thing, and of course this could be an example solution, but right now I think the problem has a solution without adding extra keywords to the language.
  - bialecki 15y ago
    I would've thought the same thing until I wrote substantial async code in JS and Python. It's not that it doesn't work and you can't do it, it's that the code becomes a mess and their example gets at that. It probably doesn't seem like a big deal, but when you constantly have to pass around callbacks and chain requests together you get this feeling the code could look a lot cleaner than it is. You want asynchronous behavior but with synchronous syntax. This isn't possible without adding something, and, having seen a lot of the solutions out there, it's nice to see someone take a stab at it by changing the language. The cost is huge (preprocessing, etc.), but, speaking from experience, the simplicity of the code you write might make the change worth it. You get the feeling in 5-10 years this will be a solved problem, but I'm not sure any of the solutions out there yet will be the accepted solution.
- sjs 15y ago
  Your code is indented less than it might otherwise be indented, and you type fewer literal functions. Oh and there wasn't any mention of error handling so apparently there is nothing to help you when one of the calls in a twait block throws. I wonder how twait interacts with try/catch. If it's purely a CPS transform then yeah, you're just fucked on error handling. mkevent is like creating a Deferred and twait is like creating a DeferredList and calling when/then on it with the remainder of the code after the twait block used as the callback. Conceptually this is a very specific and concrete subset of functionality that promises give you.
snprbob86 15y ago
Looks a lot like C# 5's await/async keywords: http://blogs.msdn.com/b/ericlippert/archive/tags/async/ http://blogs.msdn.com/b/ericlippert/archive/tags/async/ Cool to see growing interest for this at the language level.
- contextfree 15y ago
  It looks even more like F# async workflows, which compared to the C# async feature have the advantage of being implemented in the current shipping version rather than the next one.
janetjackson 15y ago
This is the wrong solution to the problem, and it's implemented poorly. Use a proper control-flow ( not "flow-control" ) library like https://github.com/caolan/async https://github.com/caolan/async. Furthermore, why would you write an entire custom js parser for this? Why not use some of the many many pre-existing ones that are much more stable, more developed, and well supported.
- __david__ 15y ago
  Seriously? The library solution is pretty ugly--though it's a good solution if you are absolutely dead set against compiling your "javscript". Tame, being a code transformer, makes the equivalent code so much more readable and clean looking. How on earth is readable, clean looking code "the wrong solution to the problem"? I would argue that it's almost always the right solution to a problem.
reustle 15y ago
I've been very happy with "parallel" in the async library by caolan https://github.com/caolan/async https://github.com/caolan/async
yaix 15y ago
This is nice on the browser, but not very useful in nodeJS. twait{} will block and stop my nodeJS process from doing anything else. It would be more useful if I could give twait{} a callback to fire when all its async events completed. Then my nodeJS process could do other stuff while waiting for a twait{} bundle to finish.
- malgorithms 15y ago
  No, that's not the case. twait won't block your Node process from handling other events.
  - yaix 15y ago
    Isn't it waiting to execute anything that follows after a twait code block? If so, then it is blocking. Otherwise, how do you manage to let other code be executed, except the code you don't want to be executed until all functions in the twait block have returned? The would need to be a callback attached to the twait block, but there isn't. So it's blocking. Because that is it's purpose, to block further executing until all data from the non-blocking functions have returned.
    - baudehlo 15y ago
      From looking at it, each twait block is blocking as a unit, but other code elsewhere from the twait block will still run.
deleted 15y ago
[deleted]
teyc 15y ago
Also see JSCEX, which uses the C# keyword await http://groups.google.com/group/nodejs/browse_thread/thread/337f83c028f371c2 http://groups.google.com/group/nodejs/browse_thread/thread/3...
trungonnews 15y ago
So the code we write is clean and easy to understand, but the debugger only work with the compiled version of the code? In another word, write in C, and debug in assembly...
sjs 15y ago
Why hasn't anyone brought up error handling yet? What happens when an error is thrown inside a twait block? What happens when 2 errors are thrown inside a twait block? Tame.js looks nice in that it's very simple to learn but ~300 lines of plain old JavaScript[1] can give you a general purpose deferred/promise library with more flexibility should you need to do something other than wait on N async operations and then use the results. [1] https://github.com/heavylifters/deferred-js/blob/master/lib/deferred.js https://github.com/heavylifters/deferred-js/blob/master/lib/...

247 points by petar 15y ago

aston 15y ago

I've been waiting for this sort of thing for the longest. The first time I saw the callback passed to a callback of a callback's callback style in Node.js, I wondered why the code couldn't look as nice as the async C++ I'd written at OkCupid. From my lips to Max & Chris's minds... (telepathy!) How long 'till the port to CoffeeScript syntax?

malgorithms 15y ago
It's funny, when Max started programming Tame.Js a couple weeks ago, I exchanged emails with a developer who tried to make something similar for CoffeeScript who couldn't convince others it was needed. He said the common response was that if you need something like Tame for async you're not "doing it right." (Obviously we disagree.)
- deleted 15y ago
  [deleted]
- jashkenas 15y ago
  I'd be more than happy to explore the addition of Tame.js-style CPS to the CoffeeScript compiler -- but there's a lot of prior work there already: https://github.com/jashkenas/coffee-script/issues/241 https://github.com/jashkenas/coffee-script/issues/241 https://github.com/jashkenas/coffee-script/issues/287 https://github.com/jashkenas/coffee-script/issues/287 https://github.com/jashkenas/coffee-script/issues/350 https://github.com/jashkenas/coffee-script/issues/350 Edit: Things look a little less promising after running a simple test. This input JavaScript: while (i--) { twait { fs.readFile("one"); fs.readFile("two"); } } Gets compiled into this resulting "tamed" JavaScript: var tame = require('tamejs').runtime; var __tame_fn_0 = function (__tame_k) { var __tame_k_implicit = {}; var __tame_fn_1 = function (__tame_k) { if (i --) { var __tame_fn_2 = function (__tame_k) { var __tame_ev = new tame.Event (__tame_k); var __tame_fn_3 = function (__tame_k) { fs .readFile ( "one" ) ; fs .readFile ( "two" ) ; tame.callChain([__tame_k]); }; __tame_fn_3(tame.end); __tame_ev.trigger(); }; tame.callChain([__tame_fn_2, __tame_fn_1, __tame_k]); } else { tame.callChain([__tame_k]); } }; __tame_k_implicit.k_break = __tame_k; __tame_k_implicit.k_continue = function() { __tame_fn_1(__tame_k); }; tame.callChain([__tame_fn_1, __tame_k]); }; __tame_fn_0 (tame.end); ... not so nice to work with or debug. The general conclusion of that series of tickets was that the code generation required to make this CPS transformation work with all edge cases is a bit too hairy to be worth it on balance. Depending on how much sequential async you're doing, YMMV.
  - jjm 15y ago
    Progress was made but looks like it's stalled atm. Since Tame has an endpoint maybe easier to just suck it in? Edit: That compiled script looks a wee scary. I need to be able to fully dive into a debugger with clarity and can't imagine if there were tens or hundreds of lines of this.
  - cpr 15y ago
    As jjm says, with Tame as a concrete, proven solution, it'd sure be great to have CS adopt something like it. Turning the problem "inside out" like this may be the best approach. I.e., it has well-defined semantics and "only" involves JS rewriting, which is CS's forte.
    - po 15y ago
      The branch which attempted the defer keyword also had working code. The input was clean and the edge cases of what to do with return, try/catch, etc had been thought through. The stopping issue was twofold: Coffeescript cares about the readability of both the compiler input and output and… Coffeescript cares about not inserting dynamic lookups and calls into the program which can add undefined performance impact. Perhaps a fork of coffeescript that is targeted for writers of async NodeJS code would be able to safely trade those off. I think jashkenas believes that Javascript will benefit from many little languages springing up to solve specific problems.
  - maxtaco 15y ago
    It's on the list of todos to preserve the input line-numbering in the output code. This would mean some changes to the code emitter, and also to the code emitted to preserve the ordering of statements. In tame/C++, debugging either compiler or runtime errors is straight-ahead, as the developer doesn't need to examine the mangled code. Now if only JavaScript had the equivalent of cpp's #line directive....
    - jashkenas 15y ago
      We're getting there ... maybe this summer. https://bugs.webkit.org/show_bug.cgi?id=63940 https://bugs.webkit.org/show_bug.cgi?id=63940 https://bugzilla.mozilla.org/show_bug.cgi?id=618650 https://bugzilla.mozilla.org/show_bug.cgi?id=618650
- sawyer 15y ago
  I think adding Tame style CPS to CoffeeScript would be amazing; it looks like an incredibly clean way to write the async code necessary for complex Node apps. From my understanding of the prior work the issue with adding defer or <- to CS was that it required too much overhead to get right in all cases. Does TameJS' approach improve that overhead in any way, or is this essentially the same work that's already been explored for CS, broken out into a dedicated compiler?
  - tjholowaychuk 15y ago
    adding something like this at the grammar level is a massive hack... we can just use coros
- Cushman 15y ago
  Not surprised. A rough transliteration of `huntMen` for example into CoffeeScript using normal node callback syntax is under 20 lines and relatively easy to understand: huntMen =(buffy)-> soulmates =(buffy, cb, mates=[])-> getMatches buffy, 10, (userids)-> for u in userids do (u)-> getThumbnail u, (thumb)-> isPicAVampire thumb, (is_vamp)-> unless is_vamp getPersonality u, (personality)-> getLastTalked u, match, (last_talked)-> soulmates.push userid: u, thumb, personality, last_talked if soulmates.length >= 10 cb(mates) else soulmates(buffy, cb, mates) else soulmates(buffy, cb, mates) soulmates buffy, (soulmates)-> #Do whatever you need to with soulmates Obviously it's still more of a hassle to deal with than a more featurey async library, but there isn't quite the panic of trying to do the same thing in JS. Edit: For comparison, the Tame.JS style transliterates to ~15 lines of CS— but presumably that would compile to dozens more lines of JavaScript. In exchange, it is lying to you about what the program is actually doing, as opposed to the callback syntax which obscures nothing. So yeah, not surprised if you don't get too much traction with examples like that.
  - malgorithms 15y ago
    I believe this code is painfully unreadable - and that's despite the fact that CoffeeScript is very elegant and easy to read. (It's not you, it's async.) Further, maybe i'm misreading this, but it appears getPersonality and getLastTalked are fired in serial. Can someone who knows CS well fix that and reply? Thanks!
    - Cushman 15y ago
      Ah, that's a good catch. Missed that. Calling those in parallel is legitimately a hassle to roll yourself, something like: ... isPicAVampire thumb, (is_vamp)-> unless is_vamp mate = userid: u, thumb: thumb finish =-> return unless mate.personality? and mate.last_talked? soulmates.push mate if soulmates.length >= 10 cb(mates) else soulmates(buffy, cb, mates) getPersonality u, (p)->finish mate.personality = p getLastTalked u, match, (lt)->finish mate.last_talked = lt Anyway, I'll grant you that code is pretty harsh. It's much nicer with syntax highlighting and wide tabs, but still, fair cop. The flip side is that that code compiles into JavaScript that does exactly what it says.

pmjordan 15y ago

The problem this solves is a serious one, in my experience, even though I find their choice of syntax rather curious. Given that JavaScript 1.7 introduces the yield keyword, it would make sense to add support for that to V8 and implement the infrastructure for concurrent asynchronous I/O around that as a library. The concurrency aspect is, after all, orthogonal to the blocking vs. callback situation, and can easily be done even when using callbacks, with a single callback function called upon completion of all concurrent I/O. I believe the Dojo framework provides such a utility, and I wrote my own simplistic stand-alone mini-library for exactly this a while back. [0] I've run into the problem of endless chained callbacks in C, where it's much worse due to the lack of nested functions, let alone closures or garbage collection.[1] I ended up using the switch block "coroutine" hack [2] for the worst cases, along with dynamically allocated "context" structs to hold "local" variables. A proper macro system would have helped transform blocking code into CPS form. I tried to integrate a SC [3] pass into our build, which could have done it, but ran into all sorts of practical/yak shaving problems, so I ended up with the C preprocessor macro/switch solution for now. In user space, explicit stack-switching with something like swapcontext() is probably preferable, if you can get away with it, but in the kernel this is rather problematic. [0] https://github.com/pmj/MultiAsync-js https://github.com/pmj/MultiAsync-js The reason I wrote my own was because I originally needed it in Rhino, the JVM-based JS implementation, and I couldn't find anything similar that worked there. [1] Yes, there are garbage collectors that work with C, but to my knowledge, none of them can be used in kernel modules. In any case, the other 2 issues are worse and aren't solveable within the language via libraries. [2] http://www.linuxhowtos.org/C_C++/coroutines.htm http://www.linuxhowtos.org/C_C++/coroutines.htm [3] http://super.para.media.kyoto-u.ac.jp/~tasuku/sc/index-e.html http://super.para.media.kyoto-u.ac.jp/~tasuku/sc/index-e.htm...

deleted 15y ago
[deleted]
maxtaco 15y ago
My first thought for implementing tame.js was with yield, but V8 doesn't currently support it (though it's reserved as a "future" keyword). A direct use of yield (without anything like twait) would probably make node code look more like Python/Twisted code, which while better than the status quo, still can get unmanageable in my experience. Agreed that twait conflates concurrency with blocking/callbacks, but it my experience, it's a natural and useful combination.

starwed 15y ago

There's a bug in huntMen. if (! is_vamp) should instead be if ( is_vamp)... :P

statictype 15y ago

Ever since I first heard of CoffeeScript, I'd been hoping that features like this would make it into the language. It's not realistic to wait for javascript interpreters in the browser to catch up, but this would be a perfect addition for a compiler like CoffeeScript.

frankdenbow 15y ago

Slightly Unrelated: There was a site recently on hn that was a listing of various js libraries, like this one, on one page. What was it?

matthiaswh 15y ago
There are two that fit your description and are really useful: http://www.everyjs.com/ http://www.everyjs.com/ http://microjs.com/ http://microjs.com/
- frankdenbow 15y ago
  thanks! microjs was what i was thinking of

va1en0k 15y ago

That reminds me of monads in Haskell (they can solve this problem (and other as well) matemagically). I've seen somewhere a proposal to add them into Javascript, but I doubt the idea will be loved by public. (By "add them to JS" I mean some syntactic sugar, not a library)

pmjordan 15y ago
Proposal? JavaScript 1.7 introduced generators circa 2006, which I do believe lets you implement something like this (in addition to a bunch of other cool things). https://developer.mozilla.org/en/New_in_JavaScript_1.7#Generators https://developer.mozilla.org/en/New_in_JavaScript_1.7#Gener... Unfortunately, V8 doesn't support generators, as far as I can tell.
- va1en0k 15y ago
  sorry? generators? well, I like generators, but what's your point?
  - pmjordan 15y ago
    yield; Will block execution until the generator is resumed and return control flow to the caller. I haven't actually tried it, but it should be pretty straightforward to wrap a generator function in such a way that it's driven (resumed) by callbacks from async I/O calls.

tjholowaychuk 15y ago

why not just use node-fibers?

maxtaco 15y ago
A key advantage of fibers of course is that they preserve exception semantics, whereas tame can't in all cases. I'm not too interested in reviving the stalemated thread v. event religious war. I prefer explicitly-managed events, but if others prefer a more thread-like API, by all means....
- jrockway 15y ago
  In the end, both are the same thing; managing action-specific state in an application-defined per-action data structure. With OS threads, you let the OS manage the state instead. This can be inefficient with a large number of actions, because most of the state kept has nothing to do with the application itself; it's OS bookkeeping overhead.
sandstrom 15y ago
I remember reading something about fibers from Asana (maybe thats the same as node-fibers): http://asana.com/2010/10/adding-fibers-to-v8-efficiency-clarity-in-ssjs/ http://asana.com/2010/10/adding-fibers-to-v8-efficiency-clar...
- tlrobinson 15y ago
  Same idea, not the same implementation: https://github.com/laverdet/node-fibers https://github.com/laverdet/node-fibers

CrabDude 15y ago

This should do the trick, though I would add a fair amount of this code could be refactored to look prettier with the new control-flow, but is left unaltered for the sake of comparison: var step = require('step'); function huntMen(buffy) { var soulmates = []; step(function() { var group = this.group(); while (soulmates.length < 10) { // Self-executing for closure (function() { var cb = group(); step(function() { // Get 10 candidates for Buffy getMatches(buffy, 10, this); }, function(err, userids) { // don't check userids.length on each loop iteration for (var i = 0, l = userids.length; u = userids[i], i < l; i++) { // Self-executing for closure, or use userids.forEach (function() { step(function() { // get their pic from our pic server getThumbnail (u, this); }, function(err, thumb) { thumb2 = thumb; // ask our pic analyzer to review isPicAVampire(thumb, this); }, function (err, is_vamp) { if (! is_vamp) { // get 2 more pieces of info getPersonality(u, this.parallel()); getLastTalked (u, match, this.parallel()); } else cb(); }, function (err, personality, last_talked) { soulmates.push({ "userid" : match, // match does not exist in example code "thumb" : thumb2, "last_talked" : last_talked, "personality" : personality }); cb(); }); })(); } }); })(); } }, function(err) { // // Our function can now continue to do // whatever it wants with // soulmates... // }); }

Pewpewarrows 15y ago

This seems neat, but after reading through it twice I can't seem to understand how this provides any advantage over just using Deferreds. Someone care to enlighten me?

diamondhead 15y ago

For those who looking for the JavaScript way of the examples; https://gist.github.com/1090228 https://gist.github.com/1090228

lzm 15y ago

The next version of Visual Studio will have something similar: http://msdn.microsoft.com/en-us/vstudio/async.aspx http://msdn.microsoft.com/en-us/vstudio/async.aspx

geuis 15y ago

In what significant ways is this different from deferreds/promises/futures?

voidfiles 15y ago
I was wondering the same thing. If the spec evolves tools to handle this stuff, thats one thing, and of course this could be an example solution, but right now I think the problem has a solution without adding extra keywords to the language.
- bialecki 15y ago
  I would've thought the same thing until I wrote substantial async code in JS and Python. It's not that it doesn't work and you can't do it, it's that the code becomes a mess and their example gets at that. It probably doesn't seem like a big deal, but when you constantly have to pass around callbacks and chain requests together you get this feeling the code could look a lot cleaner than it is. You want asynchronous behavior but with synchronous syntax. This isn't possible without adding something, and, having seen a lot of the solutions out there, it's nice to see someone take a stab at it by changing the language. The cost is huge (preprocessing, etc.), but, speaking from experience, the simplicity of the code you write might make the change worth it. You get the feeling in 5-10 years this will be a solved problem, but I'm not sure any of the solutions out there yet will be the accepted solution.
sjs 15y ago
Your code is indented less than it might otherwise be indented, and you type fewer literal functions. Oh and there wasn't any mention of error handling so apparently there is nothing to help you when one of the calls in a twait block throws. I wonder how twait interacts with try/catch. If it's purely a CPS transform then yeah, you're just fucked on error handling. mkevent is like creating a Deferred and twait is like creating a DeferredList and calling when/then on it with the remainder of the code after the twait block used as the callback. Conceptually this is a very specific and concrete subset of functionality that promises give you.

snprbob86 15y ago

Looks a lot like C# 5's await/async keywords: http://blogs.msdn.com/b/ericlippert/archive/tags/async/ http://blogs.msdn.com/b/ericlippert/archive/tags/async/ Cool to see growing interest for this at the language level.

contextfree 15y ago
It looks even more like F# async workflows, which compared to the C# async feature have the advantage of being implemented in the current shipping version rather than the next one.

janetjackson 15y ago

This is the wrong solution to the problem, and it's implemented poorly. Use a proper control-flow ( not "flow-control" ) library like https://github.com/caolan/async https://github.com/caolan/async. Furthermore, why would you write an entire custom js parser for this? Why not use some of the many many pre-existing ones that are much more stable, more developed, and well supported.

__david__ 15y ago
Seriously? The library solution is pretty ugly--though it's a good solution if you are absolutely dead set against compiling your "javscript". Tame, being a code transformer, makes the equivalent code so much more readable and clean looking. How on earth is readable, clean looking code "the wrong solution to the problem"? I would argue that it's almost always the right solution to a problem.

reustle 15y ago

I've been very happy with "parallel" in the async library by caolan https://github.com/caolan/async https://github.com/caolan/async

yaix 15y ago

This is nice on the browser, but not very useful in nodeJS. twait{} will block and stop my nodeJS process from doing anything else. It would be more useful if I could give twait{} a callback to fire when all its async events completed. Then my nodeJS process could do other stuff while waiting for a twait{} bundle to finish.

malgorithms 15y ago
No, that's not the case. twait won't block your Node process from handling other events.
- yaix 15y ago
  Isn't it waiting to execute anything that follows after a twait code block? If so, then it is blocking. Otherwise, how do you manage to let other code be executed, except the code you don't want to be executed until all functions in the twait block have returned? The would need to be a callback attached to the twait block, but there isn't. So it's blocking. Because that is it's purpose, to block further executing until all data from the non-blocking functions have returned.
  - baudehlo 15y ago
    From looking at it, each twait block is blocking as a unit, but other code elsewhere from the twait block will still run.

deleted 15y ago

[deleted]

teyc 15y ago

Also see JSCEX, which uses the C# keyword await http://groups.google.com/group/nodejs/browse_thread/thread/337f83c028f371c2 http://groups.google.com/group/nodejs/browse_thread/thread/3...

trungonnews 15y ago

So the code we write is clean and easy to understand, but the debugger only work with the compiled version of the code? In another word, write in C, and debug in assembly...

sjs 15y ago

Why hasn't anyone brought up error handling yet? What happens when an error is thrown inside a twait block? What happens when 2 errors are thrown inside a twait block? Tame.js looks nice in that it's very simple to learn but ~300 lines of plain old JavaScript[1] can give you a general purpose deferred/promise library with more flexibility should you need to do something other than wait on N async operations and then use the results. [1] https://github.com/heavylifters/deferred-js/blob/master/lib/deferred.js https://github.com/heavylifters/deferred-js/blob/master/lib/...