9 ms·

I love async, but I can't code like this

126 points by vamsee 16y ago

kevingadd 16y ago
The response by Isaac filled me with rage as soon as I got to 'No one actually writes code like this'. The guy provides a real example from his application, describes how alternative solutions stack up, and how he'd like to solve the problem, and gets called out for presenting a 'strawman'. Sigh, newsgroups... I personally share the original author's perspective on this and was disappointed to see more sophisticated methods of writing concurrent code (futures, coroutines, etc) dropped from Node in favor of raw callbacks. In my experience, chaining raw callbacks is too easy to get wrong for it to be considered a reliable way of writing all your asynchronous logic, even if it is a great primitive to build a framework on top of. In particular, the effort involved in propagating exceptions all the way through a callback chain is rather significant, and making a mistake in a single function undermines the reliability of the entire chain (usually resulting in work silently being dropped on the floor). I wish more of an effort was made to address this problem.
- abp 16y ago
  You have seen what the author posted later on in the discussion? https://github.com/sam-mccall/node-plate https://github.com/sam-mccall/node-plate
  - DanielRibeiro 16y ago
    There are soo many interesting ways to solve this: http://news.ycombinator.com/item?id=2101724 http://news.ycombinator.com/item?id=2101724
    - Sidnicious 16y ago
      Is that the wrong link, or are you threadjacking?
      - DanielRibeiro 16y ago
        Wow, wished I saw it earlier (can't edit now). Bad paste. Correct link: http://www.infoq.com/articles/surviving-asynchronous-programming-in-javascript http://www.infoq.com/articles/surviving-asynchronous-program...
    - gregschlom 16y ago
      Link unrelated
  - liuliu 16y ago
    I like this syntax more: https://github.com/creationix/do https://github.com/creationix/do
    - moe 16y ago
      I like neither. They're still riddled with boilerplate (anonymous functions) and are quite horrible to read and write (bracket nesting). I sometimes wish node.js (and js in general) would adopt co-routines. They'd seem to be the perfect middle-ground here; async code that looks almost like threaded-code, and no more callback-hell.
  - sam-mccall 16y ago
    And later still: https://github.com/sam-mccall/node-ship https://github.com/sam-mccall/node-ship (supports a few more things than just method chaining)
- patrickaljord 16y ago
  > Sigh, newsgroups... What does this have to do with newsgroups? You can get this kind of response on any medium, no need to blame it on newsgroup.
- raggi 16y ago
  I wholeheartedly agree with you that the comment "No one actually writes code like this" was inaccurate and not useful for the discussion. I see people doing this exact same style of code in EventMachine in ruby, and have done for years. I am continually having to advise newcomers to the evented space that they need to write state machines inside objects to maintain state and process flow, and advising them to wrap callbacks into message sends to these process containers. There is nothing lacking in javascript to do this (create state and function containers), just as there is no lack of OO in ruby. Despite this, newcomers to evented programming always do the same thing. They revert back to a primitive style of programming, writing very long, very tab-deep functions. This is bad practice, you don't do it in an imperative synchronous environment (after some experience), and you shouldn't do it in an async environment. You do not need to use closures for state passing. You shouldn't use closures as general state containers. They're even worse than using stack frames, as they're even less easy to debug (at least walking stack frames is easy with a decent debugger). They're really useful for the minor things, but to encode a complex process, they're completely inappropriate for reasons already discussed (error handling is an ever so common one in the network space). I think this may be what the person was trying to get at with their comment, but they completely lacked substance, which was no use to the OP. Everyone seems to end up discussing some new async-useful algorithms or datastructure that will "save the day", but in reality, until you start encoding application logic processes into decoupled, testable containers, you're always going to fight some complexity. This really has nothing to do with async, the principles of encapsulation and coupling are absolutely well discussed and documented, and merely apply even more vehemently to the async space. The unfortunate fact is that many of the users of these libraries do not factor code in this manner, and as such this discussion comes up again and again. I honestly don't know of any good open source examples of complex async applications in either language (ruby or javascript) that are well structured. I can point you time and time and time again at libraries that are poorly constructed based on the "trivia" examples that "don't require" any structure and so just use a single object/scope and some closures to achieve a minimal use case. There are many possible reasons for this lack of good examples. The reason I haven't written one is because it still takes some care, mostly in architecture, and no one will pay me to write such open source. There probably are some good examples in C, based on libev, libevent, glibs loop, etc, etc. I'm not intimately familiar enough with any example code bases to be able to call them out off the top of my head.
viraptor 16y ago
Does anyone know if there are any plans for coroutines or similar mechanisms in the new versions of ecma/javascript? Python's async frameworks like monocle and diesel work around issues like that by returning another item from a generator and receiving the result again, which works quite well and makes the code look the same as the synchronous one.
- olavk 16y ago
  There is a proposal for adding generators and the yield keyword like in Python, and Mozilla already supports it.
- grayrest 16y ago
  I know Brendan Eich has mentioned that he would like to bring python-like generators into the language. They're in Mozilla's engine, were in ES4, and they're a ES Harmony strawman. http://wiki.ecmascript.org/doku.php?id=strawman:generators http://wiki.ecmascript.org/doku.php?id=strawman:generators Otherwise, I'm not really aware of anything. I know there's been efforts around Promises but I think those have been restricted to libraries and I think that even that's stalled because the CommonJS list is split between Promises/A and Promises/B.
olavk 16y ago
There have been some different attempts to expand JavaScript syntax to make this style of programming more elegant. Narrative JavaScript http://www.neilmix.com/narrativejs/doc/ http://www.neilmix.com/narrativejs/doc/ has a single construct, the arrow which turns everything after the arrow into a closure which is passed as an argument. StratifiedJS (http://www.neilmix.com/narrativejs/doc/ http://www.neilmix.com/narrativejs/doc/) is more rich with a number of new keywords: hold, spawn, wait etc. for asynchronous programming.
- swannodette 16y ago
  Those are very cool projects, but the main issue is that their source transformation is pretty aggressive. Until someone comes up with real, usable debugging facilities for those, they're not going to catch on.
  - mpk 16y ago
    I recently attended a presentation on StratifiedJS and it is really cool, but as you say they do some rather aggressive source transformation. You are basically writing in a different language and loading in an interpreter in JS to handle the new language. ObjectiveJ does the same thing. Take JS, extend it and run it through your own interpreter. It still sort-of looks like JS, but isn't. Not only do you have the overhead of loading in an interpreter but you are also left without good debugging facilities. CoffeeScript doesn't have this problem because it compiles down to readable JS, so if you're familiar with both CoffeeScript and JS you can just use native JS debugging tools to go at it. It does, however require the programmer to be familiar with both languages, so it's usually not the right choice for writing projects that you can hand off to another team at some later point.
    - afri 16y ago
      (Disclaimer: I work on StratifiedJS) The point about not being able to use 'normal' JS debugging facilities with StratifiedJS (or other 'higher-level' languages compiling to JS) is certainly true, but IMO it is not nearly as bad as it sounds. Debugging highly asynchronous programs (whether written in 'straight' JS or in a higher-level language) with a 'normal' JS debugger isn't really much help: the callstacks that the debugger spits out in no way correlate to the asynchronous actions that are going on. Having an abstraction that sits on top of JS, as it is the case with StratifiedJS, gives us the opportunity to write a debugger that spits out stack traces that actually reflect the true causal state of your program logic. Granted, we don't have such a debugger yet, but we will in the not-too-distant future. What is already working is that StratifiedJS will amend exceptions with the correct linenumber and module name of the original StratifiedJS sources that threw the exception. So you don't have to manually correlate the generated source code with the original SJS source code, like you would need to do in systems that do a more or less 1-to-1 translation to JS.
jhpriestley 16y ago
This is why I always find it sad that people are averse to learning about "abstract, impractical" concepts like monads. Monads are abstract, but they are far from impractical: they solve problems which you are already encountering, and even already identifying as problems. The best solution would be to add monadic syntax to javascript. You would get CPS, coroutines, error propagation, early failure, and several other capabilities all using the same syntax. Shit like generators or coroutines are just ad-hoc special-cased versions.
- olavk 16y ago
  I would like to try to do something like that! Do you have thought about how a syntax like "do" would look in JavaScript?
  - jhpriestley 16y ago
    Probably a lot like javascript. with_monad(foo){ var x = bar(); baz(x); } where foo includes a definition of bind and return (return should be renamed, there is already a return in js). Subcalls like baz(x) would be interpreted within the same monad. Basically you would just make semicolon a function with dynamic extent, rather than a language builtin. I don't know if it would be possible to monadize functions which use break, continue, or return. While and if would have to be special cased.
    - olavk 16y ago
      But wouldn't that require that all calls inside the with_monad block is of the same monad type? Wouldn't the syntax need a way of distinguishing between monadic and non-monadic operations (like using "let" rather than "<-" in Haskells do-syntax)?
- swannodette 16y ago
  I'm completely convinced that the growing ubiquity of asynchronous and multithreaded code will all drive us to embrace Object Oriented Programming 2.0 AKA Functional Programming. Can't come fast enough for me.
nathansobo 16y ago
JavaScript needs coroutines. They would solve a lot of problems.
- icey 16y ago
  They won't be coming in node.js: http://twitter.com/#!/ryah/status/25685501193232384 http://twitter.com/#!/ryah/status/25685501193232384
  - silentbicycle 16y ago
    But, why not? I've done web programming in Lua (w/ WSAPI); Lua is generally like Javascript's non-glue-huffing older brother, and coroutines/iterators remove a LOT of issues. I was working on an event-loop-based web server+framework in Lua, but then I got into Erlang and realized how utterly lost the whole endeavor was. (But who wants to program in Erlang? I hear it's bad at strings, or something.)
amadiver 16y ago
`Bind` with named methods goes a long way to cleaning up nested callbacks. Here's a (quick+dirty; I know I'm missing some `do`'s in there) version of the same code in CoffeeScript: _ = fileMenu: -> mainWindow.menu "File", (e,f) => this.onFile(e,f) onFile: (err,file) -> if err then throw err file.openMenu (e,m) => this.onOpenMenu(e,m) onOpenMenu: (err,menu) -> if err then throw err menu.item "Open", (e,i) => this.onOpen(e,i) onOpen: (err,item) -> if err then throw err mainWindow.getChild type('Window'), (e,d) => this.onDialog(e,d) onDialog: (err,dialog) -> if err then throw err #... _.fileMenu() ( `=>` is CoffeeScript for a function bound to the current scope )
- flashingpumpkin 16y ago
  We've adopted a way that allows us to do basically this: mainWindow.menu "File", (err, file)-> openMenu err, (err, menu)-> getItem err, "Open", (err, item)-> click err, item, (err)-> getChild err, getChildType("Window"), (err, dialog)-> if err? and err instanceof NoFileError # do stuff if err? and err instanceof InvalidClickError # do other stuff if err? throw err # do stuff with dialog if no err Each function takes as first argument an error and as last a callback. If there's an error, the function just passes the error up the callback stack until it gets dealed with. fun = (err, args..., callback) You can't pull that off if your code is heavily object oriented though: mainWindow.menu "File", (err, file)-> # If mainWindow.menu errors, file is undefined # and the call to openMenu throws an error file.openMenu err, (err, menu)-> # ... Breaking your code/api up with modules instead into classes fixes that.
deleted 16y ago
[deleted]
teyc 16y ago
The new C# 5.0 proposes to bring asynchronous keywords, which I believe works like macros to turn imperative code into continuations. The “await” operator used twice in that method does not mean “this method now blocks the current thread until the asynchronous operation returns”. That would be making the asynchronous operation back into a synchronous operation, which is precisely what we are attempting to avoid. Rather, it means the opposite of that; it means “if the task we are awaiting has not yet completed then sign up the rest of this method as the continuation of that task, and then return to your caller immediately; the task will invoke the continuation when it completes.” http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/asynchronous-programming-in-c-5-0-part-two-whence-await.aspx http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/async... async void ArchiveDocuments(List<Url> urls) { Task archive = null; for(int i = 0; i < urls.Count; ++i) { var document = await FetchAsync(urls[i]); if (archive != null) await archive; archive = ArchiveAsync(document); } } Microsoft has been very lucky to have someone like Anders Hejlsberg, who has steadily brought a stream of features to an enterprise programming language.
- contextfree 16y ago
  Note that this was (at least conceptually) derived from a (more or less) equivalent F# feature that exists in the shipping version today.
  - gtani 16y ago
    backgrounder: http://www.infoq.com/news/2011/01/Why-Async;jsessionid=512A27F17C5A622A701F5DE128F3CB6B http://www.infoq.com/news/2011/01/Why-Async;jsessionid=512A2...
- pointernil 16y ago
  Yes, what is true is the fact, that they are finally working on a better ways to support async-code in a more frictionless way... (yes, Hejlsberg, F# team and Async Team rock, as does Erik Maijer) They are still not where they would like to be though i guess: just check out what subtle issues you can run into by using this quite intuitive way of handling async resp. auto continuations: "Await, and UI, and deadlocks! Oh my!" http://blogs.msdn.com/b/pfxteam/archive/2011/01/13/10115163.aspx http://blogs.msdn.com/b/pfxteam/archive/2011/01/13/10115163.... I'd say they are on the right track, but still not there for big time ... just imagine this in hands of millions of average devs under pressure...
EGreg 16y ago
I felt the same way, but thankfully Javascript lets you write really cool things that handle the flow of your code in ways that correspond to the way you think of it. I encourage you to check out http://news.ycombinator.com/item?id=2101789 http://news.ycombinator.com/item?id=2101789 ... with those functions javascript will be pleasant again.
mpk 16y ago
I really like JS and it has excellent closure support, but it's really obvious that heavy async programming like we're doing nowadays with AJAX is not what it was designed for. The 'nested callback' problem is one I've encountered over and over. I've figured out a relatively nice way to work around it. The solution goes something like this. - Take the individual actions that you want and put them in closures. - These closures take 1 argument, which is always a callback function that takes no arguments (this is the convention you structure everything around) - Put these closures in an array, in the order that you want them executed - Pass the array to a function that executes them for you (in Haskell it's called a sequence) I have a really quick and really dirty presentation about it here, http://www.moondust.dds.nl/text/async-presentation/ http://www.moondust.dds.nl/text/async-presentation/ Some code on github up here, https://github.com/michiel/asynchelper-js https://github.com/michiel/asynchelper-js - the sequencer.js function is verbose, if you read the next SO link I've included a three line example of implementing it. And a stackoverflow answer using this code and some explanation here, http://stackoverflow.com/questions/4462605/jquery-ajax-each-callback-next-each-firing-before-ajax-completed/4462955#4462955 http://stackoverflow.com/questions/4462605/jquery-ajax-each-... And another one as a gist on github here, https://gist.github.com/604730 https://gist.github.com/604730 (for rendering a massive table from JSON data). There's a lot more to this sequencer pattern. You can nest sequences, you can have the sequencer run each callback using setTimeout(callback, 0) to release control back to the main loop and get out of 'script is doing too much' popups, etc. Need to make an async call? Make a closure that performs the async call and invoke the callback argument in your async callback. Need to add some random JS code to a sequence? Wrap it in a closure and call the callback when you're done. If the random JS code requires an async call, see the previous paragraph. Error handling, of course, is still awful. You can extend the sequencer convention to add error closures as a second parameter, you can do everything inside the closures in try/catch blocks and finally always call the callback, etc. But that just tends to make a mess of things. So the examples just assume everything is going fine. It's not an ideal solution, but it's native JS and works everywhere. It allows you to structure your code into functional blocks and work on them separately instead of making everything one big nested callback mess.
lvh 16y ago
It would be nice if people stop saying 'async' when they mean 'callbacks'. Twisted's inlineCallbacks, Corotwine, Gevent... are all plenty async, and there's not an explicit callback in sight. (I suppose you could argue that the yield in inlineCallbacks and Monocle delimits a callback. You wouldn't be wrong.)
- wladimir 16y ago
  Of those I know gevent, which is indeed very nice, as it is 'implicitly' async. It has the performance benefits of being async, but doesn't require you to write callback-based macaroni network code.
  - lvh 16y ago
    Corotwine is exactly alike. Personally, I prefer inlineCallbacks. Explicit is better than implicit... Basically I can say, okay, so, here something slow is going to happen, and I'm willing to yield (hey look it even uses an appropriate keyword) control over to the reactor. My main point is still this: okay, yes, sometimes callback code can get ugly (with the apparent implicit assumption plain old "blocking" code is always pretty...), but that has nothing to do with the async backing.
    - wladimir 16y ago
      Plain old blocking code is not always pretty, but it is much more clear what happens, and how errors are handled; all steps are in one place. Also, in my experience, it is usually much shorter.
fleitz 16y ago
Use F# instead let async_copy (in:Stream) (out:Stream) = asnyc { let buf = Array.create 8192 let rec copy = async { return! match in.ReadAsync(0,buf.Length,buf) with | 0 -> () | x -> do! out.WriteAsnyc(0,x,buf); return! copy; } return! copy; } There's an async copy routine, not much more overhead than a sync copy. let sync_copy (in:Stream) (out:Stream) = let buf = Array.create 8192 let rec copy = match in.Read(0,buf.Length,buf) with | 0 -> () | x -> out.Write(0,x,buf); copy copy
- deleted 16y ago
  [deleted]
- icey 16y ago
  Slight typo in the first line of your code; asnyc should be async. Line 6, WriteAsnyc should be WriteAsync The intentions were very clear, just posting this in case someone tries to copy & paste it.
swannodette 16y ago
Perhaps what we need is something like CJSR (Common JavaScript Runtime). Languages that compile to JavaScript generate standardized debugging information as JSON (CoffeeScript has been pushing for this). These languages can then use a shared set of in-browser debugging tools (JSDB?). The fact that Safari, Chrome, and FF all support common browser extension architectures as this makes this solution even more compelling. Agressive source transformation becomes less of an issue.
Sephr 16y ago
Using async.js (http://eligrey.com/blog/post/pausing-javascript-with-async-js http://eligrey.com/blog/post/pausing-javascript-with-async-j...), you can do away with callbacks and still have asynchronous code by using the yield statement. The following is an example (taken from my blog post) of asynchronous code using async.js: yield to.request("/feedback", "POST", ( yield to.prompt("What are your impressions of async.js?") )); yield to.inform("Thanks for your feedback!"); // do more stuff here The equivalent code with callbacks is as follows: async.prompt( ["What are your impressions of async.js?"], function (response) { async.request( ["feedback", "POST", response], function () { async.inform( ["Thanks for your feedback!"], function () { // do more stuff here } ); } ); } ); It makes it very easy to handle DOM events, as demonstrated below. var clickEvent = yield myButton.next("click"); Full disclosure: I wrote async.js.
- Groxx 16y ago
  Since you wrote it: any idea what browser support of Javascript 1.7 is at? Wikipedia[1] currently thinks only Firefox has past 1.5, but I can't find anything else that isn't ancient (in web years), and Wikipedia's tables of things tend to get inaccurate pretty quickly. [1] http://en.wikipedia.org/wiki/JavaScript#Versions http://en.wikipedia.org/wiki/JavaScript#Versions
  - Sephr 16y ago
    Only Mozilla supports JavaScript (which is a mix of JavaScript-only features and ECMAScript; it's a little complicated), and everyone else implements ECMAScript. Some JavaScript features are in discussion of being added to ECMAScript, but no current ECMAScript engines (e.g. V8) have implemented them yet, barring the partial destructuring assignment implementation supported by Opera's Futhark ECMAScript engine.
    - sid0 16y ago
      I think yield's coming to ECMAScript Harmony. That should simplify things a lot.
statictype 16y ago
I think the most likely solution would come from a language extension to something like Coffeescript. It could work similar to the async monad in F#: #asnc x = () -> var f = window.menu("file"); var m = f.openmenu(); m.click(); would unwind as: window.menu("file",function(f) { f.open(function(m) { m.click(); }); });
locopati 16y ago
Is there a reason that events don't work in the situation? The async result handler fires an event 'getXFinished' and any function that needs to respond does so. When they are finished, they fire events. And additional functions are called. And so on.
jamwt 16y ago
Others have mentioned coroutines; just to avoid repeating a good conversation that already happened on all this, here's the link: http://news.ycombinator.com/item?id=1549023 http://news.ycombinator.com/item?id=1549023
Kilimanjaro 16y ago
There is a huge gap for a server side implementation of JS and node is not the one the masses are asking for, despite all the hype surrounding it. Just take python, ruby or php, and make JS on the server on par with them. Forget about creating servers, async stuff and coroutines, 90% of moms and pops websites don't need them. There is a huge momentum behind SSJS but it is all being wasted waiting for node to be the mother of all js apps. It will never be.
mmaunder 16y ago
I tend to use Node for specific tasks like running a process that may take from 0 to 20 seconds without having a thread block. Using it for discrete tasks that it's suited for and then doing the bulk of my programming on regular vanilla synchronous server processes has worked for me as a way to manage complexity. Until a standard model of async emerges I think programming large web apps on node will feel a little like hiking with a stone in your shoe.
nikils 16y ago
isn't arrows http://www.cs.umd.edu/projects/PL/arrowlets/example-drag-and-drop.xhtml http://www.cs.umd.edu/projects/PL/arrowlets/example-drag-and... supposed to solve this problem.
moomin 16y ago
I can't help feeling that all of the examples contain too much copy and pasting. This is dealt with by taking a higher order approach to the problem. Which is what async.js is for. doSomething = (c) -> async.waterfall [ (c) -> mainWindow.menu "File", c (file, c) -> file.openMenu c (menu, c) -> menu.item "Open", c (item, c) -> item.click c (c) -> mainWindow.getChild type("Window"), c (dialog, c) -> dialog.doSomething c null ] It's not perfect, but I think it's pretty readable and succinct.
uriel 16y ago
This is one of the reasons I love CSP and languages with CSP-like concurrency like Go (and Erlang, although it is more CSP-inspired than CSP-like). I really can't understand why anyone would want to use anything else for concurrent programming.
sam-mccall 16y ago
Sorry I missed the discussion! Just wanted to add a pointer to what I ended up using: https://github.com/sam-mccall/ship https://github.com/sam-mccall/ship It's lets you create pseudo-threads, you pretend you're calling a sync function, and get a promise back, which you can call methods on (giving more promises) etc. It needs a rewrite to allow objects in different pseudo-threads to interact - i.e. the threads to cross. Currently it's just a queue of actions to perform which doesn't allow this.

126 points by vamsee 16y ago

The response by Isaac filled me with rage as soon as I got to 'No one actually writes code like this'. The guy provides a real example from his application, describes how alternative solutions stack up, and how he'd like to solve the problem, and gets called out for presenting a 'strawman'. Sigh, newsgroups... I personally share the original author's perspective on this and was disappointed to see more sophisticated methods of writing concurrent code (futures, coroutines, etc) dropped from Node in favor of raw callbacks. In my experience, chaining raw callbacks is too easy to get wrong for it to be considered a reliable way of writing all your asynchronous logic, even if it is a great primitive to build a framework on top of. In particular, the effort involved in propagating exceptions all the way through a callback chain is rather significant, and making a mistake in a single function undermines the reliability of the entire chain (usually resulting in work silently being dropped on the floor). I wish more of an effort was made to address this problem.

abp 16y ago
You have seen what the author posted later on in the discussion? https://github.com/sam-mccall/node-plate https://github.com/sam-mccall/node-plate
- DanielRibeiro 16y ago
  There are soo many interesting ways to solve this: http://news.ycombinator.com/item?id=2101724 http://news.ycombinator.com/item?id=2101724
  - Sidnicious 16y ago
    Is that the wrong link, or are you threadjacking?
    - DanielRibeiro 16y ago
      Wow, wished I saw it earlier (can't edit now). Bad paste. Correct link: http://www.infoq.com/articles/surviving-asynchronous-programming-in-javascript http://www.infoq.com/articles/surviving-asynchronous-program...
  - gregschlom 16y ago
    Link unrelated
- liuliu 16y ago
  I like this syntax more: https://github.com/creationix/do https://github.com/creationix/do
  - moe 16y ago
    I like neither. They're still riddled with boilerplate (anonymous functions) and are quite horrible to read and write (bracket nesting). I sometimes wish node.js (and js in general) would adopt co-routines. They'd seem to be the perfect middle-ground here; async code that looks almost like threaded-code, and no more callback-hell.
- sam-mccall 16y ago
  And later still: https://github.com/sam-mccall/node-ship https://github.com/sam-mccall/node-ship (supports a few more things than just method chaining)
patrickaljord 16y ago
> Sigh, newsgroups... What does this have to do with newsgroups? You can get this kind of response on any medium, no need to blame it on newsgroup.
raggi 16y ago
I wholeheartedly agree with you that the comment "No one actually writes code like this" was inaccurate and not useful for the discussion. I see people doing this exact same style of code in EventMachine in ruby, and have done for years. I am continually having to advise newcomers to the evented space that they need to write state machines inside objects to maintain state and process flow, and advising them to wrap callbacks into message sends to these process containers. There is nothing lacking in javascript to do this (create state and function containers), just as there is no lack of OO in ruby. Despite this, newcomers to evented programming always do the same thing. They revert back to a primitive style of programming, writing very long, very tab-deep functions. This is bad practice, you don't do it in an imperative synchronous environment (after some experience), and you shouldn't do it in an async environment. You do not need to use closures for state passing. You shouldn't use closures as general state containers. They're even worse than using stack frames, as they're even less easy to debug (at least walking stack frames is easy with a decent debugger). They're really useful for the minor things, but to encode a complex process, they're completely inappropriate for reasons already discussed (error handling is an ever so common one in the network space). I think this may be what the person was trying to get at with their comment, but they completely lacked substance, which was no use to the OP. Everyone seems to end up discussing some new async-useful algorithms or datastructure that will "save the day", but in reality, until you start encoding application logic processes into decoupled, testable containers, you're always going to fight some complexity. This really has nothing to do with async, the principles of encapsulation and coupling are absolutely well discussed and documented, and merely apply even more vehemently to the async space. The unfortunate fact is that many of the users of these libraries do not factor code in this manner, and as such this discussion comes up again and again. I honestly don't know of any good open source examples of complex async applications in either language (ruby or javascript) that are well structured. I can point you time and time and time again at libraries that are poorly constructed based on the "trivia" examples that "don't require" any structure and so just use a single object/scope and some closures to achieve a minimal use case. There are many possible reasons for this lack of good examples. The reason I haven't written one is because it still takes some care, mostly in architecture, and no one will pay me to write such open source. There probably are some good examples in C, based on libev, libevent, glibs loop, etc, etc. I'm not intimately familiar enough with any example code bases to be able to call them out off the top of my head.

viraptor 16y ago

Does anyone know if there are any plans for coroutines or similar mechanisms in the new versions of ecma/javascript? Python's async frameworks like monocle and diesel work around issues like that by returning another item from a generator and receiving the result again, which works quite well and makes the code look the same as the synchronous one.

olavk 16y ago
There is a proposal for adding generators and the yield keyword like in Python, and Mozilla already supports it.
grayrest 16y ago
I know Brendan Eich has mentioned that he would like to bring python-like generators into the language. They're in Mozilla's engine, were in ES4, and they're a ES Harmony strawman. http://wiki.ecmascript.org/doku.php?id=strawman:generators http://wiki.ecmascript.org/doku.php?id=strawman:generators Otherwise, I'm not really aware of anything. I know there's been efforts around Promises but I think those have been restricted to libraries and I think that even that's stalled because the CommonJS list is split between Promises/A and Promises/B.

olavk 16y ago

There have been some different attempts to expand JavaScript syntax to make this style of programming more elegant. Narrative JavaScript http://www.neilmix.com/narrativejs/doc/ http://www.neilmix.com/narrativejs/doc/ has a single construct, the arrow which turns everything after the arrow into a closure which is passed as an argument. StratifiedJS (http://www.neilmix.com/narrativejs/doc/ http://www.neilmix.com/narrativejs/doc/) is more rich with a number of new keywords: hold, spawn, wait etc. for asynchronous programming.

swannodette 16y ago
Those are very cool projects, but the main issue is that their source transformation is pretty aggressive. Until someone comes up with real, usable debugging facilities for those, they're not going to catch on.
- mpk 16y ago
  I recently attended a presentation on StratifiedJS and it is really cool, but as you say they do some rather aggressive source transformation. You are basically writing in a different language and loading in an interpreter in JS to handle the new language. ObjectiveJ does the same thing. Take JS, extend it and run it through your own interpreter. It still sort-of looks like JS, but isn't. Not only do you have the overhead of loading in an interpreter but you are also left without good debugging facilities. CoffeeScript doesn't have this problem because it compiles down to readable JS, so if you're familiar with both CoffeeScript and JS you can just use native JS debugging tools to go at it. It does, however require the programmer to be familiar with both languages, so it's usually not the right choice for writing projects that you can hand off to another team at some later point.
  - afri 16y ago
    (Disclaimer: I work on StratifiedJS) The point about not being able to use 'normal' JS debugging facilities with StratifiedJS (or other 'higher-level' languages compiling to JS) is certainly true, but IMO it is not nearly as bad as it sounds. Debugging highly asynchronous programs (whether written in 'straight' JS or in a higher-level language) with a 'normal' JS debugger isn't really much help: the callstacks that the debugger spits out in no way correlate to the asynchronous actions that are going on. Having an abstraction that sits on top of JS, as it is the case with StratifiedJS, gives us the opportunity to write a debugger that spits out stack traces that actually reflect the true causal state of your program logic. Granted, we don't have such a debugger yet, but we will in the not-too-distant future. What is already working is that StratifiedJS will amend exceptions with the correct linenumber and module name of the original StratifiedJS sources that threw the exception. So you don't have to manually correlate the generated source code with the original SJS source code, like you would need to do in systems that do a more or less 1-to-1 translation to JS.

jhpriestley 16y ago

This is why I always find it sad that people are averse to learning about "abstract, impractical" concepts like monads. Monads are abstract, but they are far from impractical: they solve problems which you are already encountering, and even already identifying as problems. The best solution would be to add monadic syntax to javascript. You would get CPS, coroutines, error propagation, early failure, and several other capabilities all using the same syntax. Shit like generators or coroutines are just ad-hoc special-cased versions.

olavk 16y ago
I would like to try to do something like that! Do you have thought about how a syntax like "do" would look in JavaScript?
- jhpriestley 16y ago
  Probably a lot like javascript. with_monad(foo){ var x = bar(); baz(x); } where foo includes a definition of bind and return (return should be renamed, there is already a return in js). Subcalls like baz(x) would be interpreted within the same monad. Basically you would just make semicolon a function with dynamic extent, rather than a language builtin. I don't know if it would be possible to monadize functions which use break, continue, or return. While and if would have to be special cased.
  - olavk 16y ago
    But wouldn't that require that all calls inside the with_monad block is of the same monad type? Wouldn't the syntax need a way of distinguishing between monadic and non-monadic operations (like using "let" rather than "<-" in Haskells do-syntax)?
swannodette 16y ago
I'm completely convinced that the growing ubiquity of asynchronous and multithreaded code will all drive us to embrace Object Oriented Programming 2.0 AKA Functional Programming. Can't come fast enough for me.

nathansobo 16y ago

JavaScript needs coroutines. They would solve a lot of problems.

icey 16y ago
They won't be coming in node.js: http://twitter.com/#!/ryah/status/25685501193232384 http://twitter.com/#!/ryah/status/25685501193232384
- silentbicycle 16y ago
  But, why not? I've done web programming in Lua (w/ WSAPI); Lua is generally like Javascript's non-glue-huffing older brother, and coroutines/iterators remove a LOT of issues. I was working on an event-loop-based web server+framework in Lua, but then I got into Erlang and realized how utterly lost the whole endeavor was. (But who wants to program in Erlang? I hear it's bad at strings, or something.)

amadiver 16y ago

`Bind` with named methods goes a long way to cleaning up nested callbacks. Here's a (quick+dirty; I know I'm missing some `do`'s in there) version of the same code in CoffeeScript: _ = fileMenu: -> mainWindow.menu "File", (e,f) => this.onFile(e,f) onFile: (err,file) -> if err then throw err file.openMenu (e,m) => this.onOpenMenu(e,m) onOpenMenu: (err,menu) -> if err then throw err menu.item "Open", (e,i) => this.onOpen(e,i) onOpen: (err,item) -> if err then throw err mainWindow.getChild type('Window'), (e,d) => this.onDialog(e,d) onDialog: (err,dialog) -> if err then throw err #... _.fileMenu() ( `=>` is CoffeeScript for a function bound to the current scope )

flashingpumpkin 16y ago
We've adopted a way that allows us to do basically this: mainWindow.menu "File", (err, file)-> openMenu err, (err, menu)-> getItem err, "Open", (err, item)-> click err, item, (err)-> getChild err, getChildType("Window"), (err, dialog)-> if err? and err instanceof NoFileError # do stuff if err? and err instanceof InvalidClickError # do other stuff if err? throw err # do stuff with dialog if no err Each function takes as first argument an error and as last a callback. If there's an error, the function just passes the error up the callback stack until it gets dealed with. fun = (err, args..., callback) You can't pull that off if your code is heavily object oriented though: mainWindow.menu "File", (err, file)-> # If mainWindow.menu errors, file is undefined # and the call to openMenu throws an error file.openMenu err, (err, menu)-> # ... Breaking your code/api up with modules instead into classes fixes that.

deleted 16y ago

[deleted]

teyc 16y ago

The new C# 5.0 proposes to bring asynchronous keywords, which I believe works like macros to turn imperative code into continuations. The “await” operator used twice in that method does not mean “this method now blocks the current thread until the asynchronous operation returns”. That would be making the asynchronous operation back into a synchronous operation, which is precisely what we are attempting to avoid. Rather, it means the opposite of that; it means “if the task we are awaiting has not yet completed then sign up the rest of this method as the continuation of that task, and then return to your caller immediately; the task will invoke the continuation when it completes.” http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/asynchronous-programming-in-c-5-0-part-two-whence-await.aspx http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/async... async void ArchiveDocuments(List<Url> urls) { Task archive = null; for(int i = 0; i < urls.Count; ++i) { var document = await FetchAsync(urls[i]); if (archive != null) await archive; archive = ArchiveAsync(document); } } Microsoft has been very lucky to have someone like Anders Hejlsberg, who has steadily brought a stream of features to an enterprise programming language.

contextfree 16y ago
Note that this was (at least conceptually) derived from a (more or less) equivalent F# feature that exists in the shipping version today.
- gtani 16y ago
  backgrounder: http://www.infoq.com/news/2011/01/Why-Async;jsessionid=512A27F17C5A622A701F5DE128F3CB6B http://www.infoq.com/news/2011/01/Why-Async;jsessionid=512A2...
pointernil 16y ago
Yes, what is true is the fact, that they are finally working on a better ways to support async-code in a more frictionless way... (yes, Hejlsberg, F# team and Async Team rock, as does Erik Maijer) They are still not where they would like to be though i guess: just check out what subtle issues you can run into by using this quite intuitive way of handling async resp. auto continuations: "Await, and UI, and deadlocks! Oh my!" http://blogs.msdn.com/b/pfxteam/archive/2011/01/13/10115163.aspx http://blogs.msdn.com/b/pfxteam/archive/2011/01/13/10115163.... I'd say they are on the right track, but still not there for big time ... just imagine this in hands of millions of average devs under pressure...

EGreg 16y ago

I felt the same way, but thankfully Javascript lets you write really cool things that handle the flow of your code in ways that correspond to the way you think of it. I encourage you to check out http://news.ycombinator.com/item?id=2101789 http://news.ycombinator.com/item?id=2101789 ... with those functions javascript will be pleasant again.

mpk 16y ago

I really like JS and it has excellent closure support, but it's really obvious that heavy async programming like we're doing nowadays with AJAX is not what it was designed for. The 'nested callback' problem is one I've encountered over and over. I've figured out a relatively nice way to work around it. The solution goes something like this. - Take the individual actions that you want and put them in closures. - These closures take 1 argument, which is always a callback function that takes no arguments (this is the convention you structure everything around) - Put these closures in an array, in the order that you want them executed - Pass the array to a function that executes them for you (in Haskell it's called a sequence) I have a really quick and really dirty presentation about it here, http://www.moondust.dds.nl/text/async-presentation/ http://www.moondust.dds.nl/text/async-presentation/ Some code on github up here, https://github.com/michiel/asynchelper-js https://github.com/michiel/asynchelper-js - the sequencer.js function is verbose, if you read the next SO link I've included a three line example of implementing it. And a stackoverflow answer using this code and some explanation here, http://stackoverflow.com/questions/4462605/jquery-ajax-each-callback-next-each-firing-before-ajax-completed/4462955#4462955 http://stackoverflow.com/questions/4462605/jquery-ajax-each-... And another one as a gist on github here, https://gist.github.com/604730 https://gist.github.com/604730 (for rendering a massive table from JSON data). There's a lot more to this sequencer pattern. You can nest sequences, you can have the sequencer run each callback using setTimeout(callback, 0) to release control back to the main loop and get out of 'script is doing too much' popups, etc. Need to make an async call? Make a closure that performs the async call and invoke the callback argument in your async callback. Need to add some random JS code to a sequence? Wrap it in a closure and call the callback when you're done. If the random JS code requires an async call, see the previous paragraph. Error handling, of course, is still awful. You can extend the sequencer convention to add error closures as a second parameter, you can do everything inside the closures in try/catch blocks and finally always call the callback, etc. But that just tends to make a mess of things. So the examples just assume everything is going fine. It's not an ideal solution, but it's native JS and works everywhere. It allows you to structure your code into functional blocks and work on them separately instead of making everything one big nested callback mess.

lvh 16y ago

It would be nice if people stop saying 'async' when they mean 'callbacks'. Twisted's inlineCallbacks, Corotwine, Gevent... are all plenty async, and there's not an explicit callback in sight. (I suppose you could argue that the yield in inlineCallbacks and Monocle delimits a callback. You wouldn't be wrong.)

wladimir 16y ago
Of those I know gevent, which is indeed very nice, as it is 'implicitly' async. It has the performance benefits of being async, but doesn't require you to write callback-based macaroni network code.
- lvh 16y ago
  Corotwine is exactly alike. Personally, I prefer inlineCallbacks. Explicit is better than implicit... Basically I can say, okay, so, here something slow is going to happen, and I'm willing to yield (hey look it even uses an appropriate keyword) control over to the reactor. My main point is still this: okay, yes, sometimes callback code can get ugly (with the apparent implicit assumption plain old "blocking" code is always pretty...), but that has nothing to do with the async backing.
  - wladimir 16y ago
    Plain old blocking code is not always pretty, but it is much more clear what happens, and how errors are handled; all steps are in one place. Also, in my experience, it is usually much shorter.

fleitz 16y ago

Use F# instead let async_copy (in:Stream) (out:Stream) = asnyc { let buf = Array.create 8192 let rec copy = async { return! match in.ReadAsync(0,buf.Length,buf) with | 0 -> () | x -> do! out.WriteAsnyc(0,x,buf); return! copy; } return! copy; } There's an async copy routine, not much more overhead than a sync copy. let sync_copy (in:Stream) (out:Stream) = let buf = Array.create 8192 let rec copy = match in.Read(0,buf.Length,buf) with | 0 -> () | x -> out.Write(0,x,buf); copy copy

deleted 16y ago
[deleted]
icey 16y ago
Slight typo in the first line of your code; asnyc should be async. Line 6, WriteAsnyc should be WriteAsync The intentions were very clear, just posting this in case someone tries to copy & paste it.

swannodette 16y ago

Perhaps what we need is something like CJSR (Common JavaScript Runtime). Languages that compile to JavaScript generate standardized debugging information as JSON (CoffeeScript has been pushing for this). These languages can then use a shared set of in-browser debugging tools (JSDB?). The fact that Safari, Chrome, and FF all support common browser extension architectures as this makes this solution even more compelling. Agressive source transformation becomes less of an issue.

Sephr 16y ago

Using async.js (http://eligrey.com/blog/post/pausing-javascript-with-async-js http://eligrey.com/blog/post/pausing-javascript-with-async-j...), you can do away with callbacks and still have asynchronous code by using the yield statement. The following is an example (taken from my blog post) of asynchronous code using async.js: yield to.request("/feedback", "POST", ( yield to.prompt("What are your impressions of async.js?") )); yield to.inform("Thanks for your feedback!"); // do more stuff here The equivalent code with callbacks is as follows: async.prompt( ["What are your impressions of async.js?"], function (response) { async.request( ["feedback", "POST", response], function () { async.inform( ["Thanks for your feedback!"], function () { // do more stuff here } ); } ); } ); It makes it very easy to handle DOM events, as demonstrated below. var clickEvent = yield myButton.next("click"); Full disclosure: I wrote async.js.

Groxx 16y ago
Since you wrote it: any idea what browser support of Javascript 1.7 is at? Wikipedia[1] currently thinks only Firefox has past 1.5, but I can't find anything else that isn't ancient (in web years), and Wikipedia's tables of things tend to get inaccurate pretty quickly. [1] http://en.wikipedia.org/wiki/JavaScript#Versions http://en.wikipedia.org/wiki/JavaScript#Versions
- Sephr 16y ago
  Only Mozilla supports JavaScript (which is a mix of JavaScript-only features and ECMAScript; it's a little complicated), and everyone else implements ECMAScript. Some JavaScript features are in discussion of being added to ECMAScript, but no current ECMAScript engines (e.g. V8) have implemented them yet, barring the partial destructuring assignment implementation supported by Opera's Futhark ECMAScript engine.
  - sid0 16y ago
    I think yield's coming to ECMAScript Harmony. That should simplify things a lot.

statictype 16y ago

I think the most likely solution would come from a language extension to something like Coffeescript. It could work similar to the async monad in F#: #asnc x = () -> var f = window.menu("file"); var m = f.openmenu(); m.click(); would unwind as: window.menu("file",function(f) { f.open(function(m) { m.click(); }); });

locopati 16y ago

Is there a reason that events don't work in the situation? The async result handler fires an event 'getXFinished' and any function that needs to respond does so. When they are finished, they fire events. And additional functions are called. And so on.

jamwt 16y ago

Others have mentioned coroutines; just to avoid repeating a good conversation that already happened on all this, here's the link: http://news.ycombinator.com/item?id=1549023 http://news.ycombinator.com/item?id=1549023

Kilimanjaro 16y ago

There is a huge gap for a server side implementation of JS and node is not the one the masses are asking for, despite all the hype surrounding it. Just take python, ruby or php, and make JS on the server on par with them. Forget about creating servers, async stuff and coroutines, 90% of moms and pops websites don't need them. There is a huge momentum behind SSJS but it is all being wasted waiting for node to be the mother of all js apps. It will never be.

mmaunder 16y ago

I tend to use Node for specific tasks like running a process that may take from 0 to 20 seconds without having a thread block. Using it for discrete tasks that it's suited for and then doing the bulk of my programming on regular vanilla synchronous server processes has worked for me as a way to manage complexity. Until a standard model of async emerges I think programming large web apps on node will feel a little like hiking with a stone in your shoe.

nikils 16y ago

isn't arrows http://www.cs.umd.edu/projects/PL/arrowlets/example-drag-and-drop.xhtml http://www.cs.umd.edu/projects/PL/arrowlets/example-drag-and... supposed to solve this problem.

moomin 16y ago

I can't help feeling that all of the examples contain too much copy and pasting. This is dealt with by taking a higher order approach to the problem. Which is what async.js is for. doSomething = (c) -> async.waterfall [ (c) -> mainWindow.menu "File", c (file, c) -> file.openMenu c (menu, c) -> menu.item "Open", c (item, c) -> item.click c (c) -> mainWindow.getChild type("Window"), c (dialog, c) -> dialog.doSomething c null ] It's not perfect, but I think it's pretty readable and succinct.

uriel 16y ago

This is one of the reasons I love CSP and languages with CSP-like concurrency like Go (and Erlang, although it is more CSP-inspired than CSP-like). I really can't understand why anyone would want to use anything else for concurrent programming.

sam-mccall 16y ago

Sorry I missed the discussion! Just wanted to add a pointer to what I ended up using: https://github.com/sam-mccall/ship https://github.com/sam-mccall/ship It's lets you create pseudo-threads, you pretend you're calling a sync function, and get a promise back, which you can call methods on (giving more promises) etc. It needs a rewrite to allow objects in different pseudo-threads to interact - i.e. the threads to cross. Currently it's just a queue of actions to perform which doesn't allow this.