19 ms·

An open question (rant) about Node.js

93 points by jobeirne 13y ago

sync 13y ago
Async code is super easy to reason about when using ES6 Generators, which are available now in node v0.11. Here's both a series and a waterfall example: https://github.com/visionmedia/co#example https://github.com/visionmedia/co#example
sheetjs 13y ago
node.js does not impose it for everything. Every command in the fs library that has a natural synchronous interpretation (like reading a file, writing a file, ...) has a sync equivalent (readFileSync, writeFileSync, ...)
- ef4 13y ago
  True, and that helps for cases where you're willing to completely sacrifice concurrency. But there's no reason we couldn't have syntactically synchronous code that still runs concurrently.
- camus2 13y ago
  Try to do that on a production server that has to handle 1000 requests readFileSyncing at the same time ... sync apis are just a conveniance for console applications, they are not meant for a web server. So yes node.js DOES impose async programming for everything or node doesnt scale.
  - babby 13y ago
    Sync is great for initialization. Due to the fact you only start up once, you can be as reckless with your performance hindering practices as much as you want as it will never amount to anything more than a few hundred extra milliseconds of startup time.
    - akbar501 13y ago
      > Sync is great for initialization. This is an interesting point. I've found in practice that the speed of server startup times depends on the scale of operations. For example, at a small scale, a 600 ms vs. a 1.2 sec startup makes little difference. However, at scale such differences can matter. More so when something is going wrong that's causing erratic restarts.
deleted 13y ago
[deleted]
ef4 13y ago
It's worse than that: it's true that in-process asynchronicity is a performance optimization, but it doesn't follow at all that the code needs to read asynchronously. There is no performance reason to make developers hand-code in the continuation-passing style. That should be the job of a compiler or runtime environment. Erlang does it. Stackless Python does it. Go does it. The whole "look how fast Node is due to all my hand-rolled callbacks" is a total fallacy. It's a case of taking perverse pride in an accidental wart of your language. Slowly the node community is maturing and beginning to realize that if they want to build robust, fault-tolerant software they need better primitives than hand-rolled callbacks. Hence the slowly rising mindshare of Promises, which are strictly better, but would be better still if the language itself baked them in implicitly (consider that nearly any value in a Haskell program is represented by a promise under the hood, but you never have to deal with them explicitly).
- jeswin 13y ago
  There is no performance reason to make developers hand-code in the continuation-passing style. node certainly recognizes this. However, there is no need to do anything here since JS is taking care of this with ES6. Even if you don't want to use ES6, there are now multiple ways to compile it down to ES5. Hence the slowly rising mindshare of Promises Promises are declining, and honestly doesn't really solve the problem in most cases. A couple of years from now, you won't see promises on the server. You should see tjholowaychuk/visionmedia's co[1]; that's how JS is going to look like in future. In terms of expressiveness, there isn't much to choose between dynamic languages. While it took some getting used to, programming JS wasn't that different from writing Python. However, JS/node is succeeding because this is the first time ever that we have a language/platform that is truly write-once, run-anywhere. A significant advantage in favor of node. [1] https://github.com/visionmedia/co https://github.com/visionmedia/co
  - ef4 13y ago
    Yes, I'm looking forward to ES6. And I agree that Node's strength is that Javascript is truly the lingua franca of the modern web, which I why I would even bother using it. But the value of running everywhere is also why ES6 is not a panacea. As you say, soon we won't need promises "on the server". But I want my codebase to run correctly everywhere. Promises are a bandaid that I'll need for quite a while yet.
dap 13y ago
There's no sane possibility of a hybrid for Node.js. The concurrency model in JavaScript is built for one thread [that runs JavaScript]. For servers, in such a model, everything has to be async. A 20ms synchronous operation means 20ms of time that you cannot handle new connections or process others that have sent or received data. It's critical to the success of the platform that almost everyone insist on async-only code. Recall that the asynchronous model has been done before by Twisted and EventMachine, but they have exactly this problem: there's tons of Python and Ruby code out there that's synchronous and if you accidentally use it (even by using a dependency of a dependency), the server falls over. In Node, it's very hard to accidentally use it, by design: you have to write an add-on or use one of the few *Sync functions, which are well-known as red flags in a server. In terms of why this model is compelling: it scales well, and it's a relatively simple model of synchronization. The 5-line HTTP server on nodejs.org scales to a very large number of requests per second. Having spent a lot of time in the heavily multi-threaded C world (and loving it, btw), it's much harder to shoot yourself in the foot with JS and this model, and the failure modes are usually less severe. (Many C programs use this model as well, including haproxy and much of the kernel.) It's not all win. Blocked operations are harder to debug. But you can work around things like that, and overall it's a nice environment. And FWIW, once you're familiar with it, the waterfall pattern quickly becomes as second-nature as writing synchronous code. (More complex ones are still tricky, but the basics do become second-nature.) [edit: I meant that there's no sane possibility of a hybrid between synchronous and asynchronous semantics. That doesn't mean you couldn't use a language feature to make async code look synchronous. That's a much more subjective discussion, but I prefer explicitness over compiler magic.]
- ef4 13y ago
  This is precisely the fallacy I'm talking about in my top-level comment. You're conflating asynchronous implementation (which I agree is good), with structuring the code in continuation-passing style (which is bad, for a lot of reasons). It's all just plumbing, and people demonstrably do it badly if we judge by how well the typical node module deals with unexpected failures. The official node policy on exceptions is that you should let the process die. Not because it's impossible to write exception-safe code, but because almost nobody does it, because it's too hard to do when you're plumbing it all by hand.
robgering 13y ago
I understand that Node.js is really popular here and that my opinion may be an unpopular one. Maybe this will change, but I cannot find a use case for Node where I wouldn't choose something else. If I want to prototype (performance is not a constraint), I'd use Ruby (or Python) before Node. If I need concurrency or if performance is a constraint, I'd choose Go (or Clojure, or Scala, or Erlang) before Node. I understand the argument about using JavaScript throughout the stack, but I see this as specious at best. Front-end and back-end development require different domain knowledge, not just language proficiency. Perhaps using Node opens a project to a larger community of developers. But under this argument, using C or Java also seems a good choice.
- ef4 13y ago
  So I'm a node critic, and I think both the technology and the community suffer from technical immaturity. But I actually use it and depend on it, for precisely the shared-codebase reason. And it helps immensely for my use case. I actually started out with Ruby on server side, but spent increasing amounts of time duplicating the same code in both languages. Switching it all to Javascript was a demonstrable win. When people argue it's a specious benefit, it just means they haven't tried to build a truly ambitious client-side application. If performance, consistency, and security matter you're going to have a lot of code that needs to run at the client for responsiveness and at the server for safety & security. And if you care about offline-capable applications, the problem because even more acute: your clients are now a fully distributed system with their own state, and it's easier to write all the synchronization and recovery logic once, and run it everywhere on both clients and servers.
  - robgering 13y ago
    This is great feedback. Thank you.
- bryanlarsen 13y ago
  Here's an example: http://clara.io http://clara.io Obviously this a fairly heavy client application. But imports, exports and renders are server side operations. They manipulate the scene graph using the same code that the client side uses to manipulate the scene graph. Live render is a great example. When you start a live render, we start a client on the server, but instead of translating the scene graph into WebGL, it translates the scene graph into V-Ray. Any changes you make to the scene are sent to the server-side client using the same mechanism used to synchronize changes between 2 users editing the same scene. If we used a different language in the front end and the back end, we would have ended up with a lot of duplication of effort.
erjiang 13y ago
Please stop asking questions like this about Node.js. The superfluous obtuse complexity that node.js forces is what's keeping ninja programmers like us employed. If we don't keep switching to strange new things every few years, our wages will stagnate as there's less separating our rock-star tech from lower-salaried uncool "enterprise" software. Being able to manually manage continuations rather than using a language that does it for us is a high barrier that keeps us in demand, the same way that having to manage memory manually keeps people from becoming C programmers. Losing this edge is not something we should encourage. Just imagine - if all of these "learn to code" programs actually work, the market will be flooded with new programmers. Thankfully, if we make sure that they only learn easy things like Python, we can continue to convince the world that things like Node.js and MongoDB are the future. While the rest of the world is still catching up on basics like Python and Postgres, we'll be one step ahead of them. People questioning Node.js like this more and more are a sign that we need to find the next hotness and move there if/when node.js sinks. If we can convince people to build more things in the next hotness, then we can get in early before the masses pick up on it, ensuring that we can maintain our spendy lifestyles and free lunches in SF. (/s)
- ebiester 13y ago
  Node is passing the Peak of Inflated Expectations and into the Trough of disillusionment. Rails is firmly in the Plateau of Productivity. There are some genuinely interesting ideas coming out of node, and I think it's unfair to just call it obtuse complexity. I think that node.js has been a call to return to small, focused modules that will percolate back up. An attempt to keep things small and focused is nothing new -- Unix started the trend in the 70s. Everything old is new again. I think they're forging some new patterns that take advantages of small, composible blocks through NPM in a way that CPAN and the like were never quite able to exploit. I think node has an exciting opportunity, coupled with single page apps and browserify, to be a common front end to disparate web services. And, to me, that's exciting.
sergiotapia 13y ago
https://www.youtube.com/watch?v=bzkRVzciAZg https://www.youtube.com/watch?v=bzkRVzciAZg
meryn 13y ago
I posted this as a comment on the gist: It's not about performance, it's about <del>parallelism</del> concurrency. If you don't care for Node's ability to handle multiple requests <del>in parallel</del> concurrently, with a single process with only one thread, than you might not have a need for Node at all. Why not use Ruby or Python? If you really want to force Node into being a dumb scripting engine, you could use the various sync functions that Node.js provides. Problem is, they block the entire thread. But if you want to, you can. And I believe with some clever hacks, you can use the blocking nature of (for example) the fs sync functions to make other calls (including http requests) synchronous as well. But unless you use Node to build something that's not a server (people build command line tools with it as well), I don't think you want to. By using synchronous functions, everything Node is built for will go down the drain. Everything done will be done in sequence, and during the time the sequence of operations has not finished, the Node runtime will be unavailable to serve any requests. I suggest you look into promises as a somewhat saner way to deal with asynchronicity than callbacks, although you can pretty far with just the Async library. The future might be in Generators, a new feature of EcmaScript 6. This is available in V8. You can use this in Node 0.11 (unstable) if you use the --harmony flag. See here for a nice writeup: http://blogs.atlassian.com/2013/11/harmony-generators-and-promises-for-node-js-async-fun-and-profit/ http://blogs.atlassian.com/2013/11/harmony-generators-and-pr...
- bascule 13y ago
  You're confusing concurrency and parallelism. Node is a single-threaded event loop. It can only do one thing at a time. Node can perform many operations concurrently, but they're not running in parallel. If any one operation blocks, the whole event loop stalls. Parallel execution requires multiple threads running on multiple CPU cores. Node requires multiple VMs to do this.
  - meryn 13y ago
    Thanks for explaining difference between concurrency and parallelism so clearly! I'll remember this. I of course realize that Node doesn't truly handle things in parallel. It's a single thread, after all.
clux 13y ago
Asynchronicity isn't an optimization, it's a simplification. Blocking code and threads are hard to reason about, but callbacks are simple. You still have to deal with errors in a synchronous world, and an evented system allows you to deal with branching event possibilities sanely.
- ef4 13y ago
  Yes, pre-emptive blocking threads are hard to reason about. But cooperative blocking threads are just as easy to reason about as node's callbacks, while being significantly easier to structure and make fault-tolerant.
pessimizer 13y ago
I can't speak to node (don't use it), but in Erlang, I find async far easier to reason about than sync for the simple reason that I don't have to plan for what will happen if some operation over the network breaks during a request. In my head, there's no difference between async and sync. A synchronous call is just an async message + and async acknowledgement of receipt. If I write it that way, there's a lot of failure modes that I don't have to think about. Also, in my head, concurrent code looks like a sparse group of people spread over a large, grassy, hilly plot of land shouting at each other from differing heights and many yards distance over the whine of shifting winds. I find it easier to reason about the shouts alone, instead of shouts + their responses, because a response is nothing but another shout.
notastartup 13y ago
The creator of Node.js said it best himself, that his afraid people associate Node.js with scalability out of the box due to the single threaded asynchronous nature, that there exists something magical about it that would suddenly make your app scalable because it's riddled with async code. I now use Flask and Python instead of Node.js and it's a total pain reliever. Not having to deal with Javascript is a win on the server side (JS on client side is enough) and the python stack is good enough to handle every bit of Node.js Truth is, Node.js is in a bubble. I find it that even people who have never programmed before insisting that their new website be written in Node.js.... It reminded me of the dot com bubble, when your grandma would explain to you her rationale for buying dot com stocks (they had no idea).
WhiteNoiz3 13y ago
My argument would be that if you go 'sync first, async if you feel like it', everyone will just do sync, because it is easier conceptually (or atleast more familiar to a lot of people).
morganherlocker 13y ago
> If asynchronous code is harder to reason about, why would we elect to live in a world where it is the default? Because if you do not make it the default, then most libraries you use will make the choice for you by going sync.
EGreg 13y ago
Um, am I the only one who considers Node.js a beautiful tool suited for developing servers? Why async by default? Because defaults should encourage best practices. It can handle C10K problem out of the box. With libraries like Q, the code can actually be rather easy to follow and maintain. One of the biggest reasons why a multi-user server should be coded in async-by-default style is as follows: often, you want to obtain objects and not care about how they are obtained. In procedural languages, this isn't possible. Consider the following challenges: 1) Query sent to 10 different shards, wait until they all return, combine the results in the app, and then use them. In a synchronous language, you HAVE TO go to one shard, then another, then another, and the time is 10x longer. In an async style, it's only as long as the longest query! 2) Some of the objects may have already been cached, whereas others need to be obtained. In addition, the ones that need to be obtained shouldn't be requested more than once at the same time, but callbacks should be placed on a waiting list. 3) Various parts of your code (e.g. from different requests) might want to request the same object. Not only that, but you might want to batch your requests. That is to say, wait 10 milliseconds and see if any other (unrelated) parts of your code also want to grab some objects from the same DB table. Suddenly you are issuing a lot less queries because you can build middleware like this easily in JS. Not so in PHP for example - I would know. A lot is wasted there. 4) Let's consider #1 some more. The queries may go to other HTTP endpoints - do you really want to wait that long for I/O? Furthermore, even if they are going to local MySQL shards, your site becomes slower with an extra factor of O(log n) with the number of users. Kind of like when you use a relational database instead of a graph database for one-degree-away lookups. It could have been O(1) but instead you introduced another factor of O(log n). Not terrible, but certainly gives you a 5-20x cost down the line.
- bascule 13y ago
  > In a synchronous language, you HAVE TO go to one shard, then another, then another, and the time is 10x longer. Have you ever heard of threads?
  - dharbin 13y ago
    Threads are more difficult than writing event-loop based code IMO
pkinsky 13y ago
Asynchronicity is great, but javascript doesn't provide the tools to do it properly. > (...) try writing a function call that requires information from two separate HTTP API responses; I basically need to draw a diagram of what happens with async.waterfall for a task that, given synchronicity, would've been solved with a trivial three-liner. It's a lot easier to handle asynchronicity in strongly-typed languages with monadic for comprehensions. In Scala, for example: val res: Future[Foobar] = for { a <- makeHttpRequestA() b <- makeHttpRequestBFromA(a) } yield new Foobar(a, b) You can then register callback functions which operate on either a Success[Foobar] or a Failure[Throwable]. Synchronous code isn't simpler, it just hides complexity like caltrops in tall grass. (slightly modified version of my post on Github)
- eldude 13y ago
  co(function *(){ let promiseA = makeHttpRequestA() let promiseB = makeHttpRequestBFromA(yield promiseA) var res = new Foobar(yield promiseA, yield promiseB); })();
  - densh 13y ago
    with scala.async [1]: val res = async { val a = await(makeHttpRequestA()) val b = await(makeHttpRequestBFromA(a)) new Foobar(a, b) } [1] https://github.com/scala/async https://github.com/scala/async
anuraj 13y ago
In browser content may need to be asynchronous, but HTTP is request/response - nothing asynchronous. Half the web (or more) is not about feeding the browser. Use asynchronicity where needed.
bitcrusher 13y ago
I have a hard time 'listening' to these rants about how hard it is to reason about asynchronous code. It's NOT hard, it's DIFFERENT. This isn't some new Node-Sauce. The idea of an event-loop asychronous programming domain has been around forever in various forms, first in games and desktop GUI programming and the way one reasons about it hasn't changed, just because it has been applied to a new class of problems. I suspect that there are two things going on here: 1. A LOT of people dislike Javascript. As a result, they're starting with that negative point of view and then finding reasons to support it. This is not unlike people who argue about Python white-space. They're really saying "I don't like Python." The white-space issue is incidental. 2. Lack of experience / exposure I suspect that folks who have trouble with this, frankly just aren't experienced enough. I'm not saying they aren't skilled, just not experienced in this realm. Once upon a time, these same sorts of 'arguments' were used against Ruby ( ala the Rails explosion ). Nothing much has changed about Ruby, other than it has managed to be around long enough to influence a new generation of programmers.
- jerf 13y ago
  Yes, it is harder. You are attempting to reason about code in the absence of a useful stack trace. You're essentially taking your structured programming language and throwing it away and returning to the spaghetti-code era of code flow (this time with islands of structure). Both automated and human analysis is legitimately harder under these circumstances. "2. Lack of experience / exposure" Quite the contrary. I've had abundant experiences with both event-based models (which hardly started with Node!) in several paradigms (GUI, server, network) and models like Erlang, Go, and Haskell. It is no contest. The latter is sane. The former is not. If you have the opportunity to choose freely, and you choose the event-based model, you have chosen poorly. Believe me, it is not the critics of Node who are the inexperienced ones. All the evidence I've seen points strongly in the opposite direction. Node advocates could start by getting their understanding of how their competition works out of the mid-1990s. Ironically, in the end Node will eventually work their way around to working like these models. The signs are all there, the parallel evolution is clear, and apparently the process has already begun with Meteor. I wonder if the community will ever acknowledge how crazy they've been as they careen from one "solution" to the next at breakneck speed, all the while crowing about how much better their stuff is, even though what they consider "their stuff" can't seem to stay stable for 3 months?
woola 13y ago
Having written nodejs for about an year I would never choose it again if I have given the choice. * No Error handling. The process has to be restarted if something goes wrong. * No thread local storage equivalent. As a result you can't do simple thing like differentiate log originated from different http requests. * Callback makes the code unreadable * Javascript has no real method or in other words 'this' is not attached to a function and is determined by way the function is called. So you have to wrap most of the callbacks with _.bind() * Mixing callback, promise and event emitter api introduce lot of boilerplate code. * No stacktrace. Debugging error is PITA * No preemptive scheduling. So you have to be extra careful that you never spend too much CPU time. * No easy way to handle back pressure.
- MrBuddyCasino 13y ago
  These are exactly the reasons I have stayed away from it until now. I don't understand this one though, what does it mean? * No easy way to handle back pressure.
  - Detrus 13y ago
    Latest developments of Node Streams begin to address back pressure.
  - nostrademons 13y ago
    I did some Googling. "back pressure" in the Node context is basically what most people refer to as "flow control" or "overload behavior". Node programs work by enqueueing a bunch of work items for the event loop to handle, and then going on and doing a bunch more stuff until all the work items have been handled and their callbacks have been called. In the process, those callbacks may themselves enqueue further work items. The problem is that all these callbacks themselves take up memory and hold onto references in the V8 heap. And so if callbacks are being enqueued faster than they can be processed, the heap will get more full. V8's GC takes time proportional to the amount of live data, and so GCs will become more frequent and take longer. This further reduces the amount of available CPU time to process callbacks, which increases the amount of live data, until the system thrashes to a halt. Haskell programmers (who use a similar scheduling mechanism for lazy evaluation) call this a "space leak", and it's a big problem in Haskell as well. The solution is for the runtime to let the calling code know that it's overloaded, and then either temporarily block the calling code from creating more callbacks until existing ones have run, or let the calling code gracefully degrade and choose to handle things in a simpler fashion.
christiansmith 13y ago
The gist reads like someone looking for an excuse to give up and move on. Every time I run across someone bitching about Node's programming model I'm reminded of "The Blub Paradox". http://paulgraham.com/avg.html http://paulgraham.com/avg.html I have to admit that learning to think in callbacks was initially an exercise in frustration. But it wasn't that hard. Most of the pyramid-type issues can be resolved by applying principles that are also valid in synchronous environments (like modularity). If modules and named functions aren't enough, you can subclass EventEmitter to further decouple. For more complex cases there's https://github.com/caolan/async https://github.com/caolan/async, which makes many nontrivial async problems relatively easy. Why is it so hard for so many people to appreciate Node for its merits? The code isn't going to execute in a perfectly linear way, so why should it have to be written as if it were?
mattgreenrocks 13y ago
My big beef with Node.js: it could have avoided the sync/async split entirely by using a synchronous facade atop an asynchronous runtime...but it didn't. Why would you create an entire server platform and not use the chance to simplify the most common use case: fetching data from a database? I see as a great opportunity completely squandered so that it would cater to the masses: "you don't have to learn anything new!" Sometimes I think web tech likes being as shoddy and thrown-together as possible.
Cless 13y ago
You can have an async implementation and synchronous code. http://meteor.com http://meteor.com does that part quite nicely. The compiler or runtime should be handling this kind of cruft for us in most cases. Just because using Brainfuck might result in theoretically faster code than Ruby doesn't mean we should be writing our server in Brainfuck.
jwarkentin 13y ago
It seems that OP thinks the asynchronous nature of JavaScript was meant as a performance optimization. Let's go back to its origins. It was originally a scripting language for the browser. If it were to be synchronous and/or allow blocking things, such as `sleep()` or whatever else, it would have created an unusable web experience. Web pages would be constantly locking up. It's only recently that it's moved to the server where the asynchronicity on I/O isn't so mandatory. I personally still think it's a good thing, but that could definitely be debated on the server side. Unfortunately, if you want to have the advantage of writing one code base that can run on the browser or server, then it must work the same in both places.
neya 13y ago
I was reading somewhere (maybe Hackernews itself?) that synchronous programming and/or blocking are actually features and not disadvantages. After working on a huge Node.JS project, I regret taking on anyway (I had to do it for the moolah, aka money). I am now convinced that migrating from Rails to Express (A node.Js framework) was a total mistake. Please note, I do not mean to advertise rails in particular, infact, you can replace rails with anything else thats fits the analogy - perhaps maybe a python framework. However, this mistake has taught me something(s) very valuable - 1) Don't trade in a tried and tested framework for something like Node.js, that will open up a different level of problems altogether. 2) No matter what language you use, you WILL face issues at scale. Be it Node.js/Scalatra/Rails. You might as well face it with a tried and tested framework. 3) Don't re-write your own framework in something like Go or Haskell or C or C++ or etc. thinking it will be scalable, etc. You will - a) Waste massive amounts of time re-inventing the wheel, b) Open yourself to possibly dangerous security vulnerabilities (Eg: Handling encrypted cookies), c) Waste your time on things that don't really matter instead of on something that actually matters - The product itself. 4) Anything that you can scale by throwing more money at it is for the win[1] Why am I talking about scalability, when the discussion is about Async. vs Sync. programming? Because of the reasons many people cite using Node.Js and the like. It's just not worth it. Focus on the product first and the architecture later. This comes from someone who wasted a year of his life doing everything above. [1]http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html http://highscalability.com/blog/2013/4/15/scaling-pinterest-...
- mclenithan 13y ago
  Sorry you had a bad experience, bud. I am on a team of six, all pretty experienced with Node, and we don't have those issues. We've been going production on it for about a year and a half now. People cried a little in the beginning because it was different but now that we have embraced (and love) callbacks, we rip through development pretty quick. Things aren't always done for you but it's getting pretty rare at this point.
mistercow 13y ago
>but I'd be very curious to hear a defense of "async-first" thinking for problems that are typically solved on the server-side Because if your web server blocks every time it accesses a file, or talks to a database, or communicates over the network, then it won't take many requests at the same time before it's completely bogged down.
angersock 13y ago
So, here's how I think about it: Any program, of any type, has a synchronous story to it: data came in, then this happened to it, then this happened to it, then it left. If you follow the program execution, this is how it works. It's very easy to reason about, right? And it's very easy to program this way. One of the biggest breakthroughs in computer engineering was the idea of multiple concurrent processes (and nevermind that behind the scenes they were still running on a single processor!), and then later still true concurrent multi-processor architectures. We developed operating systems and compilers and runtimes specifically to let us take advantage of that without the having to break from easy-to-reason-about synchronous code. ~ Node said, hell with it, everything is async all the time. This is really annoying at first, because there are many many times where sync is the best way to reason about something. That said, there is some beauty in such a hard-line approach: if you are truly async, handling an error is the same as handling a task which takes too long. Once you've paid that price properly, you're fine. (This is also what makes Erlang kind of cool.) The big frustration is that there really ought to be a way of writing synchronous-looking Javascript code (ES5) which--behind the scenes!--is multiplexed and made to run concurrently and without blocking. The mechanisms are all there for programmers to deal with exceptional failures (exceptions, anyone?), and so are synchronization points (functions having multiple arguments should probably join on the evaluation of those arguments before being invoked--that's something the interpreter should be doing already). I guess what I'm saying is...why don't JS interpreters do lazy evaluation and late binding?
bitwize 13y ago
Here's a fun fact that every Amiga programmer seems to understand that Unix programmers just never really got: actual computing hardware is inherently asynchronous and interrupt-driven. The whole Amiga system is structured around using the CPU to program the custom chips and initiate DMA transfers, and installing interrupt handlers that perform the next phase of the program once the custom chips finished their work. It's how shit got done on the Amiga. By contrast, the lack of good support in early Unix for asynchronous I/O is said to be one of the reasons why X11 is such an ugly pile of hacks. Even on Unix, if you wrote a server procedurally what you'd end up doing is having main() start a select loop that... dispatches incoming events to callbacks. All Node does really is to abstract away the select loop -- one less thing for the developer to worry about. The Node way is simpler than the conventional C way. So no, there's nothing inherently superior about linear, procedural code, especially when such code is not at all how a computer system that must interact with the outside world works. Whether asynchronous, callback-driven code is confusing or not depends on what background you come from. Put an old-school scener in front of Node and he might form an entirely different opinion about it from OP.
- nostrademons 13y ago
  I think the question the article is asking is "Why did did Node stop at abstracting the event loop away, when it could've also abstracted the execution context?" There are languages that do this - Go, Erlang, Haskell, and Python 3.4 all provide a sequential process/coroutine abstraction on top of an implementation that's fundamentally asynchronous. The question the author has isn't about implementation, it's about programming model. Programming languages & frameworks are ultimately made for humans, and most humans find it easier to reason about "Do this thing, then this thing, then these other things, until you're finished" than "I'm doing all these things at once!" We have computers to organize all the events, stack frames, wait queues, etc. that take us from the machine to the editor window.
  - bitwize 13y ago
    What I'm saying is that a programming model that's confusing to you is perfectly natural to someone else who comes from an environment where that model is all you have. People have used interrupt/event handlers and callbacks literally since the dawn of computing. It's nothing new. It just has to be learned, the way programming itself has to be learned.
deividy 13y ago
Am I the only dev in the world that loves async?
schrijver 13y ago
As a designer I have been trying to dig this stuff. Server side JS seems like a huge win for all kinds of small websites and interesting ui experiments, because one can stay in one language and avoid code duplication. Yet in trying to set up some experiments that use a server-side git integration, I bumped into the kind of code described in this thread, and I find it super difficult to understand the async way of doing things. Here’s an example from the homepage of nodegit, which seem to be among the best maintained libraries that wrap libgit2. Thus I imagine it represents idiomatic node.js What the code does, is console.log details of all the repositories commits, git log style. // Load in the module. var git = require('nodegit'), async = require('async'); // Open the repository in the current directory. git.repo('.git', function(error, repository) { if (error) throw error; // Use the master branch (a branch is the HEAD commit) repository.branch('master', function(error, branch) { if (error) throw error; // History returns an event, and begins walking the history var history = branch.history(); // History emits 'commit' event for each commit in the branch's history history.on('commit', function(error, commit) { // Print out `git log` emulation. async.series([ function(callback) { commit.sha(callback); }, function(callback) { commit.date(callback); }, function(callback) { commit.author(function(error, author) { author.name(callback); }); }, function(callback) { commit.author(function(error, author) { author.email(callback); }); }, function(callback) { commit.message(callback); } ], function printCommit(error, results) { if (error) throw error; console.log('SHA ' + results[0]); console.log(results[1] * 1000); console.log(results[2] + ' <' + results[3] + '>'); console.log(results[4]); }); }); }); }); It looks so alien to me! Instead of the tree-walking and looping constructs I know from synchronous languages, we deal with a history that returns events as it walks the tree (btw, why is it that we can launch the history before we attach the event?). Then we have to actually import the ‘async’ module, because, uhm, we want to specify an order by which the different parts of the message gets logged. How then the actual callbacks work, I’d be happy if someone can explain that to me—why do I pass a function to an attribute of the object? Why don‘t I just return the value? Because the code is not written in a lineair way, I have to keep looking back and forth between callbacks to get a mental picture of what it does. For the record, what this translates to as synchronous Python: from pygit2 import Repository, GIT_SORT_TOPOLOGICAL repo = Repository('.git') for commit in repo.walk(repo.head.target, GIT_SORT_TOPOLOGICAL): print 'SHA %s' % commit.hex print commit.commit_time * 1000 print '%s <%s>' % (commit.author.name, commit.author.email) print commit.message I imagine that, as some of the commenters note, I will be able to start reading the async style if I get enough exposure to it. I wonder though, if it does not make the barrier to entry for server-side JavaScript too high for non-full-time programmers. In that sense it is interesting that the Meteor team chose to use the package Fibers, that allows one to write node.js in a procedural style. I imagine something like that is necessary for server side JS to become really mainstream in web design…

93 points by jobeirne 13y ago

sync 13y ago

Async code is super easy to reason about when using ES6 Generators, which are available now in node v0.11. Here's both a series and a waterfall example: https://github.com/visionmedia/co#example https://github.com/visionmedia/co#example

sheetjs 13y ago

node.js does not impose it for everything. Every command in the fs library that has a natural synchronous interpretation (like reading a file, writing a file, ...) has a sync equivalent (readFileSync, writeFileSync, ...)

ef4 13y ago
True, and that helps for cases where you're willing to completely sacrifice concurrency. But there's no reason we couldn't have syntactically synchronous code that still runs concurrently.
camus2 13y ago
Try to do that on a production server that has to handle 1000 requests readFileSyncing at the same time ... sync apis are just a conveniance for console applications, they are not meant for a web server. So yes node.js DOES impose async programming for everything or node doesnt scale.
- babby 13y ago
  Sync is great for initialization. Due to the fact you only start up once, you can be as reckless with your performance hindering practices as much as you want as it will never amount to anything more than a few hundred extra milliseconds of startup time.
  - akbar501 13y ago
    > Sync is great for initialization. This is an interesting point. I've found in practice that the speed of server startup times depends on the scale of operations. For example, at a small scale, a 600 ms vs. a 1.2 sec startup makes little difference. However, at scale such differences can matter. More so when something is going wrong that's causing erratic restarts.

deleted 13y ago

[deleted]

ef4 13y ago

It's worse than that: it's true that in-process asynchronicity is a performance optimization, but it doesn't follow at all that the code needs to read asynchronously. There is no performance reason to make developers hand-code in the continuation-passing style. That should be the job of a compiler or runtime environment. Erlang does it. Stackless Python does it. Go does it. The whole "look how fast Node is due to all my hand-rolled callbacks" is a total fallacy. It's a case of taking perverse pride in an accidental wart of your language. Slowly the node community is maturing and beginning to realize that if they want to build robust, fault-tolerant software they need better primitives than hand-rolled callbacks. Hence the slowly rising mindshare of Promises, which are strictly better, but would be better still if the language itself baked them in implicitly (consider that nearly any value in a Haskell program is represented by a promise under the hood, but you never have to deal with them explicitly).

jeswin 13y ago
There is no performance reason to make developers hand-code in the continuation-passing style. node certainly recognizes this. However, there is no need to do anything here since JS is taking care of this with ES6. Even if you don't want to use ES6, there are now multiple ways to compile it down to ES5. Hence the slowly rising mindshare of Promises Promises are declining, and honestly doesn't really solve the problem in most cases. A couple of years from now, you won't see promises on the server. You should see tjholowaychuk/visionmedia's co[1]; that's how JS is going to look like in future. In terms of expressiveness, there isn't much to choose between dynamic languages. While it took some getting used to, programming JS wasn't that different from writing Python. However, JS/node is succeeding because this is the first time ever that we have a language/platform that is truly write-once, run-anywhere. A significant advantage in favor of node. [1] https://github.com/visionmedia/co https://github.com/visionmedia/co
- ef4 13y ago
  Yes, I'm looking forward to ES6. And I agree that Node's strength is that Javascript is truly the lingua franca of the modern web, which I why I would even bother using it. But the value of running everywhere is also why ES6 is not a panacea. As you say, soon we won't need promises "on the server". But I want my codebase to run correctly everywhere. Promises are a bandaid that I'll need for quite a while yet.

dap 13y ago

There's no sane possibility of a hybrid for Node.js. The concurrency model in JavaScript is built for one thread [that runs JavaScript]. For servers, in such a model, everything has to be async. A 20ms synchronous operation means 20ms of time that you cannot handle new connections or process others that have sent or received data. It's critical to the success of the platform that almost everyone insist on async-only code. Recall that the asynchronous model has been done before by Twisted and EventMachine, but they have exactly this problem: there's tons of Python and Ruby code out there that's synchronous and if you accidentally use it (even by using a dependency of a dependency), the server falls over. In Node, it's very hard to accidentally use it, by design: you have to write an add-on or use one of the few *Sync functions, which are well-known as red flags in a server. In terms of why this model is compelling: it scales well, and it's a relatively simple model of synchronization. The 5-line HTTP server on nodejs.org scales to a very large number of requests per second. Having spent a lot of time in the heavily multi-threaded C world (and loving it, btw), it's much harder to shoot yourself in the foot with JS and this model, and the failure modes are usually less severe. (Many C programs use this model as well, including haproxy and much of the kernel.) It's not all win. Blocked operations are harder to debug. But you can work around things like that, and overall it's a nice environment. And FWIW, once you're familiar with it, the waterfall pattern quickly becomes as second-nature as writing synchronous code. (More complex ones are still tricky, but the basics do become second-nature.) [edit: I meant that there's no sane possibility of a hybrid between synchronous and asynchronous semantics. That doesn't mean you couldn't use a language feature to make async code look synchronous. That's a much more subjective discussion, but I prefer explicitness over compiler magic.]

ef4 13y ago
This is precisely the fallacy I'm talking about in my top-level comment. You're conflating asynchronous implementation (which I agree is good), with structuring the code in continuation-passing style (which is bad, for a lot of reasons). It's all just plumbing, and people demonstrably do it badly if we judge by how well the typical node module deals with unexpected failures. The official node policy on exceptions is that you should let the process die. Not because it's impossible to write exception-safe code, but because almost nobody does it, because it's too hard to do when you're plumbing it all by hand.

robgering 13y ago

I understand that Node.js is really popular here and that my opinion may be an unpopular one. Maybe this will change, but I cannot find a use case for Node where I wouldn't choose something else. If I want to prototype (performance is not a constraint), I'd use Ruby (or Python) before Node. If I need concurrency or if performance is a constraint, I'd choose Go (or Clojure, or Scala, or Erlang) before Node. I understand the argument about using JavaScript throughout the stack, but I see this as specious at best. Front-end and back-end development require different domain knowledge, not just language proficiency. Perhaps using Node opens a project to a larger community of developers. But under this argument, using C or Java also seems a good choice.

ef4 13y ago
So I'm a node critic, and I think both the technology and the community suffer from technical immaturity. But I actually use it and depend on it, for precisely the shared-codebase reason. And it helps immensely for my use case. I actually started out with Ruby on server side, but spent increasing amounts of time duplicating the same code in both languages. Switching it all to Javascript was a demonstrable win. When people argue it's a specious benefit, it just means they haven't tried to build a truly ambitious client-side application. If performance, consistency, and security matter you're going to have a lot of code that needs to run at the client for responsiveness and at the server for safety & security. And if you care about offline-capable applications, the problem because even more acute: your clients are now a fully distributed system with their own state, and it's easier to write all the synchronization and recovery logic once, and run it everywhere on both clients and servers.
- robgering 13y ago
  This is great feedback. Thank you.
bryanlarsen 13y ago
Here's an example: http://clara.io http://clara.io Obviously this a fairly heavy client application. But imports, exports and renders are server side operations. They manipulate the scene graph using the same code that the client side uses to manipulate the scene graph. Live render is a great example. When you start a live render, we start a client on the server, but instead of translating the scene graph into WebGL, it translates the scene graph into V-Ray. Any changes you make to the scene are sent to the server-side client using the same mechanism used to synchronize changes between 2 users editing the same scene. If we used a different language in the front end and the back end, we would have ended up with a lot of duplication of effort.

erjiang 13y ago

Please stop asking questions like this about Node.js. The superfluous obtuse complexity that node.js forces is what's keeping ninja programmers like us employed. If we don't keep switching to strange new things every few years, our wages will stagnate as there's less separating our rock-star tech from lower-salaried uncool "enterprise" software. Being able to manually manage continuations rather than using a language that does it for us is a high barrier that keeps us in demand, the same way that having to manage memory manually keeps people from becoming C programmers. Losing this edge is not something we should encourage. Just imagine - if all of these "learn to code" programs actually work, the market will be flooded with new programmers. Thankfully, if we make sure that they only learn easy things like Python, we can continue to convince the world that things like Node.js and MongoDB are the future. While the rest of the world is still catching up on basics like Python and Postgres, we'll be one step ahead of them. People questioning Node.js like this more and more are a sign that we need to find the next hotness and move there if/when node.js sinks. If we can convince people to build more things in the next hotness, then we can get in early before the masses pick up on it, ensuring that we can maintain our spendy lifestyles and free lunches in SF. (/s)

ebiester 13y ago
Node is passing the Peak of Inflated Expectations and into the Trough of disillusionment. Rails is firmly in the Plateau of Productivity. There are some genuinely interesting ideas coming out of node, and I think it's unfair to just call it obtuse complexity. I think that node.js has been a call to return to small, focused modules that will percolate back up. An attempt to keep things small and focused is nothing new -- Unix started the trend in the 70s. Everything old is new again. I think they're forging some new patterns that take advantages of small, composible blocks through NPM in a way that CPAN and the like were never quite able to exploit. I think node has an exciting opportunity, coupled with single page apps and browserify, to be a common front end to disparate web services. And, to me, that's exciting.

sergiotapia 13y ago

https://www.youtube.com/watch?v=bzkRVzciAZg https://www.youtube.com/watch?v=bzkRVzciAZg

meryn 13y ago

I posted this as a comment on the gist: It's not about performance, it's about <del>parallelism</del> concurrency. If you don't care for Node's ability to handle multiple requests <del>in parallel</del> concurrently, with a single process with only one thread, than you might not have a need for Node at all. Why not use Ruby or Python? If you really want to force Node into being a dumb scripting engine, you could use the various sync functions that Node.js provides. Problem is, they block the entire thread. But if you want to, you can. And I believe with some clever hacks, you can use the blocking nature of (for example) the fs sync functions to make other calls (including http requests) synchronous as well. But unless you use Node to build something that's not a server (people build command line tools with it as well), I don't think you want to. By using synchronous functions, everything Node is built for will go down the drain. Everything done will be done in sequence, and during the time the sequence of operations has not finished, the Node runtime will be unavailable to serve any requests. I suggest you look into promises as a somewhat saner way to deal with asynchronicity than callbacks, although you can pretty far with just the Async library. The future might be in Generators, a new feature of EcmaScript 6. This is available in V8. You can use this in Node 0.11 (unstable) if you use the --harmony flag. See here for a nice writeup: http://blogs.atlassian.com/2013/11/harmony-generators-and-promises-for-node-js-async-fun-and-profit/ http://blogs.atlassian.com/2013/11/harmony-generators-and-pr...

bascule 13y ago
You're confusing concurrency and parallelism. Node is a single-threaded event loop. It can only do one thing at a time. Node can perform many operations concurrently, but they're not running in parallel. If any one operation blocks, the whole event loop stalls. Parallel execution requires multiple threads running on multiple CPU cores. Node requires multiple VMs to do this.
- meryn 13y ago
  Thanks for explaining difference between concurrency and parallelism so clearly! I'll remember this. I of course realize that Node doesn't truly handle things in parallel. It's a single thread, after all.

clux 13y ago

Asynchronicity isn't an optimization, it's a simplification. Blocking code and threads are hard to reason about, but callbacks are simple. You still have to deal with errors in a synchronous world, and an evented system allows you to deal with branching event possibilities sanely.

ef4 13y ago
Yes, pre-emptive blocking threads are hard to reason about. But cooperative blocking threads are just as easy to reason about as node's callbacks, while being significantly easier to structure and make fault-tolerant.

pessimizer 13y ago

I can't speak to node (don't use it), but in Erlang, I find async far easier to reason about than sync for the simple reason that I don't have to plan for what will happen if some operation over the network breaks during a request. In my head, there's no difference between async and sync. A synchronous call is just an async message + and async acknowledgement of receipt. If I write it that way, there's a lot of failure modes that I don't have to think about. Also, in my head, concurrent code looks like a sparse group of people spread over a large, grassy, hilly plot of land shouting at each other from differing heights and many yards distance over the whine of shifting winds. I find it easier to reason about the shouts alone, instead of shouts + their responses, because a response is nothing but another shout.

notastartup 13y ago

The creator of Node.js said it best himself, that his afraid people associate Node.js with scalability out of the box due to the single threaded asynchronous nature, that there exists something magical about it that would suddenly make your app scalable because it's riddled with async code. I now use Flask and Python instead of Node.js and it's a total pain reliever. Not having to deal with Javascript is a win on the server side (JS on client side is enough) and the python stack is good enough to handle every bit of Node.js Truth is, Node.js is in a bubble. I find it that even people who have never programmed before insisting that their new website be written in Node.js.... It reminded me of the dot com bubble, when your grandma would explain to you her rationale for buying dot com stocks (they had no idea).

WhiteNoiz3 13y ago

My argument would be that if you go 'sync first, async if you feel like it', everyone will just do sync, because it is easier conceptually (or atleast more familiar to a lot of people).

morganherlocker 13y ago

> If asynchronous code is harder to reason about, why would we elect to live in a world where it is the default? Because if you do not make it the default, then most libraries you use will make the choice for you by going sync.

EGreg 13y ago

Um, am I the only one who considers Node.js a beautiful tool suited for developing servers? Why async by default? Because defaults should encourage best practices. It can handle C10K problem out of the box. With libraries like Q, the code can actually be rather easy to follow and maintain. One of the biggest reasons why a multi-user server should be coded in async-by-default style is as follows: often, you want to obtain objects and not care about how they are obtained. In procedural languages, this isn't possible. Consider the following challenges: 1) Query sent to 10 different shards, wait until they all return, combine the results in the app, and then use them. In a synchronous language, you HAVE TO go to one shard, then another, then another, and the time is 10x longer. In an async style, it's only as long as the longest query! 2) Some of the objects may have already been cached, whereas others need to be obtained. In addition, the ones that need to be obtained shouldn't be requested more than once at the same time, but callbacks should be placed on a waiting list. 3) Various parts of your code (e.g. from different requests) might want to request the same object. Not only that, but you might want to batch your requests. That is to say, wait 10 milliseconds and see if any other (unrelated) parts of your code also want to grab some objects from the same DB table. Suddenly you are issuing a lot less queries because you can build middleware like this easily in JS. Not so in PHP for example - I would know. A lot is wasted there. 4) Let's consider #1 some more. The queries may go to other HTTP endpoints - do you really want to wait that long for I/O? Furthermore, even if they are going to local MySQL shards, your site becomes slower with an extra factor of O(log n) with the number of users. Kind of like when you use a relational database instead of a graph database for one-degree-away lookups. It could have been O(1) but instead you introduced another factor of O(log n). Not terrible, but certainly gives you a 5-20x cost down the line.

bascule 13y ago
> In a synchronous language, you HAVE TO go to one shard, then another, then another, and the time is 10x longer. Have you ever heard of threads?
- dharbin 13y ago
  Threads are more difficult than writing event-loop based code IMO

pkinsky 13y ago

Asynchronicity is great, but javascript doesn't provide the tools to do it properly. > (...) try writing a function call that requires information from two separate HTTP API responses; I basically need to draw a diagram of what happens with async.waterfall for a task that, given synchronicity, would've been solved with a trivial three-liner. It's a lot easier to handle asynchronicity in strongly-typed languages with monadic for comprehensions. In Scala, for example: val res: Future[Foobar] = for { a <- makeHttpRequestA() b <- makeHttpRequestBFromA(a) } yield new Foobar(a, b) You can then register callback functions which operate on either a Success[Foobar] or a Failure[Throwable]. Synchronous code isn't simpler, it just hides complexity like caltrops in tall grass. (slightly modified version of my post on Github)

eldude 13y ago
co(function *(){ let promiseA = makeHttpRequestA() let promiseB = makeHttpRequestBFromA(yield promiseA) var res = new Foobar(yield promiseA, yield promiseB); })();
- densh 13y ago
  with scala.async [1]: val res = async { val a = await(makeHttpRequestA()) val b = await(makeHttpRequestBFromA(a)) new Foobar(a, b) } [1] https://github.com/scala/async https://github.com/scala/async

anuraj 13y ago

In browser content may need to be asynchronous, but HTTP is request/response - nothing asynchronous. Half the web (or more) is not about feeding the browser. Use asynchronicity where needed.

bitcrusher 13y ago

I have a hard time 'listening' to these rants about how hard it is to reason about asynchronous code. It's NOT hard, it's DIFFERENT. This isn't some new Node-Sauce. The idea of an event-loop asychronous programming domain has been around forever in various forms, first in games and desktop GUI programming and the way one reasons about it hasn't changed, just because it has been applied to a new class of problems. I suspect that there are two things going on here: 1. A LOT of people dislike Javascript. As a result, they're starting with that negative point of view and then finding reasons to support it. This is not unlike people who argue about Python white-space. They're really saying "I don't like Python." The white-space issue is incidental. 2. Lack of experience / exposure I suspect that folks who have trouble with this, frankly just aren't experienced enough. I'm not saying they aren't skilled, just not experienced in this realm. Once upon a time, these same sorts of 'arguments' were used against Ruby ( ala the Rails explosion ). Nothing much has changed about Ruby, other than it has managed to be around long enough to influence a new generation of programmers.

jerf 13y ago
Yes, it is harder. You are attempting to reason about code in the absence of a useful stack trace. You're essentially taking your structured programming language and throwing it away and returning to the spaghetti-code era of code flow (this time with islands of structure). Both automated and human analysis is legitimately harder under these circumstances. "2. Lack of experience / exposure" Quite the contrary. I've had abundant experiences with both event-based models (which hardly started with Node!) in several paradigms (GUI, server, network) and models like Erlang, Go, and Haskell. It is no contest. The latter is sane. The former is not. If you have the opportunity to choose freely, and you choose the event-based model, you have chosen poorly. Believe me, it is not the critics of Node who are the inexperienced ones. All the evidence I've seen points strongly in the opposite direction. Node advocates could start by getting their understanding of how their competition works out of the mid-1990s. Ironically, in the end Node will eventually work their way around to working like these models. The signs are all there, the parallel evolution is clear, and apparently the process has already begun with Meteor. I wonder if the community will ever acknowledge how crazy they've been as they careen from one "solution" to the next at breakneck speed, all the while crowing about how much better their stuff is, even though what they consider "their stuff" can't seem to stay stable for 3 months?

woola 13y ago

Having written nodejs for about an year I would never choose it again if I have given the choice. * No Error handling. The process has to be restarted if something goes wrong. * No thread local storage equivalent. As a result you can't do simple thing like differentiate log originated from different http requests. * Callback makes the code unreadable * Javascript has no real method or in other words 'this' is not attached to a function and is determined by way the function is called. So you have to wrap most of the callbacks with _.bind() * Mixing callback, promise and event emitter api introduce lot of boilerplate code. * No stacktrace. Debugging error is PITA * No preemptive scheduling. So you have to be extra careful that you never spend too much CPU time. * No easy way to handle back pressure.

MrBuddyCasino 13y ago
These are exactly the reasons I have stayed away from it until now. I don't understand this one though, what does it mean? * No easy way to handle back pressure.
- Detrus 13y ago
  Latest developments of Node Streams begin to address back pressure.
- nostrademons 13y ago
  I did some Googling. "back pressure" in the Node context is basically what most people refer to as "flow control" or "overload behavior". Node programs work by enqueueing a bunch of work items for the event loop to handle, and then going on and doing a bunch more stuff until all the work items have been handled and their callbacks have been called. In the process, those callbacks may themselves enqueue further work items. The problem is that all these callbacks themselves take up memory and hold onto references in the V8 heap. And so if callbacks are being enqueued faster than they can be processed, the heap will get more full. V8's GC takes time proportional to the amount of live data, and so GCs will become more frequent and take longer. This further reduces the amount of available CPU time to process callbacks, which increases the amount of live data, until the system thrashes to a halt. Haskell programmers (who use a similar scheduling mechanism for lazy evaluation) call this a "space leak", and it's a big problem in Haskell as well. The solution is for the runtime to let the calling code know that it's overloaded, and then either temporarily block the calling code from creating more callbacks until existing ones have run, or let the calling code gracefully degrade and choose to handle things in a simpler fashion.

christiansmith 13y ago

The gist reads like someone looking for an excuse to give up and move on. Every time I run across someone bitching about Node's programming model I'm reminded of "The Blub Paradox". http://paulgraham.com/avg.html http://paulgraham.com/avg.html I have to admit that learning to think in callbacks was initially an exercise in frustration. But it wasn't that hard. Most of the pyramid-type issues can be resolved by applying principles that are also valid in synchronous environments (like modularity). If modules and named functions aren't enough, you can subclass EventEmitter to further decouple. For more complex cases there's https://github.com/caolan/async https://github.com/caolan/async, which makes many nontrivial async problems relatively easy. Why is it so hard for so many people to appreciate Node for its merits? The code isn't going to execute in a perfectly linear way, so why should it have to be written as if it were?

mattgreenrocks 13y ago

My big beef with Node.js: it could have avoided the sync/async split entirely by using a synchronous facade atop an asynchronous runtime...but it didn't. Why would you create an entire server platform and not use the chance to simplify the most common use case: fetching data from a database? I see as a great opportunity completely squandered so that it would cater to the masses: "you don't have to learn anything new!" Sometimes I think web tech likes being as shoddy and thrown-together as possible.

Cless 13y ago

You can have an async implementation and synchronous code. http://meteor.com http://meteor.com does that part quite nicely. The compiler or runtime should be handling this kind of cruft for us in most cases. Just because using Brainfuck might result in theoretically faster code than Ruby doesn't mean we should be writing our server in Brainfuck.

jwarkentin 13y ago

It seems that OP thinks the asynchronous nature of JavaScript was meant as a performance optimization. Let's go back to its origins. It was originally a scripting language for the browser. If it were to be synchronous and/or allow blocking things, such as `sleep()` or whatever else, it would have created an unusable web experience. Web pages would be constantly locking up. It's only recently that it's moved to the server where the asynchronicity on I/O isn't so mandatory. I personally still think it's a good thing, but that could definitely be debated on the server side. Unfortunately, if you want to have the advantage of writing one code base that can run on the browser or server, then it must work the same in both places.

neya 13y ago

I was reading somewhere (maybe Hackernews itself?) that synchronous programming and/or blocking are actually features and not disadvantages. After working on a huge Node.JS project, I regret taking on anyway (I had to do it for the moolah, aka money). I am now convinced that migrating from Rails to Express (A node.Js framework) was a total mistake. Please note, I do not mean to advertise rails in particular, infact, you can replace rails with anything else thats fits the analogy - perhaps maybe a python framework. However, this mistake has taught me something(s) very valuable - 1) Don't trade in a tried and tested framework for something like Node.js, that will open up a different level of problems altogether. 2) No matter what language you use, you WILL face issues at scale. Be it Node.js/Scalatra/Rails. You might as well face it with a tried and tested framework. 3) Don't re-write your own framework in something like Go or Haskell or C or C++ or etc. thinking it will be scalable, etc. You will - a) Waste massive amounts of time re-inventing the wheel, b) Open yourself to possibly dangerous security vulnerabilities (Eg: Handling encrypted cookies), c) Waste your time on things that don't really matter instead of on something that actually matters - The product itself. 4) Anything that you can scale by throwing more money at it is for the win[1] Why am I talking about scalability, when the discussion is about Async. vs Sync. programming? Because of the reasons many people cite using Node.Js and the like. It's just not worth it. Focus on the product first and the architecture later. This comes from someone who wasted a year of his life doing everything above. [1]http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html http://highscalability.com/blog/2013/4/15/scaling-pinterest-...

mclenithan 13y ago
Sorry you had a bad experience, bud. I am on a team of six, all pretty experienced with Node, and we don't have those issues. We've been going production on it for about a year and a half now. People cried a little in the beginning because it was different but now that we have embraced (and love) callbacks, we rip through development pretty quick. Things aren't always done for you but it's getting pretty rare at this point.

mistercow 13y ago

>but I'd be very curious to hear a defense of "async-first" thinking for problems that are typically solved on the server-side Because if your web server blocks every time it accesses a file, or talks to a database, or communicates over the network, then it won't take many requests at the same time before it's completely bogged down.

angersock 13y ago

So, here's how I think about it: Any program, of any type, has a synchronous story to it: data came in, then this happened to it, then this happened to it, then it left. If you follow the program execution, this is how it works. It's very easy to reason about, right? And it's very easy to program this way. One of the biggest breakthroughs in computer engineering was the idea of multiple concurrent processes (and nevermind that behind the scenes they were still running on a single processor!), and then later still true concurrent multi-processor architectures. We developed operating systems and compilers and runtimes specifically to let us take advantage of that without the having to break from easy-to-reason-about synchronous code. ~ Node said, hell with it, everything is async all the time. This is really annoying at first, because there are many many times where sync is the best way to reason about something. That said, there is some beauty in such a hard-line approach: if you are truly async, handling an error is the same as handling a task which takes too long. Once you've paid that price properly, you're fine. (This is also what makes Erlang kind of cool.) The big frustration is that there really ought to be a way of writing synchronous-looking Javascript code (ES5) which--behind the scenes!--is multiplexed and made to run concurrently and without blocking. The mechanisms are all there for programmers to deal with exceptional failures (exceptions, anyone?), and so are synchronization points (functions having multiple arguments should probably join on the evaluation of those arguments before being invoked--that's something the interpreter should be doing already). I guess what I'm saying is...why don't JS interpreters do lazy evaluation and late binding?

bitwize 13y ago

Here's a fun fact that every Amiga programmer seems to understand that Unix programmers just never really got: actual computing hardware is inherently asynchronous and interrupt-driven. The whole Amiga system is structured around using the CPU to program the custom chips and initiate DMA transfers, and installing interrupt handlers that perform the next phase of the program once the custom chips finished their work. It's how shit got done on the Amiga. By contrast, the lack of good support in early Unix for asynchronous I/O is said to be one of the reasons why X11 is such an ugly pile of hacks. Even on Unix, if you wrote a server procedurally what you'd end up doing is having main() start a select loop that... dispatches incoming events to callbacks. All Node does really is to abstract away the select loop -- one less thing for the developer to worry about. The Node way is simpler than the conventional C way. So no, there's nothing inherently superior about linear, procedural code, especially when such code is not at all how a computer system that must interact with the outside world works. Whether asynchronous, callback-driven code is confusing or not depends on what background you come from. Put an old-school scener in front of Node and he might form an entirely different opinion about it from OP.

nostrademons 13y ago
I think the question the article is asking is "Why did did Node stop at abstracting the event loop away, when it could've also abstracted the execution context?" There are languages that do this - Go, Erlang, Haskell, and Python 3.4 all provide a sequential process/coroutine abstraction on top of an implementation that's fundamentally asynchronous. The question the author has isn't about implementation, it's about programming model. Programming languages & frameworks are ultimately made for humans, and most humans find it easier to reason about "Do this thing, then this thing, then these other things, until you're finished" than "I'm doing all these things at once!" We have computers to organize all the events, stack frames, wait queues, etc. that take us from the machine to the editor window.
- bitwize 13y ago
  What I'm saying is that a programming model that's confusing to you is perfectly natural to someone else who comes from an environment where that model is all you have. People have used interrupt/event handlers and callbacks literally since the dawn of computing. It's nothing new. It just has to be learned, the way programming itself has to be learned.

deividy 13y ago

Am I the only dev in the world that loves async?

schrijver 13y ago

As a designer I have been trying to dig this stuff. Server side JS seems like a huge win for all kinds of small websites and interesting ui experiments, because one can stay in one language and avoid code duplication. Yet in trying to set up some experiments that use a server-side git integration, I bumped into the kind of code described in this thread, and I find it super difficult to understand the async way of doing things. Here’s an example from the homepage of nodegit, which seem to be among the best maintained libraries that wrap libgit2. Thus I imagine it represents idiomatic node.js What the code does, is console.log details of all the repositories commits, git log style. // Load in the module. var git = require('nodegit'), async = require('async'); // Open the repository in the current directory. git.repo('.git', function(error, repository) { if (error) throw error; // Use the master branch (a branch is the HEAD commit) repository.branch('master', function(error, branch) { if (error) throw error; // History returns an event, and begins walking the history var history = branch.history(); // History emits 'commit' event for each commit in the branch's history history.on('commit', function(error, commit) { // Print out `git log` emulation. async.series([ function(callback) { commit.sha(callback); }, function(callback) { commit.date(callback); }, function(callback) { commit.author(function(error, author) { author.name(callback); }); }, function(callback) { commit.author(function(error, author) { author.email(callback); }); }, function(callback) { commit.message(callback); } ], function printCommit(error, results) { if (error) throw error; console.log('SHA ' + results[0]); console.log(results[1] * 1000); console.log(results[2] + ' <' + results[3] + '>'); console.log(results[4]); }); }); }); }); It looks so alien to me! Instead of the tree-walking and looping constructs I know from synchronous languages, we deal with a history that returns events as it walks the tree (btw, why is it that we can launch the history before we attach the event?). Then we have to actually import the ‘async’ module, because, uhm, we want to specify an order by which the different parts of the message gets logged. How then the actual callbacks work, I’d be happy if someone can explain that to me—why do I pass a function to an attribute of the object? Why don‘t I just return the value? Because the code is not written in a lineair way, I have to keep looking back and forth between callbacks to get a mental picture of what it does. For the record, what this translates to as synchronous Python: from pygit2 import Repository, GIT_SORT_TOPOLOGICAL repo = Repository('.git') for commit in repo.walk(repo.head.target, GIT_SORT_TOPOLOGICAL): print 'SHA %s' % commit.hex print commit.commit_time * 1000 print '%s <%s>' % (commit.author.name, commit.author.email) print commit.message I imagine that, as some of the commenters note, I will be able to start reading the async style if I get enough exposure to it. I wonder though, if it does not make the barrier to entry for server-side JavaScript too high for non-full-time programmers. In that sense it is interesting that the Meteor team chose to use the package Fibers, that allows one to write node.js in a procedural style. I imagine something like that is necessary for server side JS to become really mainstream in web design…