4 ms·

A serious problem with Node was the callback pyramid of doom. This means deeply nested callbacks such that the indentation wanders steeply to the right. When y

A serious problem with Node was the callback pyramid of doom. This means deeply nested callbacks such that the indentation wanders steeply to the right. When you do loops or have exceptions programming got very confusing. In effect you have to do a continuation passing transform. It's not difficult but the results are utterly unreadable. For example a simple for loop could be transformed into recursive function because this way it is easier to keep state between callbacks. It's a nightmare. What I found surprising in the early years of Node that some people in the mailing list had a «Real programmers don't use Pascal» attitude. Bruno Jouhier's developed streamline to simplify and automate the continuation passing transform. One could almost program in a «blocking» style. I found his work really amazing. However for that Bruno got attacked in the mailing list. Then Marcel Laverdet developed fibers. A different and equivalent way to solve the callback pyramid are fibers, coroutines and generators. This way we don't need the continuation passing transform. But the reception in the mailing list was at best only lukewarm. Anyway, the developers of Meteor saw the potential and decided to base the server side of the platform on fibers. And now JavaScript has generators and async/await. And lo and behold: with these official solutions the hard core programmers swayed and accepted «Pascal». In my opinion, Ryan Dahl has missed an opportunity. Node was by large not ready and finished when he left. He should have tried to convince the community to find a solution for the callback hell. I think I understand people like him. They are always on the lookout for fresh ideas.

nas 9y ago
This was easily foreseeable when he started on Node.js. The Twisted framework for Python had existed for years (decades?) before Node.js was created. Anyone familiar with programming in that style knew of the issues with "callback hell". I was the primary instigator of generators for Python. Part of my motivation was to do something like CSP (e.g. call-with-current-continuation, call/cc) without having to redo the whole language runtime and also kill off alternative Python implementations. Generators in Python are limited to a single stack level. They have been extended to better support async IO coding. They work and are probably nicer than using callbacks but are still pretty ugly IMHO. The message passing that Erlang does is the best solution to this kind of concurrency. You can't really bolt that on a side of an existing language though. Anyhow, I think Node is successful largely because Javascript is the language understood by web browsers. The non-blocking coding style is not some kind of magic bullet as Ryan originally suggested it was.
- k__ 9y ago
  Somehow I never experienced callback hell. It always felt like every other code-nesting problem to me. I mean, nobody talks about conditional hell, everyone acknowledges that you should flatten out your conditionals.
  - sah2ed 9y ago
    Callback hell was real, especially if you worked on projects that decided to join the node.js bandwagon in its early days. Then promises happened. https://twitter.com/winterbe_/status/541868081308790784 https://twitter.com/winterbe_/status/541868081308790784
    - mikewhy 9y ago
      Why post a contrived meme as an example? Especially one with a sane solution[1] just below. [1]: https://pbs.twimg.com/media/B4W2AmaCAAA-heo.png https://pbs.twimg.com/media/B4W2AmaCAAA-heo.png
      - sah2ed 9y ago
        It's an exaggeration, yes, but it does correctly summarize how I felt when I had to work with node.js code bases before stuff like promises became mainstream.
        andreime 9y ago
        An exaggeration could be written for any language. A decent programmer would not write code like the one you showcased as an example for callback hell. This https://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/springframework/aop/framework/AbstractSingletonProxyFactoryBean.html https://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/spr... is not an exaggeration and I still would not use it to showcase how I felt when I had to work with Java developers.
  - Scarbutt 9y ago
    In case you are just referring to the browser, callback hell is way more invasive in nodejs than in the browser.
watwut 9y ago
When your code wanders steeply to the right, then it is time to create named functions, variables or otherwise flatten it. Just like when the code nests too much for any other reason. But I agree that js attracted "Real programmers don't use Pascal" kind of programmers at first. Pascal style programmers perceived js as pure hell and treated it as abomination. Through, I am not sure whether they were swayed or rather moved elsewhere in disgust.
- int_19h 9y ago
  But that's the thing. With callbacks, code that is logically the same (except that it's async) that wasn't nesting too much before, was suddenly nesting too much after. And conversely, you can reduce nesting to, basically, zero in any imperative language, if you replace all your conditionals and loops with a bunch of labels and gotos. Which is to say, how much code nests is very much a language design issue. Good PL design should not cause unnecessary nesting where it's not warranted by the structure of the solution.
piscisaureus 9y ago
When green threads, fibers etc. were a hot topic in the node.js community I was pretty involved in core development. As far as I remember, we were never so much against the idea of it, but we didn't believe an actually good implementation was possible. The primary concern was scalability. At the time supporting high concurrency networking (10K+ connected clients) was a top priority for us. The solutions proposed by Laverdet and Jouhier used libcoro or fibers to achieve the desired "cooperative threading"; this may not involve actual OS threads but it does require the creation of a thread stack for every fiber. To handle 10K connections you would need 10K stacks so you have to allocate 40GB memory right there, too much. We were aware that there were more effectient ways to do it, but this would have big changes to the V8 javascript engine. We didn't have the resources (nor the skill) to get it done and take on the maintenance burden. There were also concerns about the surprising language semantics it created; suddenly callbacks might run "inside" a function, e.g. after it's called and before it has returned, not something a javascript user would normally expect to be possible. Nowadays with async/await the callback-hell problem is actually solvable (to the extent that node-fibers solved it, anyway). It'll take time though before that becomes noticable, because a lot of packages need to change their APIs to take advantage of it.
- _nalply 9y ago
  Just a small heads up: Bruno Jouhier's solution is a transpiler just doing the continuation passing style transform. In other words, one could program in a synchronous style and the code got transformed to callback style. Later he added features to be compatible with fibers.
andreasgonewild 9y ago
Finally :) I've been arguing for fibers, coroutines and generators with non-blocking IO as an alternative to bending everything backwards to fit in event loops for a long time. And begging for salvation from ever dumber and clumsier programming languages and more ignorant hipster programmers. I ended up writing my own language to investigate further since no one else seemed to be interested in stepping far enough out of the box. https://github.com/andreas-gone-wild/snackis/blob/master/snabel.md https://github.com/andreas-gone-wild/snackis/blob/master/sna...
erikpukinskis 9y ago
Your view is that the Node community finally "saw the light", but the other interpretation is that they buckled under pressure. I love callbacks. The best thing about them is that you always know precisely what will happen when a line of code runs. Not so with promises. Another nice thing is that you always know which line of code will run next. Not so with async. I see async in the language spec (and class too) as a concession made to people who couldn't be bothered to learn how to use JavaScript the way it was intended. Like burgers on the menu at a Mexican restaurant. And no judgement against them: I'm happy for them that they got the ruby-like JavaScript they always dreamed of. But I wish those people would be a tad less boastful and understand that they turned people like me into exiles in our own language community. I used to have a language that promoted small single purpose modules with functional interfaces. That's gone. It's mostly synchronish OO interfaces now. And bigger and bigger frameworks. My folk are going to have to basically move out. I don't see an alternative at this point but to fork NPM.

nas 9y ago

This was easily foreseeable when he started on Node.js. The Twisted framework for Python had existed for years (decades?) before Node.js was created. Anyone familiar with programming in that style knew of the issues with "callback hell". I was the primary instigator of generators for Python. Part of my motivation was to do something like CSP (e.g. call-with-current-continuation, call/cc) without having to redo the whole language runtime and also kill off alternative Python implementations. Generators in Python are limited to a single stack level. They have been extended to better support async IO coding. They work and are probably nicer than using callbacks but are still pretty ugly IMHO. The message passing that Erlang does is the best solution to this kind of concurrency. You can't really bolt that on a side of an existing language though. Anyhow, I think Node is successful largely because Javascript is the language understood by web browsers. The non-blocking coding style is not some kind of magic bullet as Ryan originally suggested it was.

k__ 9y ago
Somehow I never experienced callback hell. It always felt like every other code-nesting problem to me. I mean, nobody talks about conditional hell, everyone acknowledges that you should flatten out your conditionals.
- sah2ed 9y ago
  Callback hell was real, especially if you worked on projects that decided to join the node.js bandwagon in its early days. Then promises happened. https://twitter.com/winterbe_/status/541868081308790784 https://twitter.com/winterbe_/status/541868081308790784
  - mikewhy 9y ago
    Why post a contrived meme as an example? Especially one with a sane solution[1] just below. [1]: https://pbs.twimg.com/media/B4W2AmaCAAA-heo.png https://pbs.twimg.com/media/B4W2AmaCAAA-heo.png
    - sah2ed 9y ago
      It's an exaggeration, yes, but it does correctly summarize how I felt when I had to work with node.js code bases before stuff like promises became mainstream.
      - andreime 9y ago
        An exaggeration could be written for any language. A decent programmer would not write code like the one you showcased as an example for callback hell. This https://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/springframework/aop/framework/AbstractSingletonProxyFactoryBean.html https://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/spr... is not an exaggeration and I still would not use it to showcase how I felt when I had to work with Java developers.
- Scarbutt 9y ago
  In case you are just referring to the browser, callback hell is way more invasive in nodejs than in the browser.

watwut 9y ago

When your code wanders steeply to the right, then it is time to create named functions, variables or otherwise flatten it. Just like when the code nests too much for any other reason. But I agree that js attracted "Real programmers don't use Pascal" kind of programmers at first. Pascal style programmers perceived js as pure hell and treated it as abomination. Through, I am not sure whether they were swayed or rather moved elsewhere in disgust.

int_19h 9y ago
But that's the thing. With callbacks, code that is logically the same (except that it's async) that wasn't nesting too much before, was suddenly nesting too much after. And conversely, you can reduce nesting to, basically, zero in any imperative language, if you replace all your conditionals and loops with a bunch of labels and gotos. Which is to say, how much code nests is very much a language design issue. Good PL design should not cause unnecessary nesting where it's not warranted by the structure of the solution.

piscisaureus 9y ago

When green threads, fibers etc. were a hot topic in the node.js community I was pretty involved in core development. As far as I remember, we were never so much against the idea of it, but we didn't believe an actually good implementation was possible. The primary concern was scalability. At the time supporting high concurrency networking (10K+ connected clients) was a top priority for us. The solutions proposed by Laverdet and Jouhier used libcoro or fibers to achieve the desired "cooperative threading"; this may not involve actual OS threads but it does require the creation of a thread stack for every fiber. To handle 10K connections you would need 10K stacks so you have to allocate 40GB memory right there, too much. We were aware that there were more effectient ways to do it, but this would have big changes to the V8 javascript engine. We didn't have the resources (nor the skill) to get it done and take on the maintenance burden. There were also concerns about the surprising language semantics it created; suddenly callbacks might run "inside" a function, e.g. after it's called and before it has returned, not something a javascript user would normally expect to be possible. Nowadays with async/await the callback-hell problem is actually solvable (to the extent that node-fibers solved it, anyway). It'll take time though before that becomes noticable, because a lot of packages need to change their APIs to take advantage of it.

_nalply 9y ago
Just a small heads up: Bruno Jouhier's solution is a transpiler just doing the continuation passing style transform. In other words, one could program in a synchronous style and the code got transformed to callback style. Later he added features to be compatible with fibers.

andreasgonewild 9y ago

Finally :) I've been arguing for fibers, coroutines and generators with non-blocking IO as an alternative to bending everything backwards to fit in event loops for a long time. And begging for salvation from ever dumber and clumsier programming languages and more ignorant hipster programmers. I ended up writing my own language to investigate further since no one else seemed to be interested in stepping far enough out of the box. https://github.com/andreas-gone-wild/snackis/blob/master/snabel.md https://github.com/andreas-gone-wild/snackis/blob/master/sna...

erikpukinskis 9y ago

Your view is that the Node community finally "saw the light", but the other interpretation is that they buckled under pressure. I love callbacks. The best thing about them is that you always know precisely what will happen when a line of code runs. Not so with promises. Another nice thing is that you always know which line of code will run next. Not so with async. I see async in the language spec (and class too) as a concession made to people who couldn't be bothered to learn how to use JavaScript the way it was intended. Like burgers on the menu at a Mexican restaurant. And no judgement against them: I'm happy for them that they got the ruby-like JavaScript they always dreamed of. But I wish those people would be a tad less boastful and understand that they turned people like me into exiles in our own language community. I used to have a language that promoted small single purpose modules with functional interfaces. That's gone. It's mostly synchronish OO interfaces now. And bigger and bigger frameworks. My folk are going to have to basically move out. I don't see an alternative at this point but to fork NPM.