20 ms·

Interview with Ryan Dahl, Creator of Node.js

393 points by WhiteSource1 9y ago

s_kilk 9y ago
> That said, I think Node is not the best system to build a massive server web. I would definitely use Go for that. And honestly, that's basically the reason why I left Node. It was the realization that: oh, actually, this is not the best server side system ever. Really interesting. I can imagine others would refuse to give up on the thing they'd worked so hard on. But Dahl has the self-awareness to just step back and say "huh, guess this isn't so great after all". Huge props to the guy.
- jgrahamc 9y ago
  Agree. It takes a lot to walk away from something and say you were wrong. But, sadly, part of the saga of nodejs could have been avoided by people looking at the history of computing. If you look at Go's concurrency model it was entirely mapped out in the 1970s in CSP. There wasn't really any need for go the 'single thread, non-blocking' route that nodejs took and evangelize it as nirvana. There was a ton of distributed systems work done in the 1970s. Just look at Lamport's clocks paper. I'm not criticizing Dahl directly here, more a sense that people ignore history in a field that has only had history for far less than a century.
  - spraak 9y ago
    It is clear however that JS itself has benefited from Node in that it really increased attention on the language. It seems doubtful we'd have ES6 etc. without it
  - jrs95 9y ago
    While you are sort of right, the non-blocking thing was basically necessary to get JS on the server working. And while he was focused on the I/O thing, he accomplished a lot of things by accident, including making a really convenient server platform for JS.
    - thehardsphere 9y ago
      Right, but the whole reason for JS on the server was because JS had no I/O, so he could do the non-blocking thing. JS on the server was the means, not the end.
  - rpeden 9y ago
    It might be because it's relatively easy to end up doing something computing related without studying it directly. This was the case for Dahl and many others. It's definitely possible to catch up on the history of the field while you're working in it, but not everyone can find the time to do so.
  - peterbraden 9y ago
    While I agree that non-blocking isn't a panacea, I think that your attitude that it was an unnecessary experiment is a little dismissive. The fact is that green threads aren't a panacea either, and exploring the callback based pattern led to some really interesting development, evidenced by the explosion of javascript on the server . At the time that node took off, a huge amount of server code was python with the GIL, and there needed to be something to shake up the status quo.
  - deleted 9y ago
    [deleted]
  - amasad 9y ago
    The fact that Node survives as a platform for Fullstack JavaScript shows how serendipity works in invention: he set out to build a high performance network server and ended up building one of the largest platforms for building web applications. The reasons it ended being very big are clear in hindsight but are hard to predict: - Sharing code between server and client - Server side rendering - Lower barrier to entry than other platforms (learn once write anywhere) - Well-designed package system It was also helped by the fact that folks decided to write JS tooling in JS, be it minifier, transpilers, or test runners, without nodejs we would've built then on top of something else.
    - thehardsphere 9y ago
      What do people mean when they keep saying "server side rendering" in this specific Javascript/Node.js context? I ask because I've seen quite a few people tout that term as a recent innovation. It was my understanding that "server side rendering" was the way the web worked since the beginning, with the server generating the markup that is provided to the browser to render the page. Nobody called it "server side rendering" until we had "client side rendering", where Javascript executed in the browser provided the markup instead of whatever program was running on the server. I'm not trying to lecture or be snarky here; I'm genuinely confused as to what people mean when they say "server side rendering" as a new thing. Like, is there something novel that I'm missing because the term is overloaded with a definition I don't know?
      - amasad 9y ago
        It means you take your client side JavaScript code and render it on the server for performance and SEO. Previously you had to either duplicate the code in different languages or pick one or the other (either use angular or rails).
        steveklabnik 9y ago
        Not just that, but to accomplish it means that you need to provide the correct client-side APIs on the server side, so that your code renders the same in both places. Doing this nicely is a lot of work.
  - erokar 9y ago
    > If you look at Go's concurrency model it was entirely mapped out in the 1970s Isn't that Go's problem, though — that there isn't a single thing in the language that wasn't entierly mapped out in the 1970s?
    - geodel 9y ago
      It is only a problem to someone whose thought process dictates anything old is bad and anything new is good.
      - camus2 9y ago
        > It is only a problem to someone whose thought process dictates anything old is bad and anything new is good. What do you call new? generic programming, nullable types, type classes? all pioneered around 1973? Can you explain what "new" is?
    - spraak 9y ago
      Using ideas from history isn't a problem. And Go is unique in that it combined all these learned lessons into something new.
      - camus2 9y ago
        > Using ideas from history isn't a problem. And Go is unique in that it combined all these learned lessons into something new. what learned lessons, like pragmas and struct tags?
        jerf 9y ago
        A lot of is more in what was taken away rather than what was added. As someone who is occasionally accused of being too pro-Go on HN... I actually have no problems thinking of Go as a really nice and refined 1990s language. Another type of "Java done right". (I say "another" because I think C# has a really good claim to that as well, albeit in a very different direction.) There's a place for that in the world. As much as I love the cutting edge of programming languages too, I'm not sure that we're anywhere near as far along on knowing how to build really big systems with them as a lot of people think we are. (There's only one way to get there though, and I absolutely encourage people to keep trying.)
  - willvarfar 9y ago
    You couldn't have got CSP from writing a web server API to run with V8. To get CSP you needed green threading built into the runtime, and that's a much taller order. So it's hard to say that the not-critizing-directly Dahl should have learned from the past. We'd probably never have heard of him if he'd set out to build a new esoteric CSP language or runtime, just like we haven't heard of anyone around that time other than the golang team.
  - bbatha 9y ago
    > But, sadly, part of the saga of nodejs could have been avoided by people looking at the history of computing. If you look at Go's concurrency model it was entirely mapped out in the 1970s in CSP. There wasn't really any need for go the 'single thread, non-blocking' route that nodejs took and evangelize it as nirvana. There was a ton of distributed systems work done in the 1970s. Just look at Lamport's clocks paper. Non-Blocking IO is still the best we have because its a limitation in syscall interfaces. Node.js, nginx, go, etc all use epoll and their ilk under the hood. In fact, in go goroutines only ran by default on a single OS thread until go 1.5. The ergonomics of call-back based non-blocking io is the issue here, not non-blocking io itself. Because of javascript's history on the browser where callback based events are the only apis it made sense to copy this style of api for IO so stuff like `setTimeout` worked in both cases. Node's selling point with regard to non-blocking io was never about ergonomics, it was the fact that you had a dynamic scripting language where the entire ecosystem was using non-blocking io using the same standardized event loop. Contrast to the other popular "web" languages of the day, Ruby and Python which have had some non-blocking io for years but usage is isolated because most of the ecosystem doesn't support it at all, and when they do they completely different and incompatible low-level io stacks.
  - TheAceOfHearts 9y ago
    I agree that we regularly fail to learn from history. However, it's worth recognizing that trying to learn from history can be incredibly challenging. There's a mountains of research to filter through. Sometimes you discover a relevant document, only to find out it's not readily available or that you have to pay an exorbitant amount of money for access. You might pay a lot of money for something that doesn't even end up being useful to your problem. When there have been multiple attempts at solving the problem, each with mixed results, how do you actually identify the good parts from the bad ones? A lot of tech knowledge isn't easily discoverable. Heck, sometimes things don't even make sense without understanding their historical background. The filesystem layout in the unixes is a great example, on my system I have: /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, /usr/local/sbin.
    - komali2 9y ago
      A good example - I have no idea why unix filesystems are like that, and after some idle googling I still don't know. Can't even figure out where to start to find the history of those decisions. Certainly am finding what kind of stuff goes where.
      - nomel 9y ago
        Some old Unix devs/users ran out of disk space, so they mounted more disk to those points. This caused boot problems since some tools (like mount) needed to be available. All of that together, along with people blindly following convention, and you have the mess of the unix filesystem. Yep... [1] [1] http://lists.busybox.net/pipermail/busybox/2010-December/074114.html http://lists.busybox.net/pipermail/busybox/2010-December/074...
        komali2 9y ago
        oh my god, that's awful
  - zzzcpan 9y ago
    Lamport's paper was only the very beginning of distributed systems. And CSP was invented by mathematicians for themselves, not for real life programming. As opposed to Erlang, which was invented for real life programming of telephony applications later in 1980s. But people keep ignoring history and think that CSP is suitable somehow, and not actor model. CSP is an even bigger mistake, don't try to make it look like it isn't. Learn from history.
  - pcwalton 9y ago
    More than that. Go's model is just threads with a particularly idiosyncratic implementation, in userspace instead of in the kernel. This itself is nothing new, as it was tried by the Linux NGPT project in the early '90s. (NGPT was abandoned for being inferior to plain old 1:1 threads, which suggests that Go's approach is not the end state either.)
    - jabl 9y ago
      AFAIU a lot of the complexity and slowness of "traditional" M:N threading approaches were due to the requirement to follow POSIX semantics (signal handling, preemptivity, etc.) within a purely library/OS based approach (no changes in the code generated by the compiler). If you're doing a green threads implementation for a high level language runtime, those restrictions don't apply, to an extent. I think e.g. Erlang, Haskell or Go are examples of "green threads done right". (Now, I do think Rust did the right thing in getting rid of green threads, but that IMHO is more a result of the space Rust is in rather than a general indictment on the utility of green threads)
      - pcwalton 9y ago
        Go still has a lot of issues around the inability to preempt except at function call boundaries, which is an issue that doesn't exist in 1:1 threading. Fairness and priority inversion are likewise issues with M:N that apply equally well to Go's implementation. Signal handling is too, although any GC language pretty much has to sacrifice POSIX signal handling already... Green threads done right strikes me as something like Windows' user mode scheduling (or Google's switchto patch that sadly never made it upstream), in which threads really are 1:1 as far as the kernel is concerned, but manually scheduled by userspace. This requires kernel support, but it fixes every issue except preemption, which is better to just not fix. (Few Go programs actually benefit from multicore CPU scaling; for CPU bound tasks a better use of optimization time is just not writing in Go, which tends to result in better speedups than trying to scale Go due to Go's compiler being relatively immature.) In my ideal world, kernel-scheduled and user-scheduled threads would exist in the same language, and programmers could choose which one they want on a thread-by-thread basis. I/O calls would be equally compatible with either threading model (which could be done in a zero-overhead fashion with the proper kernel support), eliminating the problem of sync/async incompatibility. This can only really be done today on Windows, unfortunately...
- gggboxing 9y ago
  That was me and my thesis. I defended it, graduated, and the next day I was like "I don't want to ever look at this pile of trash again".
- devmunchies 9y ago
  Well he does work for Google. We don't know if there is any pressure for him to be be a Govangelist.
  - spraak 9y ago
    I see what you mean, but as a similar example, I never thought anyone at Google wouldn't use Google for search, but some use DuckDuckGo -- https://news.ycombinator.com/item?id=15068047 https://news.ycombinator.com/item?id=15068047 Plus, he said he was using Go before he joined Google.
  - pier25 9y ago
    TJ moved to Go too, and he doesn't work at Google.
    - devmunchies 9y ago
      Was I defending Node.js or something? Just stating the obvious.
- tedunangst 9y ago
  I think that happens fairly often. The first person to solve the problem learns a lot in the process. The people who come later don't see the problem, only the solution.
- jondubois 9y ago
  He doesn't give any specific reason why he thinks that Go is better. I've used both Go and Node.js and I came to the opposite conclusion. I'm not a huge fan of Goroutines spawning threads in the background. I think that spawning threads and processes should be explicit because there is a big performance penalty when multiple threads have to share a CPU core because of context switching. With Node.js, you have more control over the process count so you can minimize the amount of CPU context switching that happens by making sure that each process gets its own CPU core. That said, I do think that on a conceptual level, Go feels cleaner than Node.js... But since async/await was introduced I feel that Node.js has the upper hand again.
  - ploxiln 9y ago
    Well the thing is, Goroutines are not threads, they're coroutines. The go runtime uses up to a real thread per cpu core, and all your goroutines are run on those (fewer) real threads. The runtime also manages a shared thread pool for blocking syscalls e.g. local disk access and some kinds of name resolution depending on OS. But starting 10 goroutines is much cheaper than spawning 10 threads.
hacker21 9y ago
kaffefgerffe4
amoorthy 9y ago
Pretty cool how he took a very unusual career/personal route to become such an important figure in the programming world. Good reminder for a parent like me that getting your kids into a "top" school isn't a must to succeed.
- vqc 9y ago
  I mean, he went to two "Top 50" schools studying math. Do you mean he didn't go to an Ivy League or a school traditionally known as a top CS school?
  - amoorthy 9y ago
    Yes correct (and not saying that top schools are really top; good article on selection bias a few weeks ago on this). I was primarily pointing at the fact that he started in Community College, which is sometimes looked down upon.
  - dsacco 9y ago
    Personally, I don't particularly consider schools in the "top 50" to be "top schools." I would consider schools in the top 10, maybe 20 for a particular major to be the colloquial "top schools" in that major. So for Math and CS, yes, this mostly means mort of the Ivies and 10 or so other schools, including MIT and Stanford. In general I'm of the opinion that there is a lot of "top school" inflation in the United States. It's better to consider schools on a major by major basis, especially at the graduate level, where your advisor might be more important than the school. The reason I restrict the top to the top 10 (and top 20 for leniency) is because the admission standards are vastly different the first 10, then the next 10, then the next 10 - 20, and at that point there isn't a significant difference in "attainability" anymore. I term it this way because I think the word "top" is only useful for signaling, not for real qualitative comparison. As a specific example: NYU is listed as a top 30 school for computer science, but while it's a good program, it seems odd to list it as a "top CS school" - is it really that much better than the next 10 or so, or is the term just diluted? Likewise, NYU is in the top 10 for mathematics, which actually seems sane from my perspective. And in the top 25 schools for CS is Rice University - you need a new word to call the group that MIT, NYU and Rice are in, because "top" is no longer all that meaningful.
afshinmeh 9y ago
Ryan is so smart. I like the way he thinks. This blog post [1] is still one of favorites. [1]: http://tinyclouds.org/rant.html http://tinyclouds.org/rant.html
- mrec 9y ago
  In a similar vein, see "We Who Value Simplicity Have Built Incomprehensible Machines": http://prog21.dadgum.com/139.html http://prog21.dadgum.com/139.html
- altotrees 9y ago
  This is pretty awesome. I had never read this before. Simplicity is so easy to strive for, yet so hard to achieve sometimes.
2017Dude 9y ago
1. Put crap (JS) into paper bag at the door (server side). 2. Lit the bag and ring the doorbell (NPM). 3. Run away (Dahl).
- sctb 9y ago
  Please post substantively on HN or not at all. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
dsacco 9y ago
I find it interesting that he ended up at Google Brain working on deep learning research after writing Node. There seems to be a trend in the industry of taking people who are exceptional in one area and putting them on AI problems (e.g. Chris Lattner). I wonder how effective that cross pollination is.
- msoad 9y ago
  A lot of AI is super complicated large software systems and building those don't require academic ML knowledge rather they require knowing how to build good software.
- technofire 9y ago
  He has two math degrees from decent (R1, AAU) schools, so I wouldn't say it's a huge leap; he probably has better training for modern AI than most people working in the area with CS degrees.
- watwut 9y ago
  These AI projects require a lot of math. Someone who studied math and is also proven to be able to produce working software seems to be a great fit.
_nalply 9y ago
A serious problem with Node was the callback pyramid of doom. This means deeply nested callbacks such that the indentation wanders steeply to the right. When you do loops or have exceptions programming got very confusing. In effect you have to do a continuation passing transform. It's not difficult but the results are utterly unreadable. For example a simple for loop could be transformed into recursive function because this way it is easier to keep state between callbacks. It's a nightmare. What I found surprising in the early years of Node that some people in the mailing list had a «Real programmers don't use Pascal» attitude. Bruno Jouhier's developed streamline to simplify and automate the continuation passing transform. One could almost program in a «blocking» style. I found his work really amazing. However for that Bruno got attacked in the mailing list. Then Marcel Laverdet developed fibers. A different and equivalent way to solve the callback pyramid are fibers, coroutines and generators. This way we don't need the continuation passing transform. But the reception in the mailing list was at best only lukewarm. Anyway, the developers of Meteor saw the potential and decided to base the server side of the platform on fibers. And now JavaScript has generators and async/await. And lo and behold: with these official solutions the hard core programmers swayed and accepted «Pascal». In my opinion, Ryan Dahl has missed an opportunity. Node was by large not ready and finished when he left. He should have tried to convince the community to find a solution for the callback hell. I think I understand people like him. They are always on the lookout for fresh ideas.
- nas 9y ago
  This was easily foreseeable when he started on Node.js. The Twisted framework for Python had existed for years (decades?) before Node.js was created. Anyone familiar with programming in that style knew of the issues with "callback hell". I was the primary instigator of generators for Python. Part of my motivation was to do something like CSP (e.g. call-with-current-continuation, call/cc) without having to redo the whole language runtime and also kill off alternative Python implementations. Generators in Python are limited to a single stack level. They have been extended to better support async IO coding. They work and are probably nicer than using callbacks but are still pretty ugly IMHO. The message passing that Erlang does is the best solution to this kind of concurrency. You can't really bolt that on a side of an existing language though. Anyhow, I think Node is successful largely because Javascript is the language understood by web browsers. The non-blocking coding style is not some kind of magic bullet as Ryan originally suggested it was.
explodingcamera 9y ago
They need to fix their ui on mobile, there's no way to listen to the podcast and most people listen to podcasts on the go I would imagine.
killjoywashere 9y ago
But, but what were the colorization projects? What was the domain?
- limsup 9y ago
  http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/ http://tinyclouds.org/residency/ http://tinyclouds.org/residency/
bit_logic 9y ago
Many are forgetting the initial reason node.js became popular. Consider the popular server-side landscape before node.js. It was dominated by Java, Python, etc. The primary way these ecosystems handle concurrency is OS-level threading. There was nothing else in the popular languages. Each language had some niche library that did non-blocking I/O (Twisted for Python, Netty for Java), but these all had a critical flaw, which is the rest of the ecosystem didn't support it. Basically every library, almost all existing code used the threading model. And when you mix threaded libraries with a non-blocking I/O server, it completely falls apart because the threaded code blocks. Then came node.js. Look at the ecosystem it came into. JS itself has very little standard library, and nothing for I/O. It has a highly optimized VM supported by the resources of Google. It's a language known by many developers. node.js took this well developed clean slate and built a non-blocking I/O server on top of it. It offered a concurrency model on the server side that's completely new to many developers, an alternative to the traditional threading model everyone knew. Since server-side JS didn't exist yet, it forced all libraries to be written in this non-blocking way. Every package in NPM is written to support the core design of node.js which is non-blocking I/O. And it was exciting to many developers, it was a reason to rewrite everything in this new way.
- k__ 9y ago
  Was it? I had the feeling before Node.js, everything was PHP.
  - disordinary 9y ago
    Most things are still PHP thanks to Wordpress.
  - thehardsphere 9y ago
    They're both appealing to "web developers" of relatively less experience for similar reasons.
- valarauca1 9y ago
  Each language had some niche library that did non-blocking I/O [...] Netty for Java) Non-blocking IO has been in java standard library since 1.4 More then Netty supported it. Netty, Jetty, Dropwizard, Grizzly, VERTX. Apache Tomcat supported non-blocking IO 5 years ago.
jgrant27 9y ago
Software Engineering in 2017 : Personality cults and mostly a lack of appreciation for the history of CS.
dfabulich 9y ago
It seems weird to me to have an interview with Ryan Dahl today, August 31st, without talking about the political struggle the Node Foundation has been going through over the past week. http://www.zdnet.com/article/after-governance-breakdown-node-js-leaders-fight-for-its-survival/ http://www.zdnet.com/article/after-governance-breakdown-node...
- thehardsphere 9y ago
  What would he have to say about it? He stopped working on Node before there even was a Node Foundation.
- bdcravens 9y ago
  If you think back to when he quit node, it seemed to when it was due to not being interested in anything not involving the technology.
  - z3t4 9y ago
    You wont get much done if your focus is on politics and fashion.
thinbeige 9y ago
Even if Node is not the perfect server-server environment, you can go quite far just with knowing JS.
qaq 9y ago
Given free choice I'd use Elixir, but working with Node and Python at work have to say that for async web services to my surprise I prefer Node more and I def. like Yarn more as package manager.
tambourine_man 9y ago
There's an audio version for the lazy like me (it's a podcast) https://api.soundcloud.com/tracks/340298200/download?client_id=cUa40O3Jg3Emvp6Tv4U6ymYYO50NUGpJ https://api.soundcloud.com/tracks/340298200/download?client_...
robinduckett 9y ago
In this thread: people who don't use Node.js making wild presumptions about how it works. Callback hell? Been a few years since that's been a problem.
- vorpalhex 9y ago
  It depends :tm:. There are still a lot of codebases that are callback heavy. For one, callbacks are faster than promises in a lot of node versions by a very large margin. In other code bases, legacy rules still apply. Promises don't work well for some more complicated logical flows (though they're pretty damn perfect for the common ones) and async is still pretty new. Just because it's not a problem you deal with doesn't mean that it doesn't still affect plenty of devs.
  - robinduckett 9y ago
    You can mitigate callback hell using classes, which are far quicker than promises and easier for humans to parse than callback chains.
    - vorpalhex 9y ago
      Genuinely curious, how? Do you have some example code to post?
gaius 9y ago
this new paradigm of model view controller New... in the 1970s https://en.wikipedia.org/wiki/Model–view–controller#History https://en.wikipedia.org/wiki/Model–view–controller#History It boggles the mind how little "web devs" know about the history of the field. No wonder they keep reinventing the wheel.
- topgunsarg 9y ago
  Maybe he's talking about when it got popular, not when it was literally invented...but I guess you can just be pedantic and miss the point.
  - bdcravens 9y ago
    I was seeing frameworks in ColdFusion referencing MVC back around 2001.
    - gaius 9y ago
      Yeah, MVC was well known to the Visual C++ community in the 90s, and it was a well established concept among commercial (as opposed to academic) programmers even back then.
  - dang 9y ago
    The GP's interpretation was arguably uncharitable, but your reply was uncivil. When you have a good point, please don't use it to make the thread worse.
rmrfrmrf 9y ago
I can't help but feel like Google gently encouraged him to promote Go over Node. Or that being told by coworkers (goworkers?) for years-on-end that Go is better than Node has had an effect. Especially considering he talks about green threads like he's not quite sure what he's talking about. At any rate, there are many innovations where the creators weren't fully aware of their impact and eventually come to hate their own designs, so no love lost there.
- tgtweak 9y ago
  Node lets you do single thread non blocking. Go lets you do multi threaded non blocking. That's a pretty substantial delta in the server world where you can easily have 24+ cores available. Memory and cpu footprint of go is also drastically lower to node as it is a compiled language. That makes a difference when you are running in a cloud paying per GB of memory and per core. I say this while actively developing in node.
  - rmrfrmrf 9y ago
    The thing is, in real-world scenarios with databases, http, and regexs, golang and node.js have very similar performance with node beating golang in some cases. Furthermore, not everyone is developing cloud-based apps and trying to optimize for metered cost structures. If you need raw, parallelized, computational power, then I will without-a-doubt agree that Go is a better choice. However, to say that one would always use Go for any server (even webapps? come on...) truly doesn't make any sense. The only other benefit-of-the-doubt explanation I could come up with for such an explanation is that it seems like the creator has spent more time in academia than in "the real world" of development, and perhaps the persuit of "the ultimate async system" is more important to him than practicality.
Abishek_Muthian 9y ago
"And then, you know, maybe little servers to... maybe little development servers, and here and there, maybe some real servers serving live traffic. Node can be useful, or it can be the right choice for it. But if you're building a massively distributed DNS server, I would not choose Node." -Ryan
chj 9y ago
A lot of people talk about callback hell. For me, the biggest issue of Callbacks is that you don't have a single point to catch thrown errors. We need finer control than a global catch-all handler.
quocble 9y ago
Ryan Dahl's advice for building high performance web server: Use Go.
- romanovcode 9y ago
  nginx is faster then caddy
  - tgtweak 9y ago
    And better all around.
georgecalm 9y ago
I was always curious why Ryan Dahl left the community. He finally answered that question; not the way I thought he would though: "I think Node is not the best system to build a massive server web. I would definitely use Go for that. And honestly, that's basically the reason why I left Node." Go may be an excellent choice for massive non-web servers, I don't have enough experience in it to say. For the product I work on, though, Node.js is the way to go. It's the best framework that allows us to use the same exact code on the server and on the client to create a fast progressive site.

393 points by WhiteSource1 9y ago

> That said, I think Node is not the best system to build a massive server web. I would definitely use Go for that. And honestly, that's basically the reason why I left Node. It was the realization that: oh, actually, this is not the best server side system ever. Really interesting. I can imagine others would refuse to give up on the thing they'd worked so hard on. But Dahl has the self-awareness to just step back and say "huh, guess this isn't so great after all". Huge props to the guy.

jgrahamc 9y ago
Agree. It takes a lot to walk away from something and say you were wrong. But, sadly, part of the saga of nodejs could have been avoided by people looking at the history of computing. If you look at Go's concurrency model it was entirely mapped out in the 1970s in CSP. There wasn't really any need for go the 'single thread, non-blocking' route that nodejs took and evangelize it as nirvana. There was a ton of distributed systems work done in the 1970s. Just look at Lamport's clocks paper. I'm not criticizing Dahl directly here, more a sense that people ignore history in a field that has only had history for far less than a century.
- spraak 9y ago
  It is clear however that JS itself has benefited from Node in that it really increased attention on the language. It seems doubtful we'd have ES6 etc. without it
- jrs95 9y ago
  While you are sort of right, the non-blocking thing was basically necessary to get JS on the server working. And while he was focused on the I/O thing, he accomplished a lot of things by accident, including making a really convenient server platform for JS.
  - thehardsphere 9y ago
    Right, but the whole reason for JS on the server was because JS had no I/O, so he could do the non-blocking thing. JS on the server was the means, not the end.
- rpeden 9y ago
  It might be because it's relatively easy to end up doing something computing related without studying it directly. This was the case for Dahl and many others. It's definitely possible to catch up on the history of the field while you're working in it, but not everyone can find the time to do so.
- peterbraden 9y ago
  While I agree that non-blocking isn't a panacea, I think that your attitude that it was an unnecessary experiment is a little dismissive. The fact is that green threads aren't a panacea either, and exploring the callback based pattern led to some really interesting development, evidenced by the explosion of javascript on the server . At the time that node took off, a huge amount of server code was python with the GIL, and there needed to be something to shake up the status quo.
- deleted 9y ago
  [deleted]
- amasad 9y ago
  The fact that Node survives as a platform for Fullstack JavaScript shows how serendipity works in invention: he set out to build a high performance network server and ended up building one of the largest platforms for building web applications. The reasons it ended being very big are clear in hindsight but are hard to predict: - Sharing code between server and client - Server side rendering - Lower barrier to entry than other platforms (learn once write anywhere) - Well-designed package system It was also helped by the fact that folks decided to write JS tooling in JS, be it minifier, transpilers, or test runners, without nodejs we would've built then on top of something else.
  - thehardsphere 9y ago
    What do people mean when they keep saying "server side rendering" in this specific Javascript/Node.js context? I ask because I've seen quite a few people tout that term as a recent innovation. It was my understanding that "server side rendering" was the way the web worked since the beginning, with the server generating the markup that is provided to the browser to render the page. Nobody called it "server side rendering" until we had "client side rendering", where Javascript executed in the browser provided the markup instead of whatever program was running on the server. I'm not trying to lecture or be snarky here; I'm genuinely confused as to what people mean when they say "server side rendering" as a new thing. Like, is there something novel that I'm missing because the term is overloaded with a definition I don't know?
    - amasad 9y ago
      It means you take your client side JavaScript code and render it on the server for performance and SEO. Previously you had to either duplicate the code in different languages or pick one or the other (either use angular or rails).
      - steveklabnik 9y ago
        Not just that, but to accomplish it means that you need to provide the correct client-side APIs on the server side, so that your code renders the same in both places. Doing this nicely is a lot of work.
- erokar 9y ago
  > If you look at Go's concurrency model it was entirely mapped out in the 1970s Isn't that Go's problem, though — that there isn't a single thing in the language that wasn't entierly mapped out in the 1970s?
  - geodel 9y ago
    It is only a problem to someone whose thought process dictates anything old is bad and anything new is good.
    - camus2 9y ago
      > It is only a problem to someone whose thought process dictates anything old is bad and anything new is good. What do you call new? generic programming, nullable types, type classes? all pioneered around 1973? Can you explain what "new" is?
  - spraak 9y ago
    Using ideas from history isn't a problem. And Go is unique in that it combined all these learned lessons into something new.
    - camus2 9y ago
      > Using ideas from history isn't a problem. And Go is unique in that it combined all these learned lessons into something new. what learned lessons, like pragmas and struct tags?
      - jerf 9y ago
        A lot of is more in what was taken away rather than what was added. As someone who is occasionally accused of being too pro-Go on HN... I actually have no problems thinking of Go as a really nice and refined 1990s language. Another type of "Java done right". (I say "another" because I think C# has a really good claim to that as well, albeit in a very different direction.) There's a place for that in the world. As much as I love the cutting edge of programming languages too, I'm not sure that we're anywhere near as far along on knowing how to build really big systems with them as a lot of people think we are. (There's only one way to get there though, and I absolutely encourage people to keep trying.)
- willvarfar 9y ago
  You couldn't have got CSP from writing a web server API to run with V8. To get CSP you needed green threading built into the runtime, and that's a much taller order. So it's hard to say that the not-critizing-directly Dahl should have learned from the past. We'd probably never have heard of him if he'd set out to build a new esoteric CSP language or runtime, just like we haven't heard of anyone around that time other than the golang team.
- bbatha 9y ago
  > But, sadly, part of the saga of nodejs could have been avoided by people looking at the history of computing. If you look at Go's concurrency model it was entirely mapped out in the 1970s in CSP. There wasn't really any need for go the 'single thread, non-blocking' route that nodejs took and evangelize it as nirvana. There was a ton of distributed systems work done in the 1970s. Just look at Lamport's clocks paper. Non-Blocking IO is still the best we have because its a limitation in syscall interfaces. Node.js, nginx, go, etc all use epoll and their ilk under the hood. In fact, in go goroutines only ran by default on a single OS thread until go 1.5. The ergonomics of call-back based non-blocking io is the issue here, not non-blocking io itself. Because of javascript's history on the browser where callback based events are the only apis it made sense to copy this style of api for IO so stuff like `setTimeout` worked in both cases. Node's selling point with regard to non-blocking io was never about ergonomics, it was the fact that you had a dynamic scripting language where the entire ecosystem was using non-blocking io using the same standardized event loop. Contrast to the other popular "web" languages of the day, Ruby and Python which have had some non-blocking io for years but usage is isolated because most of the ecosystem doesn't support it at all, and when they do they completely different and incompatible low-level io stacks.
- TheAceOfHearts 9y ago
  I agree that we regularly fail to learn from history. However, it's worth recognizing that trying to learn from history can be incredibly challenging. There's a mountains of research to filter through. Sometimes you discover a relevant document, only to find out it's not readily available or that you have to pay an exorbitant amount of money for access. You might pay a lot of money for something that doesn't even end up being useful to your problem. When there have been multiple attempts at solving the problem, each with mixed results, how do you actually identify the good parts from the bad ones? A lot of tech knowledge isn't easily discoverable. Heck, sometimes things don't even make sense without understanding their historical background. The filesystem layout in the unixes is a great example, on my system I have: /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, /usr/local/sbin.
  - komali2 9y ago
    A good example - I have no idea why unix filesystems are like that, and after some idle googling I still don't know. Can't even figure out where to start to find the history of those decisions. Certainly am finding what kind of stuff goes where.
    - nomel 9y ago
      Some old Unix devs/users ran out of disk space, so they mounted more disk to those points. This caused boot problems since some tools (like mount) needed to be available. All of that together, along with people blindly following convention, and you have the mess of the unix filesystem. Yep... [1] [1] http://lists.busybox.net/pipermail/busybox/2010-December/074114.html http://lists.busybox.net/pipermail/busybox/2010-December/074...
      - komali2 9y ago
        oh my god, that's awful
- zzzcpan 9y ago
  Lamport's paper was only the very beginning of distributed systems. And CSP was invented by mathematicians for themselves, not for real life programming. As opposed to Erlang, which was invented for real life programming of telephony applications later in 1980s. But people keep ignoring history and think that CSP is suitable somehow, and not actor model. CSP is an even bigger mistake, don't try to make it look like it isn't. Learn from history.
- pcwalton 9y ago
  More than that. Go's model is just threads with a particularly idiosyncratic implementation, in userspace instead of in the kernel. This itself is nothing new, as it was tried by the Linux NGPT project in the early '90s. (NGPT was abandoned for being inferior to plain old 1:1 threads, which suggests that Go's approach is not the end state either.)
  - jabl 9y ago
    AFAIU a lot of the complexity and slowness of "traditional" M:N threading approaches were due to the requirement to follow POSIX semantics (signal handling, preemptivity, etc.) within a purely library/OS based approach (no changes in the code generated by the compiler). If you're doing a green threads implementation for a high level language runtime, those restrictions don't apply, to an extent. I think e.g. Erlang, Haskell or Go are examples of "green threads done right". (Now, I do think Rust did the right thing in getting rid of green threads, but that IMHO is more a result of the space Rust is in rather than a general indictment on the utility of green threads)
    - pcwalton 9y ago
      Go still has a lot of issues around the inability to preempt except at function call boundaries, which is an issue that doesn't exist in 1:1 threading. Fairness and priority inversion are likewise issues with M:N that apply equally well to Go's implementation. Signal handling is too, although any GC language pretty much has to sacrifice POSIX signal handling already... Green threads done right strikes me as something like Windows' user mode scheduling (or Google's switchto patch that sadly never made it upstream), in which threads really are 1:1 as far as the kernel is concerned, but manually scheduled by userspace. This requires kernel support, but it fixes every issue except preemption, which is better to just not fix. (Few Go programs actually benefit from multicore CPU scaling; for CPU bound tasks a better use of optimization time is just not writing in Go, which tends to result in better speedups than trying to scale Go due to Go's compiler being relatively immature.) In my ideal world, kernel-scheduled and user-scheduled threads would exist in the same language, and programmers could choose which one they want on a thread-by-thread basis. I/O calls would be equally compatible with either threading model (which could be done in a zero-overhead fashion with the proper kernel support), eliminating the problem of sync/async incompatibility. This can only really be done today on Windows, unfortunately...
gggboxing 9y ago
That was me and my thesis. I defended it, graduated, and the next day I was like "I don't want to ever look at this pile of trash again".
devmunchies 9y ago
Well he does work for Google. We don't know if there is any pressure for him to be be a Govangelist.
- spraak 9y ago
  I see what you mean, but as a similar example, I never thought anyone at Google wouldn't use Google for search, but some use DuckDuckGo -- https://news.ycombinator.com/item?id=15068047 https://news.ycombinator.com/item?id=15068047 Plus, he said he was using Go before he joined Google.
- pier25 9y ago
  TJ moved to Go too, and he doesn't work at Google.
  - devmunchies 9y ago
    Was I defending Node.js or something? Just stating the obvious.
tedunangst 9y ago
I think that happens fairly often. The first person to solve the problem learns a lot in the process. The people who come later don't see the problem, only the solution.
jondubois 9y ago
He doesn't give any specific reason why he thinks that Go is better. I've used both Go and Node.js and I came to the opposite conclusion. I'm not a huge fan of Goroutines spawning threads in the background. I think that spawning threads and processes should be explicit because there is a big performance penalty when multiple threads have to share a CPU core because of context switching. With Node.js, you have more control over the process count so you can minimize the amount of CPU context switching that happens by making sure that each process gets its own CPU core. That said, I do think that on a conceptual level, Go feels cleaner than Node.js... But since async/await was introduced I feel that Node.js has the upper hand again.
- ploxiln 9y ago
  Well the thing is, Goroutines are not threads, they're coroutines. The go runtime uses up to a real thread per cpu core, and all your goroutines are run on those (fewer) real threads. The runtime also manages a shared thread pool for blocking syscalls e.g. local disk access and some kinds of name resolution depending on OS. But starting 10 goroutines is much cheaper than spawning 10 threads.

hacker21 9y ago

kaffefgerffe4

amoorthy 9y ago

Pretty cool how he took a very unusual career/personal route to become such an important figure in the programming world. Good reminder for a parent like me that getting your kids into a "top" school isn't a must to succeed.

vqc 9y ago
I mean, he went to two "Top 50" schools studying math. Do you mean he didn't go to an Ivy League or a school traditionally known as a top CS school?
- amoorthy 9y ago
  Yes correct (and not saying that top schools are really top; good article on selection bias a few weeks ago on this). I was primarily pointing at the fact that he started in Community College, which is sometimes looked down upon.
- dsacco 9y ago
  Personally, I don't particularly consider schools in the "top 50" to be "top schools." I would consider schools in the top 10, maybe 20 for a particular major to be the colloquial "top schools" in that major. So for Math and CS, yes, this mostly means mort of the Ivies and 10 or so other schools, including MIT and Stanford. In general I'm of the opinion that there is a lot of "top school" inflation in the United States. It's better to consider schools on a major by major basis, especially at the graduate level, where your advisor might be more important than the school. The reason I restrict the top to the top 10 (and top 20 for leniency) is because the admission standards are vastly different the first 10, then the next 10, then the next 10 - 20, and at that point there isn't a significant difference in "attainability" anymore. I term it this way because I think the word "top" is only useful for signaling, not for real qualitative comparison. As a specific example: NYU is listed as a top 30 school for computer science, but while it's a good program, it seems odd to list it as a "top CS school" - is it really that much better than the next 10 or so, or is the term just diluted? Likewise, NYU is in the top 10 for mathematics, which actually seems sane from my perspective. And in the top 25 schools for CS is Rice University - you need a new word to call the group that MIT, NYU and Rice are in, because "top" is no longer all that meaningful.

afshinmeh 9y ago

Ryan is so smart. I like the way he thinks. This blog post [1] is still one of favorites. [1]: http://tinyclouds.org/rant.html http://tinyclouds.org/rant.html

mrec 9y ago
In a similar vein, see "We Who Value Simplicity Have Built Incomprehensible Machines": http://prog21.dadgum.com/139.html http://prog21.dadgum.com/139.html
altotrees 9y ago
This is pretty awesome. I had never read this before. Simplicity is so easy to strive for, yet so hard to achieve sometimes.

2017Dude 9y ago

1. Put crap (JS) into paper bag at the door (server side). 2. Lit the bag and ring the doorbell (NPM). 3. Run away (Dahl).

sctb 9y ago
Please post substantively on HN or not at all. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

dsacco 9y ago

I find it interesting that he ended up at Google Brain working on deep learning research after writing Node. There seems to be a trend in the industry of taking people who are exceptional in one area and putting them on AI problems (e.g. Chris Lattner). I wonder how effective that cross pollination is.

msoad 9y ago
A lot of AI is super complicated large software systems and building those don't require academic ML knowledge rather they require knowing how to build good software.
technofire 9y ago
He has two math degrees from decent (R1, AAU) schools, so I wouldn't say it's a huge leap; he probably has better training for modern AI than most people working in the area with CS degrees.
watwut 9y ago
These AI projects require a lot of math. Someone who studied math and is also proven to be able to produce working software seems to be a great fit.

_nalply 9y ago

A serious problem with Node was the callback pyramid of doom. This means deeply nested callbacks such that the indentation wanders steeply to the right. When you do loops or have exceptions programming got very confusing. In effect you have to do a continuation passing transform. It's not difficult but the results are utterly unreadable. For example a simple for loop could be transformed into recursive function because this way it is easier to keep state between callbacks. It's a nightmare. What I found surprising in the early years of Node that some people in the mailing list had a «Real programmers don't use Pascal» attitude. Bruno Jouhier's developed streamline to simplify and automate the continuation passing transform. One could almost program in a «blocking» style. I found his work really amazing. However for that Bruno got attacked in the mailing list. Then Marcel Laverdet developed fibers. A different and equivalent way to solve the callback pyramid are fibers, coroutines and generators. This way we don't need the continuation passing transform. But the reception in the mailing list was at best only lukewarm. Anyway, the developers of Meteor saw the potential and decided to base the server side of the platform on fibers. And now JavaScript has generators and async/await. And lo and behold: with these official solutions the hard core programmers swayed and accepted «Pascal». In my opinion, Ryan Dahl has missed an opportunity. Node was by large not ready and finished when he left. He should have tried to convince the community to find a solution for the callback hell. I think I understand people like him. They are always on the lookout for fresh ideas.

nas 9y ago
This was easily foreseeable when he started on Node.js. The Twisted framework for Python had existed for years (decades?) before Node.js was created. Anyone familiar with programming in that style knew of the issues with "callback hell". I was the primary instigator of generators for Python. Part of my motivation was to do something like CSP (e.g. call-with-current-continuation, call/cc) without having to redo the whole language runtime and also kill off alternative Python implementations. Generators in Python are limited to a single stack level. They have been extended to better support async IO coding. They work and are probably nicer than using callbacks but are still pretty ugly IMHO. The message passing that Erlang does is the best solution to this kind of concurrency. You can't really bolt that on a side of an existing language though. Anyhow, I think Node is successful largely because Javascript is the language understood by web browsers. The non-blocking coding style is not some kind of magic bullet as Ryan originally suggested it was.

explodingcamera 9y ago

They need to fix their ui on mobile, there's no way to listen to the podcast and most people listen to podcasts on the go I would imagine.

killjoywashere 9y ago

But, but what were the colorization projects? What was the domain?

limsup 9y ago
http://tinyclouds.org/colorize/ http://tinyclouds.org/colorize/ http://tinyclouds.org/residency/ http://tinyclouds.org/residency/

bit_logic 9y ago

Many are forgetting the initial reason node.js became popular. Consider the popular server-side landscape before node.js. It was dominated by Java, Python, etc. The primary way these ecosystems handle concurrency is OS-level threading. There was nothing else in the popular languages. Each language had some niche library that did non-blocking I/O (Twisted for Python, Netty for Java), but these all had a critical flaw, which is the rest of the ecosystem didn't support it. Basically every library, almost all existing code used the threading model. And when you mix threaded libraries with a non-blocking I/O server, it completely falls apart because the threaded code blocks. Then came node.js. Look at the ecosystem it came into. JS itself has very little standard library, and nothing for I/O. It has a highly optimized VM supported by the resources of Google. It's a language known by many developers. node.js took this well developed clean slate and built a non-blocking I/O server on top of it. It offered a concurrency model on the server side that's completely new to many developers, an alternative to the traditional threading model everyone knew. Since server-side JS didn't exist yet, it forced all libraries to be written in this non-blocking way. Every package in NPM is written to support the core design of node.js which is non-blocking I/O. And it was exciting to many developers, it was a reason to rewrite everything in this new way.

k__ 9y ago
Was it? I had the feeling before Node.js, everything was PHP.
- disordinary 9y ago
  Most things are still PHP thanks to Wordpress.
- thehardsphere 9y ago
  They're both appealing to "web developers" of relatively less experience for similar reasons.
valarauca1 9y ago
Each language had some niche library that did non-blocking I/O [...] Netty for Java) Non-blocking IO has been in java standard library since 1.4 More then Netty supported it. Netty, Jetty, Dropwizard, Grizzly, VERTX. Apache Tomcat supported non-blocking IO 5 years ago.

jgrant27 9y ago

Software Engineering in 2017 : Personality cults and mostly a lack of appreciation for the history of CS.

dfabulich 9y ago

It seems weird to me to have an interview with Ryan Dahl today, August 31st, without talking about the political struggle the Node Foundation has been going through over the past week. http://www.zdnet.com/article/after-governance-breakdown-node-js-leaders-fight-for-its-survival/ http://www.zdnet.com/article/after-governance-breakdown-node...

thehardsphere 9y ago
What would he have to say about it? He stopped working on Node before there even was a Node Foundation.
bdcravens 9y ago
If you think back to when he quit node, it seemed to when it was due to not being interested in anything not involving the technology.
- z3t4 9y ago
  You wont get much done if your focus is on politics and fashion.

thinbeige 9y ago

Even if Node is not the perfect server-server environment, you can go quite far just with knowing JS.

qaq 9y ago

Given free choice I'd use Elixir, but working with Node and Python at work have to say that for async web services to my surprise I prefer Node more and I def. like Yarn more as package manager.

tambourine_man 9y ago

There's an audio version for the lazy like me (it's a podcast) https://api.soundcloud.com/tracks/340298200/download?client_id=cUa40O3Jg3Emvp6Tv4U6ymYYO50NUGpJ https://api.soundcloud.com/tracks/340298200/download?client_...

robinduckett 9y ago

In this thread: people who don't use Node.js making wild presumptions about how it works. Callback hell? Been a few years since that's been a problem.

vorpalhex 9y ago
It depends :tm:. There are still a lot of codebases that are callback heavy. For one, callbacks are faster than promises in a lot of node versions by a very large margin. In other code bases, legacy rules still apply. Promises don't work well for some more complicated logical flows (though they're pretty damn perfect for the common ones) and async is still pretty new. Just because it's not a problem you deal with doesn't mean that it doesn't still affect plenty of devs.
- robinduckett 9y ago
  You can mitigate callback hell using classes, which are far quicker than promises and easier for humans to parse than callback chains.
  - vorpalhex 9y ago
    Genuinely curious, how? Do you have some example code to post?

gaius 9y ago

this new paradigm of model view controller New... in the 1970s https://en.wikipedia.org/wiki/Model–view–controller#History https://en.wikipedia.org/wiki/Model–view–controller#History It boggles the mind how little "web devs" know about the history of the field. No wonder they keep reinventing the wheel.

topgunsarg 9y ago
Maybe he's talking about when it got popular, not when it was literally invented...but I guess you can just be pedantic and miss the point.
- bdcravens 9y ago
  I was seeing frameworks in ColdFusion referencing MVC back around 2001.
  - gaius 9y ago
    Yeah, MVC was well known to the Visual C++ community in the 90s, and it was a well established concept among commercial (as opposed to academic) programmers even back then.
- dang 9y ago
  The GP's interpretation was arguably uncharitable, but your reply was uncivil. When you have a good point, please don't use it to make the thread worse.

rmrfrmrf 9y ago

I can't help but feel like Google gently encouraged him to promote Go over Node. Or that being told by coworkers (goworkers?) for years-on-end that Go is better than Node has had an effect. Especially considering he talks about green threads like he's not quite sure what he's talking about. At any rate, there are many innovations where the creators weren't fully aware of their impact and eventually come to hate their own designs, so no love lost there.

tgtweak 9y ago
Node lets you do single thread non blocking. Go lets you do multi threaded non blocking. That's a pretty substantial delta in the server world where you can easily have 24+ cores available. Memory and cpu footprint of go is also drastically lower to node as it is a compiled language. That makes a difference when you are running in a cloud paying per GB of memory and per core. I say this while actively developing in node.
- rmrfrmrf 9y ago
  The thing is, in real-world scenarios with databases, http, and regexs, golang and node.js have very similar performance with node beating golang in some cases. Furthermore, not everyone is developing cloud-based apps and trying to optimize for metered cost structures. If you need raw, parallelized, computational power, then I will without-a-doubt agree that Go is a better choice. However, to say that one would always use Go for any server (even webapps? come on...) truly doesn't make any sense. The only other benefit-of-the-doubt explanation I could come up with for such an explanation is that it seems like the creator has spent more time in academia than in "the real world" of development, and perhaps the persuit of "the ultimate async system" is more important to him than practicality.

Abishek_Muthian 9y ago

"And then, you know, maybe little servers to... maybe little development servers, and here and there, maybe some real servers serving live traffic. Node can be useful, or it can be the right choice for it. But if you're building a massively distributed DNS server, I would not choose Node." -Ryan

chj 9y ago

A lot of people talk about callback hell. For me, the biggest issue of Callbacks is that you don't have a single point to catch thrown errors. We need finer control than a global catch-all handler.

quocble 9y ago

Ryan Dahl's advice for building high performance web server: Use Go.

romanovcode 9y ago
nginx is faster then caddy
- tgtweak 9y ago
  And better all around.

georgecalm 9y ago

I was always curious why Ryan Dahl left the community. He finally answered that question; not the way I thought he would though: "I think Node is not the best system to build a massive server web. I would definitely use Go for that. And honestly, that's basically the reason why I left Node." Go may be an excellent choice for massive non-web servers, I don't have enough experience in it to say. For the product I work on, though, Node.js is the way to go. It's the best framework that allows us to use the same exact code on the server and on the client to create a fast progressive site.