5 ms·

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but it

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, that is what you want to optimize for. My interview screening question was pretty simple- "Is node.js single threaded or multithreaded?" And to most, they spit back the blogspam headline- "Single threaded!" I think the most correct answer is "its complicated" but would accept that because most people would say that is the "right" answer. So I would follow up with- "what exactly happens in a default installation if we have say... 5 requests come in at exactly the same time to just return some static content from disk?" (Node's default threadpool is 4). And here is where you could see their understanding just fell apart. Some would say they would be handled entirely synchronously, others completely in parallel- but then had no idea what the cause of the parallelism was. Very few actually understood that node is an event loop executing javascript backed by a threadpool for async operations. Before reading this post, I was like eh this is a waste of time- its typical medium bullshit- they almost certainly found they were doing some blocking call in the event loop and then removed it and voila, 30x speedup. It was interesting because it was a lot worse! They spent all this time and hard work figuring out everything but what was taking so long in the event loop, and it seems that was the last place they actually looked. Anyway, node can be a highly scalable platform (https://changelog.com/podcast/116 https://changelog.com/podcast/116) but you need to understand it or else it will bite you in the foot. When I was last doing this stuff, upwards of 80% of our time was being spent essentially just JSON.parse()'ing, and we were looking to move to protobufs to avoid that.

pier25 7y ago
Here is a SO answer that expands a bit more on kevstev excellent comment: https://stackoverflow.com/a/22644735/816478 https://stackoverflow.com/a/22644735/816478
x86_64Ubuntu 7y ago
What does happen with the 5th request.
- kevstev 7y ago
  It will get queued, until one of the 4 requests in front of it has its task, to return the file, complete.
  - jwalton 7y ago
    Well... yes. If you're using `express.static()`, or if all five requests are getting different things from the disk. If you're using something that caches your static content in-memory, then the first request will use the thread pool to read that content, but the other four requests won't touch the thread pool - it's all async IO at that point, so it's all happening "concurrently" in a single thread, being multiplexed right here: https://github.com/libuv/libuv/blob/1ce6393a5780538ad8601cae00c5bd079b9415a9/src/unix/posix-poll.c#L188 https://github.com/libuv/libuv/blob/1ce6393a5780538ad8601cae...
- NathanKP 7y ago
  It waits in a queue in a background thread in the node.js http library until your code can execute to handle it. So if your code takes a long time before returning back to the event loop the request will just wait in that queue for a long time before the next opportunity for the event loop to execute code in response to the event.
NathanKP 7y ago
This is why I recommend that anyone running Node in production use a tracing tool like New Relic. It's super easy to see what is blocking the event loop. Just choose a duration (say 10ms) and look for any execution spans that are longer than that duration. Ideally you want to be yielding back to the event loop at least every 1 ms. Anything that takes too long without yielding will show up as a latency delay before your code is able to start handling a new request (technically a background thread in Node.js will pick up the request, but your code won't start executing in response to it until you yield back to the event loop again). To be honest the more difficult thing to diagnose sometimes is event loop overburdening. If each of your execution spans are taking 1ms, then you can only do a max of 1000 of them per second (assuming there was no delay between executions, but there is). So if you are trying to handle a large number of requests per second the event loop may end up with say 1005 execution spans per second that it needs to execute to handle that request volume. Because you can't do 1005ms of work in 1000ms the extra work will queue up. So gradually you will end up with 5 backlogged execution spans stacking up per second. Each second you will get 5ms more latency. The overall request latency will just gradually increase and increase as work gets further and further delayed in the queue. Overall I just think of Node.js as a fancy CPU scheduler. As long as you give it even, decently sized chunks of work to schedule, and you don't give it too many to schedule you will be fine. Anyway I'm a huge fan of Node.js but yeah its easy to fall into some gotcha's if you don't study how it works. The simplicity is a bit misleading
- inglor 7y ago
  Elastic APM is better than New Relic in how it traces node and it is completely free and open source (you can use a cloud product). Disclosure and bias: I work on Node core and always hear ranting about incorrect usage in async_hooks in anyone but Elastic APM in core meetings. I used both products and have no affiliation to other companies.
  - NathanKP 7y ago
    Interesting! I've had too many bad experiences over the years with Elasticsearch and ELK stack so I usually avoid Elasticsearch based products like the plague. Maybe if someone else runs it for me though... Do you know what they are doing differently with async_hooks in Elastic APM?
inglor 7y ago
> Very few actually understood that node is an event loop executing javascript backed by a threadpool for async operations. This is true, and that JavaScript is mostly a synchronous programming language with host environments that can provide asynchronisity. A caveat though is that the most important part of I/O is network I/O (tcp/udp sockets) and Node uses real async operations there rather than a threadpool. FS is just really hard to get right in a cross platform way and that's why it's on the threadpool. Some other stuff like dns is also famously on the threadpool but tcp sockets are not - it's a big part of why Node is fast.
gameswithgo 7y ago
>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he comes up with something -> its dynamically typed -> lets run a huge banking infrastructure on this wat
- a13n 7y ago
  > safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.
  - Nextgrid 7y ago
    You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?
    - kevstev 7y ago
      I can appreciate your suspicion of a dynamically typed language, but type script isn't really a "hack" its a superset of the language. I am not a huge fan of it, to me its kind of the worst of both worlds- if you need type safety, use something else. However, if you haven't taken a look at Joi- https://hapi.dev/family/joi/?v=16.1.8 https://hapi.dev/family/joi/?v=16.1.8 you probably should. To me this was a very happy medium- you can enforce your "types" at the point of ingress and egress at your api, and even do validations there, while still being able to enjoy the flexibility of type safety within your code.
      - rclayton 7y ago
        Lol - I don’t think I’ve ever met someone as enamored with Joi as I am. I think I abuse it in all the same ways as you do. I even publish my Joi schemas for request validation with my API specifications.
Kiro 7y ago
I have a Node service where I get tens of thousands requests a second and I still thought Node was single threaded. Where can I read about this?
- rclayton 7y ago
  The event loop is single threaded. Async tasks are executed on I/O threads which are configurable. So if your app is I/O bound, the event loop will typically dequeue tasks pretty quickly allowing lots of requests.
golergka 7y ago
> When I was last doing this stuff, upwards of 80% of our time was being spent essentially just JSON.parse()'ing, and we were looking to move to protobufs to avoid that. It's only tangentially related to your question, but I can't help but ask this question: why people use JSON instead of protobufs at all? I'm mostly a client-side developer, and most of my server-side experience is in hobby projects; still, I always used protobufs and loved it. They never damaged my feature velocity, apart from an hour to set up the build system in the beginning, and type safety helped me quite a few times when I forgot to sync changes in protocol on client and server side. Are there some secret advantages of going with json that I don't see because of limited experience?
- kevstev 7y ago
  JSON existed before protobufs is really it. When I left the node world 3 years ago, protobufs were the new hot thing. Any new project should start with them over JSON imho. There is some friction to them though, and I think a lot of it is that most tutorials and beginner books like to keep things as simple as possible, and people start their little project, it gets traction, and then they figure out they need protobufs but now its hard to introduce. In most projects, even today, it seems that its the version 2.0 that gets protobufs, v1.0 keeps JSON for simplicity, unless you have a bunch of seasoned devs involved.
- deleted 7y ago
  [deleted]
avip 7y ago
That's a great interview question (especially if you're not so much into hiring :)) Another one is: what happens when a node process completed execution? // node ex.js function foo() { // something async here } foo() console.log('bye...') This is a fun question to discuss (I think some consider this a bug in node).
specialist 7y ago
At my last gig, I maintained a handful of existing, and created some new nodejs things. I had previously done a lot of Java and even hacked on Apache. I had no prior nodejs experience. The first thing that really bothered me about our use of nodejs was no one could say why stuff would fail in production. So many moving parts. One of my team members figured out some edge case interactions between nodejs and nginx (used for HTTPS), which I would have never figured out on my own. It wouldn't have even occurred to me to look there. But other crashes, caused by apparent leaks, were mystifying. The second, and bigger, thing that really bothered me about nodejs, and expressjs in particular, was the notion of back pressure is completely missing. If it's in there, I couldn't find it. So our endpoints were still accepting new socket connections without processing responses from backend services (eg redis, other nodejs endpoints, auth services), which would either zombie or ABEND those backends. And no one could figure out why. I only understood what was happening because I'd already been through all that "architecture" madness a decade earlier with Java services. I guess what I'm saying is while I LOVE nodejs' closeness to the metal, I didn't like going back in time 10-15 years. Also, npm is crap.
z3t4 7y ago
Node.js is a good abstraction layer. In my experience, everything get leaky once you get hundreds of concurrent users.
rclayton 7y ago
Is it really only 4? When I look at my VM stats I seem to recall it having something like 15. Of course, this could be a config change on the Node Alpine container.

pier25 7y ago

Here is a SO answer that expands a bit more on kevstev excellent comment: https://stackoverflow.com/a/22644735/816478 https://stackoverflow.com/a/22644735/816478

x86_64Ubuntu 7y ago

What does happen with the 5th request.

kevstev 7y ago
It will get queued, until one of the 4 requests in front of it has its task, to return the file, complete.
- jwalton 7y ago
  Well... yes. If you're using `express.static()`, or if all five requests are getting different things from the disk. If you're using something that caches your static content in-memory, then the first request will use the thread pool to read that content, but the other four requests won't touch the thread pool - it's all async IO at that point, so it's all happening "concurrently" in a single thread, being multiplexed right here: https://github.com/libuv/libuv/blob/1ce6393a5780538ad8601cae00c5bd079b9415a9/src/unix/posix-poll.c#L188 https://github.com/libuv/libuv/blob/1ce6393a5780538ad8601cae...
NathanKP 7y ago
It waits in a queue in a background thread in the node.js http library until your code can execute to handle it. So if your code takes a long time before returning back to the event loop the request will just wait in that queue for a long time before the next opportunity for the event loop to execute code in response to the event.

NathanKP 7y ago

This is why I recommend that anyone running Node in production use a tracing tool like New Relic. It's super easy to see what is blocking the event loop. Just choose a duration (say 10ms) and look for any execution spans that are longer than that duration. Ideally you want to be yielding back to the event loop at least every 1 ms. Anything that takes too long without yielding will show up as a latency delay before your code is able to start handling a new request (technically a background thread in Node.js will pick up the request, but your code won't start executing in response to it until you yield back to the event loop again). To be honest the more difficult thing to diagnose sometimes is event loop overburdening. If each of your execution spans are taking 1ms, then you can only do a max of 1000 of them per second (assuming there was no delay between executions, but there is). So if you are trying to handle a large number of requests per second the event loop may end up with say 1005 execution spans per second that it needs to execute to handle that request volume. Because you can't do 1005ms of work in 1000ms the extra work will queue up. So gradually you will end up with 5 backlogged execution spans stacking up per second. Each second you will get 5ms more latency. The overall request latency will just gradually increase and increase as work gets further and further delayed in the queue. Overall I just think of Node.js as a fancy CPU scheduler. As long as you give it even, decently sized chunks of work to schedule, and you don't give it too many to schedule you will be fine. Anyway I'm a huge fan of Node.js but yeah its easy to fall into some gotcha's if you don't study how it works. The simplicity is a bit misleading

inglor 7y ago
Elastic APM is better than New Relic in how it traces node and it is completely free and open source (you can use a cloud product). Disclosure and bias: I work on Node core and always hear ranting about incorrect usage in async_hooks in anyone but Elastic APM in core meetings. I used both products and have no affiliation to other companies.
- NathanKP 7y ago
  Interesting! I've had too many bad experiences over the years with Elasticsearch and ELK stack so I usually avoid Elasticsearch based products like the plague. Maybe if someone else runs it for me though... Do you know what they are doing differently with async_hooks in Elastic APM?

inglor 7y ago

> Very few actually understood that node is an event loop executing javascript backed by a threadpool for async operations. This is true, and that JavaScript is mostly a synchronous programming language with host environments that can provide asynchronisity. A caveat though is that the most important part of I/O is network I/O (tcp/udp sockets) and Node uses real async operations there rather than a threadpool. FS is just really hard to get right in a cross platform way and that's why it's on the threadpool. Some other stuff like dns is also famously on the threadpool but tcp sockets are not - it's a big part of why Node is fast.

gameswithgo 7y ago

>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he comes up with something -> its dynamically typed -> lets run a huge banking infrastructure on this wat

a13n 7y ago
> safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.
- Nextgrid 7y ago
  You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?
  - kevstev 7y ago
    I can appreciate your suspicion of a dynamically typed language, but type script isn't really a "hack" its a superset of the language. I am not a huge fan of it, to me its kind of the worst of both worlds- if you need type safety, use something else. However, if you haven't taken a look at Joi- https://hapi.dev/family/joi/?v=16.1.8 https://hapi.dev/family/joi/?v=16.1.8 you probably should. To me this was a very happy medium- you can enforce your "types" at the point of ingress and egress at your api, and even do validations there, while still being able to enjoy the flexibility of type safety within your code.
    - rclayton 7y ago
      Lol - I don’t think I’ve ever met someone as enamored with Joi as I am. I think I abuse it in all the same ways as you do. I even publish my Joi schemas for request validation with my API specifications.

Kiro 7y ago

I have a Node service where I get tens of thousands requests a second and I still thought Node was single threaded. Where can I read about this?

rclayton 7y ago
The event loop is single threaded. Async tasks are executed on I/O threads which are configurable. So if your app is I/O bound, the event loop will typically dequeue tasks pretty quickly allowing lots of requests.

golergka 7y ago

> When I was last doing this stuff, upwards of 80% of our time was being spent essentially just JSON.parse()'ing, and we were looking to move to protobufs to avoid that. It's only tangentially related to your question, but I can't help but ask this question: why people use JSON instead of protobufs at all? I'm mostly a client-side developer, and most of my server-side experience is in hobby projects; still, I always used protobufs and loved it. They never damaged my feature velocity, apart from an hour to set up the build system in the beginning, and type safety helped me quite a few times when I forgot to sync changes in protocol on client and server side. Are there some secret advantages of going with json that I don't see because of limited experience?

kevstev 7y ago
JSON existed before protobufs is really it. When I left the node world 3 years ago, protobufs were the new hot thing. Any new project should start with them over JSON imho. There is some friction to them though, and I think a lot of it is that most tutorials and beginner books like to keep things as simple as possible, and people start their little project, it gets traction, and then they figure out they need protobufs but now its hard to introduce. In most projects, even today, it seems that its the version 2.0 that gets protobufs, v1.0 keeps JSON for simplicity, unless you have a bunch of seasoned devs involved.
deleted 7y ago
[deleted]

avip 7y ago

That's a great interview question (especially if you're not so much into hiring :)) Another one is: what happens when a node process completed execution? // node ex.js function foo() { // something async here } foo() console.log('bye...') This is a fun question to discuss (I think some consider this a bug in node).

specialist 7y ago

At my last gig, I maintained a handful of existing, and created some new nodejs things. I had previously done a lot of Java and even hacked on Apache. I had no prior nodejs experience. The first thing that really bothered me about our use of nodejs was no one could say why stuff would fail in production. So many moving parts. One of my team members figured out some edge case interactions between nodejs and nginx (used for HTTPS), which I would have never figured out on my own. It wouldn't have even occurred to me to look there. But other crashes, caused by apparent leaks, were mystifying. The second, and bigger, thing that really bothered me about nodejs, and expressjs in particular, was the notion of back pressure is completely missing. If it's in there, I couldn't find it. So our endpoints were still accepting new socket connections without processing responses from backend services (eg redis, other nodejs endpoints, auth services), which would either zombie or ABEND those backends. And no one could figure out why. I only understood what was happening because I'd already been through all that "architecture" madness a decade earlier with Java services. I guess what I'm saying is while I LOVE nodejs' closeness to the metal, I didn't like going back in time 10-15 years. Also, npm is crap.

z3t4 7y ago

Node.js is a good abstraction layer. In my experience, everything get leaky once you get hundreds of concurrent users.

rclayton 7y ago

Is it really only 4? When I look at my VM stats I seem to recall it having something like 15. Of course, this could be a config change on the Node Alpine container.