8 ms·

More than one million requests per second in Node.js

255 points by alexhultman 10y ago

halayli 10y ago
often times webserver benchmarks are misleading because of how the tests were done. nginx is a fully fledged webserver with logging enabled out of the box, and other bells and whistles. By just having logs enabled for example you're adding significant load on the server because of log formatting, writes to disk, etc. At the very least include the configs of each server tested.
- henridf 10y ago
  And details on the wrk (load gen) setup too, please.
  - alexhultman 10y ago
    The pipelining benchmark is identical to that of Japronto (another, very similar thing posted here on HN a few days ago). Japronto's repo on GitHub holds the wrk pipelining script used. I haven't had the time to add configurations for every server tested (esp. Apache & NGINX) but the main point here is to showcase the Node.js vs. Node.js with µWS perf. difference.
    - swineflu 10y ago
      Does this pass all the HTTP tests in Node.js repo? If not the perf diff is irrelevant.
      - daliwali 10y ago
        It has a mostly-compatible API but strict conformance doesn't seem to be the goal here. If your application does not make use of obscure features provided by core http (it could probably be refactored to do without anyways), then it's a free boost in performance.
        swineflu 10y ago
        Is the req object a readable stream? is the res object a writable stream? How do you handle backpressure with this mostly-compatible API?
    - pdimitar 10y ago
      How did you not have the time? Apologies, I might be missing something, but was this an emergency work assignment? If not, then you should have taken the time to provide the information for a fair comparison with the other stacks. As it is, you're just asking the community to take your word for it.
      - SparkyMcUnicorn 10y ago
        We don't need to take his word for it. It's open source, so we can run the tests ourselves. I think it's completely understandable that he threw in the others, probably default config, without caring much about it since they weren't the point of the writeup.
notzorbo3 10y ago
It's important to note that with all these "X requests per second" benchmarks, they're almost never testing actual performance, but rather just less features. The architecture (event loop, forking, threading or any combination of those) also matters a lot, but they serve completely different purposes. For example, they're using Apache as a reference point, but Apache does so much more than their code example. For one thing, you'll want to try disabling .htaccess support and static file serving so Apache doesn't actually hit the disk, like their code example doesn't. I've found it trivial to make Python perform on the order of dozens of millions of requests per second, and I can keep scaling that basically indefinitely. But all I'm really testing, as is the given code example in the article, is a bit of looping and string manipulation.
- alexhultman 10y ago
  > I've found it trivial to make Python perform on the order of dozens of millions of requests per second [...] No, no you haven't. This is not a cluster of servers, this is one single thread serving 1 million responses per second. Inside of Node.js.
  - notzorbo3 10y ago
    Yes I have, and I'm not talking about a cluster of servers.
    - erikbye 10y ago
      With HTTP pipelining then?
- lessclue 10y ago
  > I've found it trivial to make Python perform on the order of dozens of millions of requests per second Really curious. How did you achieve that? When you say "dozens of millions", it implies a minimum of 24+ million requests per second, which is quite unbelievable.
  - 10y ago
albertTJames 10y ago
Thanks for this amazing work ! Can't wait to use it. As a community we have to work on addons and make node the true versatile and performant language it should be :) https://nodejs.org/api/addons.html https://nodejs.org/api/addons.html
- alexhultman 10y ago
  I definitely agree the Node.js universe needs to take a better look at using addons. My opinion is that one should only use JS for the application logic, which requires high productivity, and only (or mostly) implement core modules as addons. It makes sense to use JS where productivity matters and to skip it where performance matters.
c8g 10y ago
>HTTP pipelining (made famous by Japronto) >Japronto's own (ridiculous) pipeline script are you trolling? :)
- E6300 10y ago
  He is (a little bit).
TheAceOfHearts 10y ago
Although it mirrors what the other parent comments are making, I wish there was more information readily available (or maybe it is, and I'm just not aware of where to look for it?) information about what real-world performance is like in different cases. For example, in my job, since none of the frontend APIs need to handle that many requests at once, we're considering setting up a few node "frontend APIs" to lift application complexities from our JS single page app up one level. Stuff like having to hit multiple inconsistent APIs, dealing with formatting issues, etc. If you have a single API it seems much easier to deal with that, as well as expand it as time goes on. But due to lack of knowledge and experience, I don't have as much confidence with pushing this decision as I'd like. We'll obviously end up investing time and effort in performing benchmarks to make sure it meets our requirements first, but as since we're a startup that's not so large, we can't realistically afford to dump THAT much time into something that doesn't end up getting us some clear benefits. A bit related to the topic... I know it's not exciting and sexy, but I wish more people wrote about larger non-trivial applications and how they end up tackling the challenges they encountered and details of the kinds of scales they handled. Both with respect to architecture and scaling. Maybe it's my lack of experience, but I find it really difficult to guess at how much money certain things will end up costing before doing a "close-to-real-world implementation".
- martin_bech 10y ago
  If you need a consistent api (based on exisisting apis/endpoints) with formatting options you should consider graphql. Its made for exactly that purpose.
- dfischer 10y ago
  Sounds like GraphQL
  - nstj 10y ago
    Or a proxy such as AWS API Gateway or Apigee
- cel1ne 10y ago
  Look into postgrest instead of graphQL as well and use the database as a flexible middle layer to construct your API on demand. http://postgrest.com/en/v0.4/ http://postgrest.com/en/v0.4/
jterry 10y ago
This looks interesting. I'm surprised there aren't many existing native HTTP modules for NodeJS. Found websockets/was as an alternative https://github.com/websockets/ws https://github.com/websockets/ws It would be a fun experiment to implement a native HTTP module in Rust using Neon. https://github.com/neon-bindings/neon https://github.com/neon-bindings/neon
ttt111222333 10y ago
> As http sockets in µWS build on the same networking stack as its websockets, you can easily scale to millions of long polling connections, just like you can with the websockets. This is simply not possible with the built-in Node.js http networking stack as its connections are very heavyweight in comparison. I'm a bit confused by what's going on here. Are you saying the network stack required to do websockets is vastly superior to the network stack of http, and hence using a websockets network stack in http calls can produce superior results? (I didn't know the underlying networking would be different and any clarity would be helpful). I'm not really understanding the differences but it is definitely interesting nonetheless.
- alexhultman 10y ago
  What I mean with this is that any connection (socket) in Node.js builds on net.Socket which builds on uv_tcp_t which together requires a significant amount of memory (bloat). A socket in the networking stack of µWS is far more lightweight (which already has been shown when it comes to µWS's websockets). The "HttpSocket" of µWS is about as lightweight in memory usage as "WebSocket", which is far more lightweight than net.Socket in Node.js. One million WebSockets require about 300 mb of user space memory in µWS while this number is somewhere between 8 and 16 GB of user space memory using the built-in Node.js http server. µWS is a play on its "micro" (small) sockets.
  - azinman2 10y ago
    I feel like "bloat" is thrown about so much these days, with little credence to actually defining it in a per situation context. It would be far more credible to me to not use such a handwavey term and instead talk about what the memory differences are, and why one might use much less memory than the other. Often times one person's "bloat" is another's necessary feature to accomplish their goals. It's like saying Django has a lot of bloat in comparison to some super basic http lib, except it has all the features I'll need to build a non-trivial app.
deleted 10y ago
[deleted]
erichocean 10y ago
Happy µWS "customer" here. I've been using the C++ library standalone in production since October (i.e. without Node.js). Crazy fast, ultra-low memory usage, and was easy to integrate into our codebase. Author is hilarious and deeply cares about performance. Easily the best C++ WebSocket library. I'm not at all surprised Alex has managed to get some additional performance out of HTTP on Node.js as well.
- spiderkeys 10y ago
  We are also using the uWS C++ library in production and have been extremely pleased with the performance. Integrating it was trivial and we haven't had any issues. Alex has always been very responsive and helpful and his focus on performance is always extremely refreshing in the wake of the webdev world's "eh, good enough" mentality.
- hellofunk 10y ago
  I'll have to give µWS a try. I've been using websocket++ [0] which I've also found to be excellent and stable, but a bit verbose to use. [0] https://github.com/zaphoyd/websocketpp https://github.com/zaphoyd/websocketpp
xorcist 10y ago
Really? 5x faster than plain nginx? That's .. remarkable, if true. I can't seem to find the sources for that benchmark however.
- throwawayish 10y ago
  Wholly irrelevant, just like the Python "benchmark" that was on the front page yesterday.
  - xorcist 10y ago
    Irrelevant for what? It's opening and closing an unfathomable number of sockets in a short time span, so it would be limited to how many the kernel can handle. Maybe subject to limitations in the glibc epoll() wrapper as well. So it's not irrelevant if you want to benchmark some change in the kernel for example. (There is a http parser in there as well, but I don't think even replacing it with a dummy one would quadruple total throughput even. Which is why I'm skeptical. The Python thing didn't claim performance above nginx.)
- IshKebab 10y ago
  Yeah the thing is these benchmarks are measuring the bit that isn't slow. Think about it - if this were a significant benchmark and they could do a million requests a second then they could literally run 20 Googles on one machine. Obviously they can't, and the reason is that it isn't a significant benchmark. It doesn't actually do anything.
PunchTornado 10y ago
This kind of benchmark is completely useless. You tune the heck out of node.js and then take another tool without tuning it (JVM, apache, nginx etc), give it a ridiculous task that you'll never find in real world and present your results as if they are meaningful. Why do people still waste time doing it?
- romanovcode 10y ago
  Actually it's kind of funny how Node community are obsessed about benchmark and speed because Node is very slow on CPU and cannot even do parallelism properly.
  - parhurs 10y ago
    Could you mention what this is based on, please?
  - k__ 10y ago
    I remember the dev of uws got some hate last year. Ppl were complaining that uws would only perform well on C++ stacks and be slow on Node. So I guess he simply wanted to prove them wrong.
  - anilgulecha 10y ago
    That's completely incorrect. Well known cluster modules allows very easy parallelism at any level in your application. Plus Node is among the fastest interpreted languages around today -- coming close to the jvm in performance.
    - sanjayts 10y ago
      I am awfully wary of these statements which paint languages like Python (via PyPy), Javascript (via Node) as very close competitors of the JVM. Once the JIT engine kicks in, on "real" workloads, JVM beats the lights out of these carefully tuned interpreted languages on a CPU intensive workload.
      - 10y ago
qaq 10y ago
I think the key benefit is actually significantly reduced memory footprint.
- alexhultman 10y ago
  I agree, this is one single factor that is constant in all apps: your long kept sockets will require far less memory which directly impacts the number of long polling clients you can have. Having fast throughput is just a bonus.
diegorbaquero 10y ago
uWS certainly reduces the overhead to a minimum, saving lots of memory that can be used to scale up and saving CPU left to your app's code. I wrote this article a few months ago when I switched to uws in the WebTorrent tracker. https://hackernoon.com/%C2%B5ws-as-your-next-websocket-library-d34209686357?source=linkShare-a1d0f3f9aca2-1486218881 https://hackernoon.com/%C2%B5ws-as-your-next-websocket-libra...
- lpinca 10y ago
  Did you ever try to disable permessage-deflate with ws? It will never be as lightweight as uws because ws is built on top of `net.Socket` but I think you hit this ws issue https://github.com/websockets/ws/issues/804 https://github.com/websockets/ws/issues/804 in the WebTorrent tracker. I think ws will use 3/4 times more memory than uws with permessage-deflate disabled which is a lot, but far different from 47 times as advertised.
rafaelferreira 10y ago
I'd love to see a design document explaining the differences from, say, nginx, that enable this kind of performance results.
bhouston 10y ago
This is pretty great stuff. Please keep it up and don't pay attention to the naysayers. This type of optimization is great and will pay dividends down the road for a lot of projects if this can take off.
bfrog 10y ago
I'm more interested in the techempower style benchmarks, at least those show some sort of semblance of real life usage. Do some queries, return encoded json etc.
deleted 10y ago
[deleted]
socmag 10y ago
Fantastic work Alex. I sent you email earlier when I saw this. It really is stunning, and yes microbenchmarks are very important to me and my product. I personally really do want to know how much every piece costs so I can budget memory cycles and machines. So thanks for providing the data. Even if it is slightly "ballpark". We use it in our server as well (and have done for ages), and uWS just plain rocks. Highly recommended
kisuna 10y ago
Super Bowl 2017 Live Stream Free https://www.reddit.com/r/sb51live2017tv/ https://www.reddit.com/r/sb51live2017tv/
yunda 10y ago
What if standard http module to be replace with this in express? Is it going to work?

255 points by alexhultman 10y ago

often times webserver benchmarks are misleading because of how the tests were done. nginx is a fully fledged webserver with logging enabled out of the box, and other bells and whistles. By just having logs enabled for example you're adding significant load on the server because of log formatting, writes to disk, etc. At the very least include the configs of each server tested.

henridf 10y ago
And details on the wrk (load gen) setup too, please.
- alexhultman 10y ago
  The pipelining benchmark is identical to that of Japronto (another, very similar thing posted here on HN a few days ago). Japronto's repo on GitHub holds the wrk pipelining script used. I haven't had the time to add configurations for every server tested (esp. Apache & NGINX) but the main point here is to showcase the Node.js vs. Node.js with µWS perf. difference.
  - swineflu 10y ago
    Does this pass all the HTTP tests in Node.js repo? If not the perf diff is irrelevant.
    - daliwali 10y ago
      It has a mostly-compatible API but strict conformance doesn't seem to be the goal here. If your application does not make use of obscure features provided by core http (it could probably be refactored to do without anyways), then it's a free boost in performance.
      - swineflu 10y ago
        Is the req object a readable stream? is the res object a writable stream? How do you handle backpressure with this mostly-compatible API?
  - pdimitar 10y ago
    How did you not have the time? Apologies, I might be missing something, but was this an emergency work assignment? If not, then you should have taken the time to provide the information for a fair comparison with the other stacks. As it is, you're just asking the community to take your word for it.
    - SparkyMcUnicorn 10y ago
      We don't need to take his word for it. It's open source, so we can run the tests ourselves. I think it's completely understandable that he threw in the others, probably default config, without caring much about it since they weren't the point of the writeup.

notzorbo3 10y ago

It's important to note that with all these "X requests per second" benchmarks, they're almost never testing actual performance, but rather just less features. The architecture (event loop, forking, threading or any combination of those) also matters a lot, but they serve completely different purposes. For example, they're using Apache as a reference point, but Apache does so much more than their code example. For one thing, you'll want to try disabling .htaccess support and static file serving so Apache doesn't actually hit the disk, like their code example doesn't. I've found it trivial to make Python perform on the order of dozens of millions of requests per second, and I can keep scaling that basically indefinitely. But all I'm really testing, as is the given code example in the article, is a bit of looping and string manipulation.

alexhultman 10y ago
> I've found it trivial to make Python perform on the order of dozens of millions of requests per second [...] No, no you haven't. This is not a cluster of servers, this is one single thread serving 1 million responses per second. Inside of Node.js.
- notzorbo3 10y ago
  Yes I have, and I'm not talking about a cluster of servers.
  - erikbye 10y ago
    With HTTP pipelining then?
lessclue 10y ago
> I've found it trivial to make Python perform on the order of dozens of millions of requests per second Really curious. How did you achieve that? When you say "dozens of millions", it implies a minimum of 24+ million requests per second, which is quite unbelievable.
- 10y ago

albertTJames 10y ago

Thanks for this amazing work ! Can't wait to use it. As a community we have to work on addons and make node the true versatile and performant language it should be :) https://nodejs.org/api/addons.html https://nodejs.org/api/addons.html

alexhultman 10y ago
I definitely agree the Node.js universe needs to take a better look at using addons. My opinion is that one should only use JS for the application logic, which requires high productivity, and only (or mostly) implement core modules as addons. It makes sense to use JS where productivity matters and to skip it where performance matters.

c8g 10y ago

>HTTP pipelining (made famous by Japronto) >Japronto's own (ridiculous) pipeline script are you trolling? :)

E6300 10y ago
He is (a little bit).

TheAceOfHearts 10y ago

Although it mirrors what the other parent comments are making, I wish there was more information readily available (or maybe it is, and I'm just not aware of where to look for it?) information about what real-world performance is like in different cases. For example, in my job, since none of the frontend APIs need to handle that many requests at once, we're considering setting up a few node "frontend APIs" to lift application complexities from our JS single page app up one level. Stuff like having to hit multiple inconsistent APIs, dealing with formatting issues, etc. If you have a single API it seems much easier to deal with that, as well as expand it as time goes on. But due to lack of knowledge and experience, I don't have as much confidence with pushing this decision as I'd like. We'll obviously end up investing time and effort in performing benchmarks to make sure it meets our requirements first, but as since we're a startup that's not so large, we can't realistically afford to dump THAT much time into something that doesn't end up getting us some clear benefits. A bit related to the topic... I know it's not exciting and sexy, but I wish more people wrote about larger non-trivial applications and how they end up tackling the challenges they encountered and details of the kinds of scales they handled. Both with respect to architecture and scaling. Maybe it's my lack of experience, but I find it really difficult to guess at how much money certain things will end up costing before doing a "close-to-real-world implementation".

martin_bech 10y ago
If you need a consistent api (based on exisisting apis/endpoints) with formatting options you should consider graphql. Its made for exactly that purpose.
dfischer 10y ago
Sounds like GraphQL
- nstj 10y ago
  Or a proxy such as AWS API Gateway or Apigee
cel1ne 10y ago
Look into postgrest instead of graphQL as well and use the database as a flexible middle layer to construct your API on demand. http://postgrest.com/en/v0.4/ http://postgrest.com/en/v0.4/

jterry 10y ago

This looks interesting. I'm surprised there aren't many existing native HTTP modules for NodeJS. Found websockets/was as an alternative https://github.com/websockets/ws https://github.com/websockets/ws It would be a fun experiment to implement a native HTTP module in Rust using Neon. https://github.com/neon-bindings/neon https://github.com/neon-bindings/neon

ttt111222333 10y ago

> As http sockets in µWS build on the same networking stack as its websockets, you can easily scale to millions of long polling connections, just like you can with the websockets. This is simply not possible with the built-in Node.js http networking stack as its connections are very heavyweight in comparison. I'm a bit confused by what's going on here. Are you saying the network stack required to do websockets is vastly superior to the network stack of http, and hence using a websockets network stack in http calls can produce superior results? (I didn't know the underlying networking would be different and any clarity would be helpful). I'm not really understanding the differences but it is definitely interesting nonetheless.

alexhultman 10y ago
What I mean with this is that any connection (socket) in Node.js builds on net.Socket which builds on uv_tcp_t which together requires a significant amount of memory (bloat). A socket in the networking stack of µWS is far more lightweight (which already has been shown when it comes to µWS's websockets). The "HttpSocket" of µWS is about as lightweight in memory usage as "WebSocket", which is far more lightweight than net.Socket in Node.js. One million WebSockets require about 300 mb of user space memory in µWS while this number is somewhere between 8 and 16 GB of user space memory using the built-in Node.js http server. µWS is a play on its "micro" (small) sockets.
- azinman2 10y ago
  I feel like "bloat" is thrown about so much these days, with little credence to actually defining it in a per situation context. It would be far more credible to me to not use such a handwavey term and instead talk about what the memory differences are, and why one might use much less memory than the other. Often times one person's "bloat" is another's necessary feature to accomplish their goals. It's like saying Django has a lot of bloat in comparison to some super basic http lib, except it has all the features I'll need to build a non-trivial app.

deleted 10y ago

[deleted]

erichocean 10y ago

Happy µWS "customer" here. I've been using the C++ library standalone in production since October (i.e. without Node.js). Crazy fast, ultra-low memory usage, and was easy to integrate into our codebase. Author is hilarious and deeply cares about performance. Easily the best C++ WebSocket library. I'm not at all surprised Alex has managed to get some additional performance out of HTTP on Node.js as well.

spiderkeys 10y ago
We are also using the uWS C++ library in production and have been extremely pleased with the performance. Integrating it was trivial and we haven't had any issues. Alex has always been very responsive and helpful and his focus on performance is always extremely refreshing in the wake of the webdev world's "eh, good enough" mentality.
hellofunk 10y ago
I'll have to give µWS a try. I've been using websocket++ [0] which I've also found to be excellent and stable, but a bit verbose to use. [0] https://github.com/zaphoyd/websocketpp https://github.com/zaphoyd/websocketpp

xorcist 10y ago

Really? 5x faster than plain nginx? That's .. remarkable, if true. I can't seem to find the sources for that benchmark however.

throwawayish 10y ago
Wholly irrelevant, just like the Python "benchmark" that was on the front page yesterday.
- xorcist 10y ago
  Irrelevant for what? It's opening and closing an unfathomable number of sockets in a short time span, so it would be limited to how many the kernel can handle. Maybe subject to limitations in the glibc epoll() wrapper as well. So it's not irrelevant if you want to benchmark some change in the kernel for example. (There is a http parser in there as well, but I don't think even replacing it with a dummy one would quadruple total throughput even. Which is why I'm skeptical. The Python thing didn't claim performance above nginx.)
IshKebab 10y ago
Yeah the thing is these benchmarks are measuring the bit that isn't slow. Think about it - if this were a significant benchmark and they could do a million requests a second then they could literally run 20 Googles on one machine. Obviously they can't, and the reason is that it isn't a significant benchmark. It doesn't actually do anything.

PunchTornado 10y ago

This kind of benchmark is completely useless. You tune the heck out of node.js and then take another tool without tuning it (JVM, apache, nginx etc), give it a ridiculous task that you'll never find in real world and present your results as if they are meaningful. Why do people still waste time doing it?

romanovcode 10y ago
Actually it's kind of funny how Node community are obsessed about benchmark and speed because Node is very slow on CPU and cannot even do parallelism properly.
- parhurs 10y ago
  Could you mention what this is based on, please?
- k__ 10y ago
  I remember the dev of uws got some hate last year. Ppl were complaining that uws would only perform well on C++ stacks and be slow on Node. So I guess he simply wanted to prove them wrong.
- anilgulecha 10y ago
  That's completely incorrect. Well known cluster modules allows very easy parallelism at any level in your application. Plus Node is among the fastest interpreted languages around today -- coming close to the jvm in performance.
  - sanjayts 10y ago
    I am awfully wary of these statements which paint languages like Python (via PyPy), Javascript (via Node) as very close competitors of the JVM. Once the JIT engine kicks in, on "real" workloads, JVM beats the lights out of these carefully tuned interpreted languages on a CPU intensive workload.
    - 10y ago

qaq 10y ago

I think the key benefit is actually significantly reduced memory footprint.

alexhultman 10y ago
I agree, this is one single factor that is constant in all apps: your long kept sockets will require far less memory which directly impacts the number of long polling clients you can have. Having fast throughput is just a bonus.

diegorbaquero 10y ago

uWS certainly reduces the overhead to a minimum, saving lots of memory that can be used to scale up and saving CPU left to your app's code. I wrote this article a few months ago when I switched to uws in the WebTorrent tracker. https://hackernoon.com/%C2%B5ws-as-your-next-websocket-library-d34209686357?source=linkShare-a1d0f3f9aca2-1486218881 https://hackernoon.com/%C2%B5ws-as-your-next-websocket-libra...

lpinca 10y ago
Did you ever try to disable permessage-deflate with ws? It will never be as lightweight as uws because ws is built on top of `net.Socket` but I think you hit this ws issue https://github.com/websockets/ws/issues/804 https://github.com/websockets/ws/issues/804 in the WebTorrent tracker. I think ws will use 3/4 times more memory than uws with permessage-deflate disabled which is a lot, but far different from 47 times as advertised.

rafaelferreira 10y ago

I'd love to see a design document explaining the differences from, say, nginx, that enable this kind of performance results.

bhouston 10y ago

This is pretty great stuff. Please keep it up and don't pay attention to the naysayers. This type of optimization is great and will pay dividends down the road for a lot of projects if this can take off.

bfrog 10y ago

I'm more interested in the techempower style benchmarks, at least those show some sort of semblance of real life usage. Do some queries, return encoded json etc.

deleted 10y ago

[deleted]

socmag 10y ago

Fantastic work Alex. I sent you email earlier when I saw this. It really is stunning, and yes microbenchmarks are very important to me and my product. I personally really do want to know how much every piece costs so I can budget memory cycles and machines. So thanks for providing the data. Even if it is slightly "ballpark". We use it in our server as well (and have done for ages), and uWS just plain rocks. Highly recommended

kisuna 10y ago

Super Bowl 2017 Live Stream Free https://www.reddit.com/r/sb51live2017tv/ https://www.reddit.com/r/sb51live2017tv/

yunda 10y ago

What if standard http module to be replace with this in express? Is it going to work?