9 ms·

Millions of active WebSockets with Node.js

94 points by selvan 4y ago

par 4y ago
Still love seeing such great performance from asynchronous processing. But I’m curious, is there something special about node providing this? Or can we achieve similar performance with other async web servers (python, go, swift, rust etc?)
- foepys 4y ago
  The library mentioned in the article is C++ code that extends Nodejs and basically bypasses all of it. Nothing is making Nodejs special here. The Readme says: > µWebSockets.js is a web server bypass for Node.js that reimplements eventing, networking, encryption, web protocols, routing and pub/sub in highly optimized C++
- fastball 4y ago
  Well this isn't really NodeJS, it's written in C++. The whole point of Node is that it is a wrapper around the V8 JS engine (the one used in Chrome) that allows you to run JS specifically in a non-browser environment. This project is more an extension to Node (or as it is called in the GitHub README, a "web server bypass for Node.js). You do actually see the same kinda thing happening in other ecosystems, especially Python – something runs slower than you want it to (because Python), so you re-write it in Rust/C/C++ and hook into the extension system of the lang for a seamless DX. Off the top of my head things like `asyncpg`, `orson`, `uvloop` and esp relevant in this case `websockets` are all async libs that do this. Mildly tangential but as someone who has spent a lot of time making the Python backend for my startup run as fast as possible, I would recommend avoiding Python if performance is your primary concern.
- akmittal 4y ago
  Definitely possible to get similar or better performance using go, swift and rust. Python is slower in most cases so i dont think it would be possible using python(unless calling C from python)
  - josephg 4y ago
    Yep. I'm also not sure if the server libraries in swift are mature & well optimized enough to handle millions of active websocket connections. Maybe someone with more swift experience can chime in! Whats the state of swift on the server these days?
- nivertech 4y ago
  C1M is not that impressive anymore, these days one need to do C10M;) Most of the work of accepting and holding concurrent TCP sockets done by the OS, not the language runtime. One can easily tune Linux kernel to 1M concurrent sockets. The real issues: memory usage per concurrent socket (idle or active), and ability to do something useful with all these active connections, e.g. send pings every 30s, or broadcast a message to all of them. I'm not sure NodeJS/C++ based system from this post will allow sending pings every 30s to 1M websockets, let alone to do something useful with them, beside some low-traffic or infrequent notifications (Of course one always need to perform realistic loadtests in order to answer these kinds of questions). Erlang/Elixir/BEAM have a relatively large memory usage per active socket, but it allows doing something useful with them under an easy to use programming model (read: no callback hell).
  - nkristoffersen 4y ago
    isn't "callback hell" a bit old fashioned now? With promises and async/await etc.
    - nivertech 4y ago
      Even with async/await it's still single-threaded. I might be mistaken, since I'm not up-to-date with the multithreading/concurrency/isolates in nodejs/V8. IMO async/await is error-prone and isn't an ideal programming model.
      - hnbad 4y ago
        > Even with async/await it's still single-threaded. That doesn't mean anything. V8 is single-threaded but Node.js I/O is non-blocking. The reason Node became popular in the first place is that companies started adopting it to fill the gaps in their existing infrastructure (e.g. Java) to offer "realtime" (i.e. web sockets or its experimental equivalents) communication. > IMO async/await is error-prone and isn't an ideal programming model. What's the point of unsubstantiated statements like that (btw "x is error-prone" is an empirical claim, so prefixing it with "IMO" just means "I can't back this up and don't care if it's true") other than stirring up pointless language rivalries? Just acknowledge that your off-hand comment about "callback hell" was anachronistic and don't try to come up with excuses to justify your preferences. I think Elixir is neat and hope it can see sufficient adoption for me to justify getting invested in it but that doesn't justify poopooing other languages, especially ones you admit not to have up-to-date knowledge about.
- joshxyz 4y ago
  Great thing for me is having nodejs bindings for the underlying c++ core is i get to iterate fast on nodejs. Any developer out there who have been fascinated by and have used socket.io should already be using this.
- s6ro 4y ago
  If anything, the article proves that when you want performance don't use Node.js - or more generally, abstractions have a cost. The actual library, as already pointed out, is written is C/C++ and bypasses Node for everything, other than providing an usage API layer. Its the same as claiming that Tensorflow is written in Python ;)
arsome 4y ago
"The theoretical limit is 65k connections per IP address" What's the driving factor for that limitation?
- awaisraad 4y ago
  "The TCP protocol provides 16 bits for the port number, and this is interpreted as an unsigned integer; all values are valid, apart from 0, and so the largest port number is (2^16 - 1) or 65,535."
  - arsome 4y ago
    Ah, on the client side not the server, makes sense. I was doing the math as the full IP + port, which would offer 48 bits, but with only one client IP you really only have a client port to play with and you must bind as long as the connection remains open.
    - miyuru 4y ago
      but the blog post says different. > The client side has similar settings but does not need to set up multiple IP addresses, obviously Also as I know, Linux will only use one IP for outbound connections, unless forcefully bind to another IP address in the code. The server only uses only one port. 9001 according to the code. https://github.com/uNetworking/uWebSockets.js/blob/875f16e1fa8483f1cbe6e50fdb25efe4fc383731/examples/WebSockets.js#L4 https://github.com/uNetworking/uWebSockets.js/blob/875f16e1f... This blog post does not make any sense.
      - Ndymium 4y ago
        A connection is identified by the tuple {host IP, host port, client IP, client port}. The server listens on one port, but the client has a separate port for each connection. This means that there can be 65k connections for a single {host IP, client IP} combination. There's two ways to get more connections: use more client IPs or use more host IPs. In this post the OP has decided to add more host IPs.
supermatt 4y ago
> The theoretical limit is 65k connections per IP address but the actual limit is often more like 20k, so we use multiple addresses to connect 20k to each (50 * 20k = 1 mil). Thats 65k per CLIENT IP address (i.e one per client IP:port pair). There should be no reason to use multiple IP addresses on the server. If (for some reason??) you need each client to have more than 65k connections, you can add a port instead of an IP > The client side has similar settings but does not need to set up multiple IP addresses, obviously. You got this backwards and are now having to use a pool of server IPs to connect to instead of a single one...
- deleted 4y ago
  [deleted]
- punnerud 4y ago
  The 65k limit is because of the port limit. WebSocket is using the “trick” of opening a port without answering before the server have any news to give, this way the messages are more push-like. How can this be fixed without more IP-addresses?
  - supermatt 4y ago
    What port limit? The server is listening on a single port.
    - forkerenok 4y ago
      EDIT: I was wrong, see child comments and the sibling comment from jacuqesm. --- Yes, it is listening on a single port, but accepted connections bind to a separate socket. Here's how it works under the hood: > The accept() call creates a new socket descriptor with the same properties as socket and returns it to the caller. [...] The new socket descriptor cannot be used to accept new connections. The original socket, socket, remains available to accept more connection requests. Pulled this from random IBM zOS docs, but it's in compliance with the Unix standard: https://www.ibm.com/docs/en/zos/2.4.0?topic=functions-accept-accept-new-connection-socket https://www.ibm.com/docs/en/zos/2.4.0?topic=functions-accept...
conradfr 4y ago
(2019)
akshayrajp 4y ago
I am currently looking for ways to build a service that can handle around 100k-200k active concurrent websocket connections on production. It's wild seeing this article here. Does anyone know of any alternative ways to do this? Most people seem to suggest using Elixir but I wonder if I can achieve the same using a more "conventional" language such as Java or Golang. This article covers Node.js for me, I guess.
- deleted 4y ago
  [deleted]
- lawik 4y ago
  Elixir is well suited to highly concurrent systems and work like this. I'm big on the whole Elixir ecosystem though so I haven't explored other options. I don't see why there would be anything stopping Go from being similarly capable as it also has a good reputation for concurrency and what I hear does preemptive scheduling. Java can probably do anything except be fun and lightweight so assuming you want to figure out the hoops to jump through. I assume it could.. Elixir can do it with the ergonomics and expressiveness of Python/Ruby. If you enjoy that level of abstraction I recommend it.
  - szundi 4y ago
    Java is slowly absorbing the ideas from other systems and is much more fun than it was. Also versatile.
  - a_c 4y ago
    Do you have any pointer, book preferably, in starting an exploratory Elixir project? I don't have any objective apart from giving the ecosystem a taste
    - conradfr 4y ago
      If you really want a book pick one from here [0]. First one is good. Personally I think just following the official guide [1] will give you all you need to get a taste of the language and the platform and decide if you like it or not. If you were talking about websockets in particular I guess realistically most people use Phoenix Channels [2] that give you websockets in ten lines of code. [0] https://elixir-lang.org/learning.html https://elixir-lang.org/learning.html [1] https://elixir-lang.org/getting-started/introduction.html https://elixir-lang.org/getting-started/introduction.html [2] https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html
sidcool 4y ago
Ok, I am a dev but I could not get the actual tech part of this. Can someone ELI5 dev?
nirvgorilla 4y ago
I don't know anything about node.js or websockets but I actually set them up and used them in Jitsi Meet's version of the Excalidraw Backend for its recent whiteboard implementation. Here's me looking at the websocket traffic (I think): https://youtu.be/4rlffwHUchk?t=1857 https://youtu.be/4rlffwHUchk?t=1857 I might not fully understand the technicals of it but I got it up and running and use it almost every day! :D Maybe I'll someday understand it.

94 points by selvan 4y ago

par 4y ago

Still love seeing such great performance from asynchronous processing. But I’m curious, is there something special about node providing this? Or can we achieve similar performance with other async web servers (python, go, swift, rust etc?)

foepys 4y ago
The library mentioned in the article is C++ code that extends Nodejs and basically bypasses all of it. Nothing is making Nodejs special here. The Readme says: > µWebSockets.js is a web server bypass for Node.js that reimplements eventing, networking, encryption, web protocols, routing and pub/sub in highly optimized C++
fastball 4y ago
Well this isn't really NodeJS, it's written in C++. The whole point of Node is that it is a wrapper around the V8 JS engine (the one used in Chrome) that allows you to run JS specifically in a non-browser environment. This project is more an extension to Node (or as it is called in the GitHub README, a "web server bypass for Node.js). You do actually see the same kinda thing happening in other ecosystems, especially Python – something runs slower than you want it to (because Python), so you re-write it in Rust/C/C++ and hook into the extension system of the lang for a seamless DX. Off the top of my head things like `asyncpg`, `orson`, `uvloop` and esp relevant in this case `websockets` are all async libs that do this. Mildly tangential but as someone who has spent a lot of time making the Python backend for my startup run as fast as possible, I would recommend avoiding Python if performance is your primary concern.
akmittal 4y ago
Definitely possible to get similar or better performance using go, swift and rust. Python is slower in most cases so i dont think it would be possible using python(unless calling C from python)
- josephg 4y ago
  Yep. I'm also not sure if the server libraries in swift are mature & well optimized enough to handle millions of active websocket connections. Maybe someone with more swift experience can chime in! Whats the state of swift on the server these days?
nivertech 4y ago
C1M is not that impressive anymore, these days one need to do C10M;) Most of the work of accepting and holding concurrent TCP sockets done by the OS, not the language runtime. One can easily tune Linux kernel to 1M concurrent sockets. The real issues: memory usage per concurrent socket (idle or active), and ability to do something useful with all these active connections, e.g. send pings every 30s, or broadcast a message to all of them. I'm not sure NodeJS/C++ based system from this post will allow sending pings every 30s to 1M websockets, let alone to do something useful with them, beside some low-traffic or infrequent notifications (Of course one always need to perform realistic loadtests in order to answer these kinds of questions). Erlang/Elixir/BEAM have a relatively large memory usage per active socket, but it allows doing something useful with them under an easy to use programming model (read: no callback hell).
- nkristoffersen 4y ago
  isn't "callback hell" a bit old fashioned now? With promises and async/await etc.
  - nivertech 4y ago
    Even with async/await it's still single-threaded. I might be mistaken, since I'm not up-to-date with the multithreading/concurrency/isolates in nodejs/V8. IMO async/await is error-prone and isn't an ideal programming model.
    - hnbad 4y ago
      > Even with async/await it's still single-threaded. That doesn't mean anything. V8 is single-threaded but Node.js I/O is non-blocking. The reason Node became popular in the first place is that companies started adopting it to fill the gaps in their existing infrastructure (e.g. Java) to offer "realtime" (i.e. web sockets or its experimental equivalents) communication. > IMO async/await is error-prone and isn't an ideal programming model. What's the point of unsubstantiated statements like that (btw "x is error-prone" is an empirical claim, so prefixing it with "IMO" just means "I can't back this up and don't care if it's true") other than stirring up pointless language rivalries? Just acknowledge that your off-hand comment about "callback hell" was anachronistic and don't try to come up with excuses to justify your preferences. I think Elixir is neat and hope it can see sufficient adoption for me to justify getting invested in it but that doesn't justify poopooing other languages, especially ones you admit not to have up-to-date knowledge about.
joshxyz 4y ago
Great thing for me is having nodejs bindings for the underlying c++ core is i get to iterate fast on nodejs. Any developer out there who have been fascinated by and have used socket.io should already be using this.
s6ro 4y ago
If anything, the article proves that when you want performance don't use Node.js - or more generally, abstractions have a cost. The actual library, as already pointed out, is written is C/C++ and bypasses Node for everything, other than providing an usage API layer. Its the same as claiming that Tensorflow is written in Python ;)

arsome 4y ago

"The theoretical limit is 65k connections per IP address" What's the driving factor for that limitation?

awaisraad 4y ago
"The TCP protocol provides 16 bits for the port number, and this is interpreted as an unsigned integer; all values are valid, apart from 0, and so the largest port number is (2^16 - 1) or 65,535."
- arsome 4y ago
  Ah, on the client side not the server, makes sense. I was doing the math as the full IP + port, which would offer 48 bits, but with only one client IP you really only have a client port to play with and you must bind as long as the connection remains open.
  - miyuru 4y ago
    but the blog post says different. > The client side has similar settings but does not need to set up multiple IP addresses, obviously Also as I know, Linux will only use one IP for outbound connections, unless forcefully bind to another IP address in the code. The server only uses only one port. 9001 according to the code. https://github.com/uNetworking/uWebSockets.js/blob/875f16e1fa8483f1cbe6e50fdb25efe4fc383731/examples/WebSockets.js#L4 https://github.com/uNetworking/uWebSockets.js/blob/875f16e1f... This blog post does not make any sense.
    - Ndymium 4y ago
      A connection is identified by the tuple {host IP, host port, client IP, client port}. The server listens on one port, but the client has a separate port for each connection. This means that there can be 65k connections for a single {host IP, client IP} combination. There's two ways to get more connections: use more client IPs or use more host IPs. In this post the OP has decided to add more host IPs.

supermatt 4y ago

> The theoretical limit is 65k connections per IP address but the actual limit is often more like 20k, so we use multiple addresses to connect 20k to each (50 * 20k = 1 mil). Thats 65k per CLIENT IP address (i.e one per client IP:port pair). There should be no reason to use multiple IP addresses on the server. If (for some reason??) you need each client to have more than 65k connections, you can add a port instead of an IP > The client side has similar settings but does not need to set up multiple IP addresses, obviously. You got this backwards and are now having to use a pool of server IPs to connect to instead of a single one...

deleted 4y ago
[deleted]
punnerud 4y ago
The 65k limit is because of the port limit. WebSocket is using the “trick” of opening a port without answering before the server have any news to give, this way the messages are more push-like. How can this be fixed without more IP-addresses?
- supermatt 4y ago
  What port limit? The server is listening on a single port.
  - forkerenok 4y ago
    EDIT: I was wrong, see child comments and the sibling comment from jacuqesm. --- Yes, it is listening on a single port, but accepted connections bind to a separate socket. Here's how it works under the hood: > The accept() call creates a new socket descriptor with the same properties as socket and returns it to the caller. [...] The new socket descriptor cannot be used to accept new connections. The original socket, socket, remains available to accept more connection requests. Pulled this from random IBM zOS docs, but it's in compliance with the Unix standard: https://www.ibm.com/docs/en/zos/2.4.0?topic=functions-accept-accept-new-connection-socket https://www.ibm.com/docs/en/zos/2.4.0?topic=functions-accept...

conradfr 4y ago

(2019)

akshayrajp 4y ago

I am currently looking for ways to build a service that can handle around 100k-200k active concurrent websocket connections on production. It's wild seeing this article here. Does anyone know of any alternative ways to do this? Most people seem to suggest using Elixir but I wonder if I can achieve the same using a more "conventional" language such as Java or Golang. This article covers Node.js for me, I guess.

deleted 4y ago
[deleted]
lawik 4y ago
Elixir is well suited to highly concurrent systems and work like this. I'm big on the whole Elixir ecosystem though so I haven't explored other options. I don't see why there would be anything stopping Go from being similarly capable as it also has a good reputation for concurrency and what I hear does preemptive scheduling. Java can probably do anything except be fun and lightweight so assuming you want to figure out the hoops to jump through. I assume it could.. Elixir can do it with the ergonomics and expressiveness of Python/Ruby. If you enjoy that level of abstraction I recommend it.
- szundi 4y ago
  Java is slowly absorbing the ideas from other systems and is much more fun than it was. Also versatile.
- a_c 4y ago
  Do you have any pointer, book preferably, in starting an exploratory Elixir project? I don't have any objective apart from giving the ecosystem a taste
  - conradfr 4y ago
    If you really want a book pick one from here [0]. First one is good. Personally I think just following the official guide [1] will give you all you need to get a taste of the language and the platform and decide if you like it or not. If you were talking about websockets in particular I guess realistically most people use Phoenix Channels [2] that give you websockets in ten lines of code. [0] https://elixir-lang.org/learning.html https://elixir-lang.org/learning.html [1] https://elixir-lang.org/getting-started/introduction.html https://elixir-lang.org/getting-started/introduction.html [2] https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html

sidcool 4y ago

Ok, I am a dev but I could not get the actual tech part of this. Can someone ELI5 dev?

nirvgorilla 4y ago

I don't know anything about node.js or websockets but I actually set them up and used them in Jitsi Meet's version of the Excalidraw Backend for its recent whiteboard implementation. Here's me looking at the websocket traffic (I think): https://youtu.be/4rlffwHUchk?t=1857 https://youtu.be/4rlffwHUchk?t=1857 I might not fully understand the technicals of it but I got it up and running and use it almost every day! :D Maybe I'll someday understand it.