9 ms·

I am currently looking for ways to build a service that can handle around 100k-200k active concurrent websocket connections on production. It's wild seeing this

I am currently looking for ways to build a service that can handle around 100k-200k active concurrent websocket connections on production. It's wild seeing this article here. Does anyone know of any alternative ways to do this? Most people seem to suggest using Elixir but I wonder if I can achieve the same using a more "conventional" language such as Java or Golang. This article covers Node.js for me, I guess.

deleted 4y ago
[deleted]
lawik 4y ago
Elixir is well suited to highly concurrent systems and work like this. I'm big on the whole Elixir ecosystem though so I haven't explored other options. I don't see why there would be anything stopping Go from being similarly capable as it also has a good reputation for concurrency and what I hear does preemptive scheduling. Java can probably do anything except be fun and lightweight so assuming you want to figure out the hoops to jump through. I assume it could.. Elixir can do it with the ergonomics and expressiveness of Python/Ruby. If you enjoy that level of abstraction I recommend it.
- szundi 4y ago
  Java is slowly absorbing the ideas from other systems and is much more fun than it was. Also versatile.
- a_c 4y ago
  Do you have any pointer, book preferably, in starting an exploratory Elixir project? I don't have any objective apart from giving the ecosystem a taste
  - conradfr 4y ago
    If you really want a book pick one from here [0]. First one is good. Personally I think just following the official guide [1] will give you all you need to get a taste of the language and the platform and decide if you like it or not. If you were talking about websockets in particular I guess realistically most people use Phoenix Channels [2] that give you websockets in ten lines of code. [0] https://elixir-lang.org/learning.html https://elixir-lang.org/learning.html [1] https://elixir-lang.org/getting-started/introduction.html https://elixir-lang.org/getting-started/introduction.html [2] https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html
  - dns_snek 4y ago
    I can highly recommend Elixir in Action, 2nd ed. This talk by the same author is also a good introduction in video format: https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE
latch 4y ago
Honestly, what matters is (a) what you're going to be doing with those connections and (b) your hardware. As a generalization (again, really depends what you're going to be doing), I'd expect people to get a lot further with a Go or Java based implementations. Specifically, if those connections are interacting with each other in any meaningful way, I think shared data is still too useful to pass up. I've written a websocket server implementation in Zig(1) and Elixir(2) (1) https://github.com/karlseguin/websocket.zig https://github.com/karlseguin/websocket.zig (2) https://github.com/karlseguin/exws https://github.com/karlseguin/exws
- newjersey 4y ago
  > Specifically, if those connections are interacting with each other in any meaningful way, I think shared data is still too useful to pass up. What does this mean? What are some scenarios where connections interact with each other? I work with dotnet. To me, every request is standalone and doesn’t need to know any other request exists. At the most, I can see doing some kind of caching where if someone does a GET /person/12345 and someone else does the same, I maybe able to do some caching. However, I don’t think this is what you meant by shared data. Did you mean like if someone does a PUT /person/12345/email hikingfan@gmail.com instead of the next get request reaching to the database, you keep it in the application memory and just use it? Or am I completely missing the point and you’re talking about near real-time stuff like calls and screen sharing?
  - wonnage 4y ago
    Many languages (e.g, NodeJS) won’t even let you share code. So you can’t really do stuff like have hundreds of threads without being very careful with the size of your application code, because each thread will get a copy.
  - winrid 4y ago
    If you need to send messages to other channels, or use shared caches, or have shared state like a game server.
  - latch 4y ago
hknmtt 4y ago
Node is a joke. It's not good for this. Check out https://github.com/panjf2000/gnet https://github.com/panjf2000/gnet, it also has some links at the end.
- Sosh101 4y ago
  Care to explain why node is "a joke"? It's often used for these kind of applications.
inglor 4y ago
We did this with Node.js and uWebSockets and it scaled easily to a few million web sockets on ~10 machines so I can confirm the stack works in practice
- prox 4y ago
  I am trying to imagine why one would need millions of web sockets :) What are the use cases here?
  - winrid 4y ago
    Millions of clients. IOT devices? Who knows.
    - ThePhysicist 4y ago
      It's unlikely you'd want to connect IOT devices to a backend using web sockets, I'd use a UDP based protocol for that, e.g. QUIC. But for web clients it makes sense.
      - winrid 4y ago
        MQTT is usually the go to protocol for IOT devices. You can do MQTT over WebSockets to help prevent issues using odd ports on home networks etc.
    - inglor 4y ago
      This was working for Peer5 (YC startup) - building a p2p CDN, these were video viewers in live events (e.g. the world cup).
- r1ch 4y ago
  We used the C++ version of uWebSockets to replace a legacy node app. We went from four fully loaded cores to about 20% of a single core and a fraction of the memory usage. It's a great library.
pritambarhate 4y ago
With Netty and Java you can easily handle 100-200k active web socket connections on a single server. It was being done 7-8 years ago. If you search you should find a few articles on this.
- winrid 4y ago
  Considering someone had a 100k+ idle connections on a raspberry pi with Java/Netty, yeah, you could get to a million today with some mid tier hardware and Linux tuning, pretty easily.
bob1029 4y ago
.NET7 and Kestrel are likely able to pull this off if properly configured. Kestrel/AspNetCore routinely shows up in the top 10 techempower web benchmarks.
- tester756 4y ago
  I'm not trusting benchmarks that I didn't fake myself. Anyway - there's a lot of "non-standard" stuff in ASP's code there.
  - bob1029 4y ago
    Can you explain how you think the techempower benchmarks are faked?
    - mardifoufs 4y ago
      I think it's referring to this? https://dusted.codes/how-fast-is-really-aspnet-core https://dusted.codes/how-fast-is-really-aspnet-core
- opendomain 4y ago
  Can you please suggest a link to show how to do this?
  - bob1029 4y ago
    https://learn.microsoft.com/en-us/aspnet/core/fundamentals/websockets?view=aspnetcore-7.0 https://learn.microsoft.com/en-us/aspnet/core/fundamentals/w...
norman784 4y ago
Node might be faster to write, but harder to maintain in the long run, also is not as reliable as Go or Rust, I personally will pick Rust because I have experience with it, but AFAIK Go has a very good reputation, the only "difference" with Rust is the GC (I mean "difference", because Go performance is not that far off from Rust, and seems also easy to write in Go than in Rust). Also IMHO it's better to have a strong typed language behind your project, if it will be big, dynamic languages and big projects tend to be a nightmare for me.
- okal 4y ago
  Would you mind unpacking how, in your view, Go/Rust/compiled strongly-typed languages lead to more *reliable* software? I can see how performance and maintainability* are sort of self-evident arguments in favour of them, but not sure how reliability could be a feature inherent to a language/runtime. * As a build/compile-time concern, using Node doesn't preclude strong-typing, so maintainability is also not a strong argument against the runtime itself, given you can use e.g. TypeScript.
  - norman784 4y ago
    I think this blog post[0] describes what level of reliability you can achieve with Rust, specifically: > In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible. For sure not everyone will be able to achieve that in their first try or when getting started, but for sure is possible, but with Node I'm not confident enough to say that, for sure if works to hack something quickly and put in online, with Rust it takes longer and there are not too many platforms yet where you can easily deploy your app. [0] https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...
winrid 4y ago
Pretty much any modern runtime (Java/Go/Node w/ native bindings) can handle that many connections per machine. You probably want to horizontally scale it with kafka or similar, but anyway, a single machine will work to start.
tmikaeld 4y ago
Check out Centrifugal: https://github.com/centrifugal/centrifugo https://github.com/centrifugal/centrifugo (Server/Admin) https://github.com/centrifugal/centrifuge https://github.com/centrifugal/centrifuge (Server core) https://github.com/centrifugal/centrifuge-js https://github.com/centrifugal/centrifuge-js (Library) It's a complete solution, including server, admin panel and client library.

deleted 4y ago

[deleted]

lawik 4y ago

Elixir is well suited to highly concurrent systems and work like this. I'm big on the whole Elixir ecosystem though so I haven't explored other options. I don't see why there would be anything stopping Go from being similarly capable as it also has a good reputation for concurrency and what I hear does preemptive scheduling. Java can probably do anything except be fun and lightweight so assuming you want to figure out the hoops to jump through. I assume it could.. Elixir can do it with the ergonomics and expressiveness of Python/Ruby. If you enjoy that level of abstraction I recommend it.

szundi 4y ago
Java is slowly absorbing the ideas from other systems and is much more fun than it was. Also versatile.
a_c 4y ago
Do you have any pointer, book preferably, in starting an exploratory Elixir project? I don't have any objective apart from giving the ecosystem a taste
- conradfr 4y ago
  If you really want a book pick one from here [0]. First one is good. Personally I think just following the official guide [1] will give you all you need to get a taste of the language and the platform and decide if you like it or not. If you were talking about websockets in particular I guess realistically most people use Phoenix Channels [2] that give you websockets in ten lines of code. [0] https://elixir-lang.org/learning.html https://elixir-lang.org/learning.html [1] https://elixir-lang.org/getting-started/introduction.html https://elixir-lang.org/getting-started/introduction.html [2] https://hexdocs.pm/phoenix/channels.html https://hexdocs.pm/phoenix/channels.html
- dns_snek 4y ago
  I can highly recommend Elixir in Action, 2nd ed. This talk by the same author is also a good introduction in video format: https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE

latch 4y ago

Honestly, what matters is (a) what you're going to be doing with those connections and (b) your hardware. As a generalization (again, really depends what you're going to be doing), I'd expect people to get a lot further with a Go or Java based implementations. Specifically, if those connections are interacting with each other in any meaningful way, I think shared data is still too useful to pass up. I've written a websocket server implementation in Zig(1) and Elixir(2) (1) https://github.com/karlseguin/websocket.zig https://github.com/karlseguin/websocket.zig (2) https://github.com/karlseguin/exws https://github.com/karlseguin/exws

newjersey 4y ago
> Specifically, if those connections are interacting with each other in any meaningful way, I think shared data is still too useful to pass up. What does this mean? What are some scenarios where connections interact with each other? I work with dotnet. To me, every request is standalone and doesn’t need to know any other request exists. At the most, I can see doing some kind of caching where if someone does a GET /person/12345 and someone else does the same, I maybe able to do some caching. However, I don’t think this is what you meant by shared data. Did you mean like if someone does a PUT /person/12345/email hikingfan@gmail.com instead of the next get request reaching to the database, you keep it in the application memory and just use it? Or am I completely missing the point and you’re talking about near real-time stuff like calls and screen sharing?
- wonnage 4y ago
  Many languages (e.g, NodeJS) won’t even let you share code. So you can’t really do stuff like have hundreds of threads without being very careful with the size of your application code, because each thread will get a copy.
- winrid 4y ago
  If you need to send messages to other channels, or use shared caches, or have shared state like a game server.
- latch 4y ago

hknmtt 4y ago

Node is a joke. It's not good for this. Check out https://github.com/panjf2000/gnet https://github.com/panjf2000/gnet, it also has some links at the end.

Sosh101 4y ago
Care to explain why node is "a joke"? It's often used for these kind of applications.

inglor 4y ago

We did this with Node.js and uWebSockets and it scaled easily to a few million web sockets on ~10 machines so I can confirm the stack works in practice

prox 4y ago
I am trying to imagine why one would need millions of web sockets :) What are the use cases here?
- winrid 4y ago
  Millions of clients. IOT devices? Who knows.
  - ThePhysicist 4y ago
    It's unlikely you'd want to connect IOT devices to a backend using web sockets, I'd use a UDP based protocol for that, e.g. QUIC. But for web clients it makes sense.
    - winrid 4y ago
      MQTT is usually the go to protocol for IOT devices. You can do MQTT over WebSockets to help prevent issues using odd ports on home networks etc.
  - inglor 4y ago
    This was working for Peer5 (YC startup) - building a p2p CDN, these were video viewers in live events (e.g. the world cup).
r1ch 4y ago
We used the C++ version of uWebSockets to replace a legacy node app. We went from four fully loaded cores to about 20% of a single core and a fraction of the memory usage. It's a great library.

pritambarhate 4y ago

With Netty and Java you can easily handle 100-200k active web socket connections on a single server. It was being done 7-8 years ago. If you search you should find a few articles on this.

winrid 4y ago
Considering someone had a 100k+ idle connections on a raspberry pi with Java/Netty, yeah, you could get to a million today with some mid tier hardware and Linux tuning, pretty easily.

bob1029 4y ago

.NET7 and Kestrel are likely able to pull this off if properly configured. Kestrel/AspNetCore routinely shows up in the top 10 techempower web benchmarks.

tester756 4y ago
I'm not trusting benchmarks that I didn't fake myself. Anyway - there's a lot of "non-standard" stuff in ASP's code there.
- bob1029 4y ago
  Can you explain how you think the techempower benchmarks are faked?
  - mardifoufs 4y ago
    I think it's referring to this? https://dusted.codes/how-fast-is-really-aspnet-core https://dusted.codes/how-fast-is-really-aspnet-core
opendomain 4y ago
Can you please suggest a link to show how to do this?
- bob1029 4y ago
  https://learn.microsoft.com/en-us/aspnet/core/fundamentals/websockets?view=aspnetcore-7.0 https://learn.microsoft.com/en-us/aspnet/core/fundamentals/w...

norman784 4y ago

Node might be faster to write, but harder to maintain in the long run, also is not as reliable as Go or Rust, I personally will pick Rust because I have experience with it, but AFAIK Go has a very good reputation, the only "difference" with Rust is the GC (I mean "difference", because Go performance is not that far off from Rust, and seems also easy to write in Go than in Rust). Also IMHO it's better to have a strong typed language behind your project, if it will be big, dynamic languages and big projects tend to be a nightmare for me.

okal 4y ago
Would you mind unpacking how, in your view, Go/Rust/compiled strongly-typed languages lead to more *reliable* software? I can see how performance and maintainability* are sort of self-evident arguments in favour of them, but not sure how reliability could be a feature inherent to a language/runtime. * As a build/compile-time concern, using Node doesn't preclude strong-typing, so maintainability is also not a strong argument against the runtime itself, given you can use e.g. TypeScript.
- norman784 4y ago
  I think this blog post[0] describes what level of reliability you can achieve with Rust, specifically: > In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible. For sure not everyone will be able to achieve that in their first try or when getting started, but for sure is possible, but with Node I'm not confident enough to say that, for sure if works to hack something quickly and put in online, with Rust it takes longer and there are not too many platforms yet where you can easily deploy your app. [0] https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/ https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...

winrid 4y ago

Pretty much any modern runtime (Java/Go/Node w/ native bindings) can handle that many connections per machine. You probably want to horizontally scale it with kafka or similar, but anyway, a single machine will work to start.

tmikaeld 4y ago

Check out Centrifugal: https://github.com/centrifugal/centrifugo https://github.com/centrifugal/centrifugo (Server/Admin) https://github.com/centrifugal/centrifuge https://github.com/centrifugal/centrifuge (Server core) https://github.com/centrifugal/centrifuge-js https://github.com/centrifugal/centrifuge-js (Library) It's a complete solution, including server, admin panel and client library.