4 ms·

Have any Node.js/websocket implementations scaled on this order of magnitude? Would like to do a read-up of any challenges faced.

wrong_variable 8y ago
I have given up on NodeJS and moved solely to Erlang for long term projects. I still love NodeJS ! but once you learn Erlang its hard to go back, it reminds me of PG's having a higher bird eye view. NodeJS, Python, Haskell (!), PHP, C - all belong to the same class of coding style with the same type of problems. You use Erlang not because of playing the testosterone game of nominal performance - but because you really want some guarantee. ( Money should be not a problem since the world seems flush with cash - if your manager is complaining its because he wants his bonus to be higher. )
- tehsauce 8y ago
  Care to elaborate what these types of problems are and how erland fixes them?
  - wrong_variable 8y ago
    Sure ! Think about how prinf / console.log / print .... works in traditional settings. - How would you make it so that printf doesn't crash your entire program if the console hangs. - how would you isolate a single codebase's IO operations ? - How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Haskell sort of tries to answer these questions but I am not sure how its going go work out for them, in Erlang's process based universe all those questions have been answered already !
    - stingraycharles 8y ago
      Sounds like reasons to prefer FP over OOP, not necessarily Erlang. I do really appreciate Erlang / Elixir (have contributed several libraries), but the problems you describe are not uniquely solved by Erlang. Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager).
      - wrong_variable 8y ago
        > but the problems you describe are not uniquely solved by Erlang I agree, I do not any Scala / Akka experience so I cannot argue for or against due to ignorance. But as you say at least the platform / language addresses these concerns.
      - innocentoldguy 8y ago
        Where I live, it is easier to find local Elixir talent than Scala. There was a big Ruby community here and many of them have moved to Elixir.
      - pdimitar 8y ago
        > Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager). They solve a subset of the problems that Erlang's OTP solves. They don't have the entire package.
      - yawaramin 8y ago
        You should really read jhgg's response describing the BEAM runtime's pre-emptive scheduling of its lightweight processes: https://news.ycombinator.com/item?id=19241194 https://news.ycombinator.com/item?id=19241194 . No other runtime can do that for you, not even the JVM.
    - protomikron 8y ago
      How is this related to programming languages - I think these are platform issues and design choices. How does Erlang help there (curiously interested)?
      - wrong_variable 8y ago
        Erlang's language syntax is kinda awful but that is besides the point. Issues I have raised are 100x harder to address then some syntax.
      - pdimitar 8y ago
        You should read on the fault tolerance guarantees of Erlang OTP. It's really hard to be summarized. Supervision trees, "let it crash and get safely restarted", super-mini-processes that have preemptive scheduling built-in, soft real-time... TL;DR: No, no other language or framework in the world has the primitives that Erlang and Elixir have. People on HN and Reddit really love acting non-impressed and claiming the pain points are easily solved in other languages. My 17 years of career say this is not true at all.
    - sonnyblarney 8y ago
      Those are good points, but does node fail in the scale requirement? I mean supposing we can get past console printing issues?
      - wrong_variable 8y ago
        Node is awesome for fast and quick projects / cmd tools. - For example I have a project involving smart electrical inverters, I need some guarantees regarding crash handling / low latency. Not a lot of scaling issues. - With scaling involving NodeJS, I consistency had issues with exploding RAM and crashes - unable to isolate part of codebase. I struggled a lot to fix these problems, so while searching for a solution I came across Erlang and haven't looked back.
    - Thaxll 8y ago
      The problem with your reasoning is you're trying to make your program very resilient when it shouldn't be, let the underlying infrastructure deal with that, if it crashes it will be re-created. BEAM and Erlang are the wrong solution for those problems because not language neutral. I can make my nodejs / PHP app as resiliant as Erlang using Kubernetes.
      - tigershark 8y ago
        Seriously? You’d rather let a whole container crash and respin and start all the services that will rebind again hoping that everything goes as planned in the restart? I’d rather have a language in which you can’t have null pointer exceptions and you handle everything that can possibly go wrong rather than write sloppy code that causes entire processes to crash, honestly.
        Thaxll 8y ago
        But Erlang is a dynamic language...
      - zzzcpan 8y ago
        Infrastructure can never get to any reasonable level of resilience alone without programs being designed for resilience and such ignorance will likely lead to nasty stuff, like cascading failures, because of massive load differences between normal operations and attempts to handle errors by recreating entire services. But Kubernetes itself is not known for its resilience, on the contrary, its reliability reputation is pretty bad at this point, unlike that of Erlang. Design for resilience is not a small thing. I suggest to read Joe Armstrong's "Making reliable distributed systems in the presence of software errors" [1] as a starting point on this. [1] http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf
        Thaxll 8y ago
        I'm not saying that you should not add resilience in your program, but that OP claims about "what if printf crashes" well that's it it will crash deal with it. You should ensure your platform will recover if you program crash not try everything possible to not make your program not crash. "Pet vs Cattle" The thing I don't like about Erlang is about the runtime that is mixed between code and infra which I think is not a good idea, it was designed before we made progress with HA platform like Kubernetes. https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md https://github.com/kubernetes/community/blob/master/sig-scal... It should be separated and it's what pretty much everyone is doing nowdays.
    - jondubois 8y ago
      >> How would you make it so that printf doesn't crash your entire program if the console hangs. Why would the console hang? If that happened, it would signal a major issue at the OS level and probably not related to your application (unless you're trying to log an extremely massive string; which is a bad idea and you'd probably already have run out of memory before that could happen). I have never seen the console/stdout hanging in production and I've built some pretty high-traffic distributed systems with Node.js. >> how would you isolate a single codebase's IO operations ? What sort of IO operations are we talking about? Network, Disk? There are many ways to inspect different kinds of IO operations. The Node.js ecosystem offers a large number of modules which would let you achieve that. >> How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Node.js is perfect for running on Kubernetes. There are many K8s dashboards and tools which allow you to browse and aggregate logs from thousands of machines with very little effort. I don't see how this point has anything to do with Erlang specifically. A language-agnostic container orchestrator like Kubernetes is the best way to go over a tool which only works with a specific language. There are Node.js frameworks which offer kubetnetes .yaml files and CLI tools which allow you to deploy a highly scalable cluster to K8s in a few minutes.
      - jhgg 8y ago
        Can we all take a step back here and for a second realize that Node.JS is based upon a language/event model that was purpose-built and designed to handle client side browser operations, and has since been expanded into the server space. And Erlang/OTP/BEAM was built and designed for running reliable soft real-time distributed telecom systems. By default (and unless you go out of your way w/ web-workers) the javascript event-loop (and thus nodes event loop) is single threaded. To work around this, you can bind many node processes to a given port to load balance requests (SO_REUSEADDR, anyone?) - or simply run many smaller instances of node (perhaps in a bunch of containers) where traffic ingresses in via some form of load balancer. The load balancing problem is unavoidable, and you will definitely need the same if you want to send requests to multiple BEAM nodes. However, BEAM can schedule your work across all the cores you give it. But let's talk about work for a second, and about a very special thing that the BEAM VM gives you, that other runtimes (whether it be node, JVM, golang's, etc...) aside from the actual operating system of your computer does not. And that's specifically preemptive scheduling. Suppose you have a single core computer that's running Linux, and you have a process that is sitting there busy looping. Let's say we just make a simple script that does nothing infinitely in a loop. Does your computer grind to a halt? Most likely, no. You can still probably use your terminal, move your mouse, operate your web browser, etc... You can thank pre-emptive scheduling for that. The OS suspends the process to allow other processes to do work - hopefully in a fair manner (on linux, the CFS (aptly named Completely Fair Scheduler) does this). Now let's say you have a single node process serving requests. Let's say that a specific kind of requests requires 250ms of CPU time to compute - and does not explicitly yield back to the event loop. (You can imagine doing some processing of input data, deserialization, serialization, aggregation, etc...). During this computation, nothing else within the node process can progress. This means that requests that may not take a lot of time to compute now have to wait 250ms to be processed. Generally, I see node deployments not having single request/response request handling, but rather many concurrent requests/responses being handled at any given time, using promises/callbacks to allow the event loop to progress while waiting on IO from something else. During the periods of expensive computation from a given request handler, the entire event loop is stalled, and the response time percentiles of your requests spike. A pathological case would be something like `setTimeout(() => while(1) { }, 1000)` deadlocking your entire node process after a whole second, as the loop does not yield back to the event loop. In BEAM, this does not exist. Processes are scheduled and pre-empted - to allow for fair utilization of the underlying computation resources (very much like how your OS does it.) This means that a computationally intensive process does not stall the event loop for all other processes, meaning that your response times and percentiles remain low for all other work within the system. Now of course, you could hand-craft your javascript code to explicitly yield to the scheduler every so often, but that's a lot of work that you as a programmer are now doing that your runtime could be doing for you, and if you forget to do it, could be catastrophic to the performance of your soft-realtime system. This is only one of the many benefits that OTP/BEAM provide over other runtimes. But one compelling enough for Discord as a company to bet on it. For a given service, we run entirely homogeneous infrastructure. We do not need to allocate or dedicate special resources to our largest servers (100k CCU/350k members), and instead can run and schedule it alongside the millions of other small servers that exist on Discord - all without negatively impacting the performance, percentiles, or soft-realtime guarantees of your chat with a few of your friends.
Thaxll 8y ago
It's easy to scale to 5m websockets with almost any languages, the real question is what those connections are doing, looking at Discord we can assume that a large % of users are idle for example. My Discord client is always on but I do nothing on it, I'm pretty sure it's the case for most people since it starts at windows startup. I actually think that Elixir / Erlang are actually not great for those kind of problems because they're slow language and consume a lot of memory. They allow you to do easy message passing + horizontal scaling but the runtime is inefficient. Java / C# net core / Rust / C++ / Go are much faster than Elixir. ( if you actually read on it you'll see that they use a lot of C / C++ / Rust to make it fast which is not something you need to with the above languages ). And for deployment / scaling just use Kubernetes or equivalent, better than BEAM trust me.
- jhgg 8y ago
  Since this blog post, we are well over 5m ccu (I don't think we have shared our current peaks publicly yet). We do egress 4m-6m websocket messages/sec through our system though at peak. So it's a lot more than just holding onto idle websockets :P The power of BEAM is that although the performance may not be the best, throughput and response time is consistently low throughout the system - allowing no single process (or actor) from monopolizing resources of the system. When you use a homogenous server configuration like we do, this makes a lot of sense. Our largest guilds (100k ccu, 350k members) are on the same nodes as all of our other servers. And when they're busy, the small guilds notice no performance degradation. Other chat products out there (that I hear use the JVM for their real time stuff) have to spin up dedicated resources to hosting their larger servers/clients - and even then cannot handle servers with as many users or concurrents that we can. Every single discord server runs in a homogenous cluster, without special dedicated resources for our largest instances.
tigershark 8y ago
Am I the only one thinking that node.js is maybe not the right choice for global-wide services?
- tomc1985 8y ago
  node ain't the right choice for anything
jondubois 8y ago
5M concurrent users is a pretty big number. There are plenty of Node.js systems/frameworks which could scale to that size with minimal effort. SocketCluster has been used in production to service hundreds of thousands of concurrent users and it can handle millions. I've had one report of a chat system (adult industry) which could handle 250K concurrent users using only two large servers. Also, I once did some consulting work for a popular cryptocurrency trading platform which handled tens of thousands of concurrent users/trading bots (with very high frequency of messages). After I was done, that company didn't talk to me for 6 months straight; it turns out that they hadn't had any issues with their pub/sub cluster since. Unfortunately SocketCluster doesn't get discussed very often among influential circles. I have no idea why because the feedback I get from users is essentially 100% positive. I guess Node.js doesn't get much hype these days.

wrong_variable 8y ago

I have given up on NodeJS and moved solely to Erlang for long term projects. I still love NodeJS ! but once you learn Erlang its hard to go back, it reminds me of PG's having a higher bird eye view. NodeJS, Python, Haskell (!), PHP, C - all belong to the same class of coding style with the same type of problems. You use Erlang not because of playing the testosterone game of nominal performance - but because you really want some guarantee. ( Money should be not a problem since the world seems flush with cash - if your manager is complaining its because he wants his bonus to be higher. )

tehsauce 8y ago
Care to elaborate what these types of problems are and how erland fixes them?
- wrong_variable 8y ago
  Sure ! Think about how prinf / console.log / print .... works in traditional settings. - How would you make it so that printf doesn't crash your entire program if the console hangs. - how would you isolate a single codebase's IO operations ? - How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Haskell sort of tries to answer these questions but I am not sure how its going go work out for them, in Erlang's process based universe all those questions have been answered already !
  - stingraycharles 8y ago
    Sounds like reasons to prefer FP over OOP, not necessarily Erlang. I do really appreciate Erlang / Elixir (have contributed several libraries), but the problems you describe are not uniquely solved by Erlang. Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager).
    - wrong_variable 8y ago
      > but the problems you describe are not uniquely solved by Erlang I agree, I do not any Scala / Akka experience so I cannot argue for or against due to ignorance. But as you say at least the platform / language addresses these concerns.
    - innocentoldguy 8y ago
      Where I live, it is easier to find local Elixir talent than Scala. There was a big Ruby community here and many of them have moved to Elixir.
    - pdimitar 8y ago
      > Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager). They solve a subset of the problems that Erlang's OTP solves. They don't have the entire package.
    - yawaramin 8y ago
      You should really read jhgg's response describing the BEAM runtime's pre-emptive scheduling of its lightweight processes: https://news.ycombinator.com/item?id=19241194 https://news.ycombinator.com/item?id=19241194 . No other runtime can do that for you, not even the JVM.
  - protomikron 8y ago
    How is this related to programming languages - I think these are platform issues and design choices. How does Erlang help there (curiously interested)?
    - wrong_variable 8y ago
      Erlang's language syntax is kinda awful but that is besides the point. Issues I have raised are 100x harder to address then some syntax.
    - pdimitar 8y ago
      You should read on the fault tolerance guarantees of Erlang OTP. It's really hard to be summarized. Supervision trees, "let it crash and get safely restarted", super-mini-processes that have preemptive scheduling built-in, soft real-time... TL;DR: No, no other language or framework in the world has the primitives that Erlang and Elixir have. People on HN and Reddit really love acting non-impressed and claiming the pain points are easily solved in other languages. My 17 years of career say this is not true at all.
  - sonnyblarney 8y ago
    Those are good points, but does node fail in the scale requirement? I mean supposing we can get past console printing issues?
    - wrong_variable 8y ago
      Node is awesome for fast and quick projects / cmd tools. - For example I have a project involving smart electrical inverters, I need some guarantees regarding crash handling / low latency. Not a lot of scaling issues. - With scaling involving NodeJS, I consistency had issues with exploding RAM and crashes - unable to isolate part of codebase. I struggled a lot to fix these problems, so while searching for a solution I came across Erlang and haven't looked back.
  - Thaxll 8y ago
    The problem with your reasoning is you're trying to make your program very resilient when it shouldn't be, let the underlying infrastructure deal with that, if it crashes it will be re-created. BEAM and Erlang are the wrong solution for those problems because not language neutral. I can make my nodejs / PHP app as resiliant as Erlang using Kubernetes.
    - tigershark 8y ago
      Seriously? You’d rather let a whole container crash and respin and start all the services that will rebind again hoping that everything goes as planned in the restart? I’d rather have a language in which you can’t have null pointer exceptions and you handle everything that can possibly go wrong rather than write sloppy code that causes entire processes to crash, honestly.
      - Thaxll 8y ago
        But Erlang is a dynamic language...
    - zzzcpan 8y ago
      Infrastructure can never get to any reasonable level of resilience alone without programs being designed for resilience and such ignorance will likely lead to nasty stuff, like cascading failures, because of massive load differences between normal operations and attempts to handle errors by recreating entire services. But Kubernetes itself is not known for its resilience, on the contrary, its reliability reputation is pretty bad at this point, unlike that of Erlang. Design for resilience is not a small thing. I suggest to read Joe Armstrong's "Making reliable distributed systems in the presence of software errors" [1] as a starting point on this. [1] http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf
      - Thaxll 8y ago
        I'm not saying that you should not add resilience in your program, but that OP claims about "what if printf crashes" well that's it it will crash deal with it. You should ensure your platform will recover if you program crash not try everything possible to not make your program not crash. "Pet vs Cattle" The thing I don't like about Erlang is about the runtime that is mixed between code and infra which I think is not a good idea, it was designed before we made progress with HA platform like Kubernetes. https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md https://github.com/kubernetes/community/blob/master/sig-scal... It should be separated and it's what pretty much everyone is doing nowdays.
  - jondubois 8y ago
    >> How would you make it so that printf doesn't crash your entire program if the console hangs. Why would the console hang? If that happened, it would signal a major issue at the OS level and probably not related to your application (unless you're trying to log an extremely massive string; which is a bad idea and you'd probably already have run out of memory before that could happen). I have never seen the console/stdout hanging in production and I've built some pretty high-traffic distributed systems with Node.js. >> how would you isolate a single codebase's IO operations ? What sort of IO operations are we talking about? Network, Disk? There are many ways to inspect different kinds of IO operations. The Node.js ecosystem offers a large number of modules which would let you achieve that. >> How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Node.js is perfect for running on Kubernetes. There are many K8s dashboards and tools which allow you to browse and aggregate logs from thousands of machines with very little effort. I don't see how this point has anything to do with Erlang specifically. A language-agnostic container orchestrator like Kubernetes is the best way to go over a tool which only works with a specific language. There are Node.js frameworks which offer kubetnetes .yaml files and CLI tools which allow you to deploy a highly scalable cluster to K8s in a few minutes.
    - jhgg 8y ago
      Can we all take a step back here and for a second realize that Node.JS is based upon a language/event model that was purpose-built and designed to handle client side browser operations, and has since been expanded into the server space. And Erlang/OTP/BEAM was built and designed for running reliable soft real-time distributed telecom systems. By default (and unless you go out of your way w/ web-workers) the javascript event-loop (and thus nodes event loop) is single threaded. To work around this, you can bind many node processes to a given port to load balance requests (SO_REUSEADDR, anyone?) - or simply run many smaller instances of node (perhaps in a bunch of containers) where traffic ingresses in via some form of load balancer. The load balancing problem is unavoidable, and you will definitely need the same if you want to send requests to multiple BEAM nodes. However, BEAM can schedule your work across all the cores you give it. But let's talk about work for a second, and about a very special thing that the BEAM VM gives you, that other runtimes (whether it be node, JVM, golang's, etc...) aside from the actual operating system of your computer does not. And that's specifically preemptive scheduling. Suppose you have a single core computer that's running Linux, and you have a process that is sitting there busy looping. Let's say we just make a simple script that does nothing infinitely in a loop. Does your computer grind to a halt? Most likely, no. You can still probably use your terminal, move your mouse, operate your web browser, etc... You can thank pre-emptive scheduling for that. The OS suspends the process to allow other processes to do work - hopefully in a fair manner (on linux, the CFS (aptly named Completely Fair Scheduler) does this). Now let's say you have a single node process serving requests. Let's say that a specific kind of requests requires 250ms of CPU time to compute - and does not explicitly yield back to the event loop. (You can imagine doing some processing of input data, deserialization, serialization, aggregation, etc...). During this computation, nothing else within the node process can progress. This means that requests that may not take a lot of time to compute now have to wait 250ms to be processed. Generally, I see node deployments not having single request/response request handling, but rather many concurrent requests/responses being handled at any given time, using promises/callbacks to allow the event loop to progress while waiting on IO from something else. During the periods of expensive computation from a given request handler, the entire event loop is stalled, and the response time percentiles of your requests spike. A pathological case would be something like `setTimeout(() => while(1) { }, 1000)` deadlocking your entire node process after a whole second, as the loop does not yield back to the event loop. In BEAM, this does not exist. Processes are scheduled and pre-empted - to allow for fair utilization of the underlying computation resources (very much like how your OS does it.) This means that a computationally intensive process does not stall the event loop for all other processes, meaning that your response times and percentiles remain low for all other work within the system. Now of course, you could hand-craft your javascript code to explicitly yield to the scheduler every so often, but that's a lot of work that you as a programmer are now doing that your runtime could be doing for you, and if you forget to do it, could be catastrophic to the performance of your soft-realtime system. This is only one of the many benefits that OTP/BEAM provide over other runtimes. But one compelling enough for Discord as a company to bet on it. For a given service, we run entirely homogeneous infrastructure. We do not need to allocate or dedicate special resources to our largest servers (100k CCU/350k members), and instead can run and schedule it alongside the millions of other small servers that exist on Discord - all without negatively impacting the performance, percentiles, or soft-realtime guarantees of your chat with a few of your friends.

Thaxll 8y ago

It's easy to scale to 5m websockets with almost any languages, the real question is what those connections are doing, looking at Discord we can assume that a large % of users are idle for example. My Discord client is always on but I do nothing on it, I'm pretty sure it's the case for most people since it starts at windows startup. I actually think that Elixir / Erlang are actually not great for those kind of problems because they're slow language and consume a lot of memory. They allow you to do easy message passing + horizontal scaling but the runtime is inefficient. Java / C# net core / Rust / C++ / Go are much faster than Elixir. ( if you actually read on it you'll see that they use a lot of C / C++ / Rust to make it fast which is not something you need to with the above languages ). And for deployment / scaling just use Kubernetes or equivalent, better than BEAM trust me.

jhgg 8y ago
Since this blog post, we are well over 5m ccu (I don't think we have shared our current peaks publicly yet). We do egress 4m-6m websocket messages/sec through our system though at peak. So it's a lot more than just holding onto idle websockets :P The power of BEAM is that although the performance may not be the best, throughput and response time is consistently low throughout the system - allowing no single process (or actor) from monopolizing resources of the system. When you use a homogenous server configuration like we do, this makes a lot of sense. Our largest guilds (100k ccu, 350k members) are on the same nodes as all of our other servers. And when they're busy, the small guilds notice no performance degradation. Other chat products out there (that I hear use the JVM for their real time stuff) have to spin up dedicated resources to hosting their larger servers/clients - and even then cannot handle servers with as many users or concurrents that we can. Every single discord server runs in a homogenous cluster, without special dedicated resources for our largest instances.

tigershark 8y ago

Am I the only one thinking that node.js is maybe not the right choice for global-wide services?

tomc1985 8y ago
node ain't the right choice for anything

jondubois 8y ago

5M concurrent users is a pretty big number. There are plenty of Node.js systems/frameworks which could scale to that size with minimal effort. SocketCluster has been used in production to service hundreds of thousands of concurrent users and it can handle millions. I've had one report of a chat system (adult industry) which could handle 250K concurrent users using only two large servers. Also, I once did some consulting work for a popular cryptocurrency trading platform which handled tens of thousands of concurrent users/trading bots (with very high frequency of messages). After I was done, that company didn't talk to me for 6 months straight; it turns out that they hadn't had any issues with their pub/sub cluster since. Unfortunately SocketCluster doesn't get discussed very often among influential circles. I have no idea why because the feedback I get from users is essentially 100% positive. I guess Node.js doesn't get much hype these days.