11 ms·
600k concurrent websocket connections on AWS using Node.js (2015)
- RossM 7y agoInteresting details; it would be nice to see how those ulimit/networking numbers were arrived at. The title should have [2015].
- The_rationalist 7y agoWould be nice to rebench with the new websocketstream API! https://github.com/ricea/websocketstream-explainer/blob/master/README.md https://github.com/ricea/websocketstream-explainer/blob/mast...
- cpursley 7y agoSo 1/4 of what Elixir/Erlang can handle, but more difficult and less reliable: https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://phoenixframework.org/blog/the-road-to-2-million-webs...
- conradfr 7y agoBoth articles from 2015.
- SEMW 7y agoGary was using a much more powerful instance (both in cpu and memory) that this post uses when he reached 2 million with Phoenix. But Phoenix is also doing quite a lot more per connection than node. Basically for multiple reasons they're incomparable experiments, there's really no basis here to draw conclusions like '1/4 of what Elixir/Erlang can handle' from.
- truth_seeker 7y agoNot fair ! You are comparing "4 CPUs and 15GB of memory" for NodeJS with "40 CPUs and 128 GB of memory" for Elixir/Phoenix
- cpursley 7y agoOk, here's a better comparison: "Websocket Shootout: Clojure, C++, Elixir, Go, NodeJS, and Ruby" https://hashrocket.com/blog/posts/websocket-shootout https://hashrocket.com/blog/posts/websocket-shootout
- truth_seeker 7y ago> NodeJS performance is hampered by its single-threaded architecture, but given that fact it performs very well. The NodeJS server was the smallest overall in lines of code and extremely quick and easy to write. Javascript is the lingua franca for web development, so NodeJS is probably the easiest platform for which to hire or train developers. Do you actually read the blog post before making the claim ? The author is not using "Cluster" module to scale connections across CPUs with "Sticky-session" library. Also, neither he mentioned the runtime flags with which the launched all VMs in comparison nor he mentioned the optimization at OS level.
- deorder 7y agoIt seems that the author actually used the "cluster" module, but just does not mention it in the article: https://github.com/hashrocket/websocket-shootout/blob/master/js/run-cluster.js https://github.com/hashrocket/websocket-shootout/blob/master... I expected all benchmarks that use the (linux) kernel for I/O events would perform similarly. The code is so small that the interpreter overhead should not degrade the performance too much. May be the JSON parsing / serialization?
- truth_seeker 7y agoSo you think that "sticky-session" is not important ? Can you tell why ?
- folkhack 7y ago"but more difficult" sounds pretty subjective... I can get behind the "less reliable" comment because Erlang is rock solid. Sorry to be that guy, but I can find a TON more people capable of supporting ECMA vs. Erlang and/or Elixr. I say this as a huge fanboy of both. I know I'll get rolled on HN for saying this because we all drink the optimization Koolaid but I feel this is worth mentioning.
- rlander 7y agoYou have a fair point, but if by "more difficult" the grandparent meant "more difficult to maintain", I'd have to agree with him. The failure modes of processes running in the BEAM are much more transparent and understandable than the usual ad-hoc JS nodejs server.
- folkhack 7y agoIMO they're on the same level but I think it's just my style of how I manage my Node.js applications down my try/catch blocks, exception handling, and primarily process isolation. > usual ad-hoc JS nodejs server Yep - that's the key why I feel they're on the same level. I heavily rely on the cluster module and IPC to get my work done which gives me true process isolation/safety however I admit it's more code in Node.js to make it rock-solid ;) Anecdotal, but I find that the typical Node.js developer is 100% "single-process" with weird supervisors like PM2. I also see over-engineered fleets of EC2 instances behind ELB/ALBs with health checking kicking bad instances out + a way to replace them... this is also stupid common in Docker/docker-compose/K8s as well. For the record I think that development pattern described in the above paragraph is total crap and I put it on the same level of idiots giving modern PHP7 a bad name. Devs who practice like this completely blew the content from their CS100/200 level courses out their butt (yeah yeah I say this as a dropout myself whatever ha).
- joaojeronimo 7y agoSome dude in 2012 did 1 million active connections with whatever node.js and v8 versions we had at that time :) http://blog.caustik.com/2012/08/19/node-js-w1m-concurrent-connections/ http://blog.caustik.com/2012/08/19/node-js-w1m-concurrent-co... Edit: I wasn't aware but there was some build-up to it: - 100k connections: http://blog.caustik.com/2012/04/08/scaling-node-js-to-100k-concurrent-connections/ http://blog.caustik.com/2012/04/08/scaling-node-js-to-100k-c... - 250k connections: http://blog.caustik.com/2012/04/10/node-js-w250k-concurrent-connections/ http://blog.caustik.com/2012/04/10/node-js-w250k-concurrent-...
- t0astbread 7y agoWhat do you need this many connections for?
- IshKebab 7y agoLots of people using your site.
- esnard 7y agoOne possible use is for high traffic websites. For example, StackExchange peaked at 500k concurrent websocket connections back in 2016. [0] [0] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/#websockets-httpsgithubcomstackexchangenetgain https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
- zaphirplane 7y agoSurely not on one endpoint
- littlecranky67 7y agoMaybe just a couple. If you google the StackExchange Hardware Setup, you will find they use surprisingly little Hardware. Back in 2013, they could go with 2 single webservers not accounting for failover reduncancy [0]. They have upgraded since, but still, if you consider the amount of hits they get, they run a pretty humble hardware [1]. One of the arguments I make when people say microservices + cloud built to the core is the only way to scale - clearly a cleverly architected approach can save you lots on hardware/hosting money. [0]: https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/#core-hardware https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-... [1]: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
- deanward81 7y agoWe’re spread over 9 servers but only due to ephemeral port and handle exhaustion issues. Each server is fronted by 4 HAProxy frontends that each handle ~18k connections. Since Nick’s post we’ve moved from StackExchange.NetGain to the managed websocket implementation in .NET Core 3 using Kestrel and libuv. That sits at around 2.3GB RAM and 0.4% CPU. Memory could be better (it used to be < 1GB with NetGain) but improvements would likely come from tweaks to how we configure GC under .NET Core which we haven’t really investigated yet. We could run on far fewer machines but we have 9 sitting there serving traffic for the sites themselves so no harm in spreading the load a little!
- buboard 7y agoWhy is this specific to AWS ?
- mping 7y agoYeah, we used nginx-push-stream-module[1] to support 1M connections with lower boxes. Websocket-as-a-service. Really cool module. Was a realtime project for a live TV contest where people could participate with their phones. [1] https://github.com/wandenberg/nginx-push-stream-module https://github.com/wandenberg/nginx-push-stream-module
- sciurus 7y agoA few pieces of advice based on running https://github.com/mozilla-services/autopush-rs https://github.com/mozilla-services/autopush-rs, which handles tens of millions of concurrent connections across a fleet of small EC2 instances. 1) Consider not running the largest instance you need to handle your workload, but instead distributing it across smaller instances. This allows for progressive rollout to test new versions, reduces the thundering herd when you restart or replace an instance, etc. 2) Don't set up security group rules that limit what addresses can connect to your websocket port. As soon as you do that connection tracking kicks in and you'll hit undocumented hard limits on the number of established connections to an instance. These limits vary based on the instance size and can easily become your bottleneck. 3) Beware of ELBs. Under the hood an ELB is made of multiple load balancers and is supposed to scale out when those load balancers hit capacity. A single load balancer can only handle a certain number of concurrent connections. In my experience ELBs don't automatically scale our when that limit is reached. You need AWS support to manually do that for you. At a certain traffic level, expect support to tell you to create multiple ELBs and distribute traffic across them yourself. ALBs or NLBs may handle this better; I'm not sure. If possible design your system to distribute connections itself instead of requiring a load balancer. 2 and 3 are frustrating because they happen at a layer of EC2 that you have little visibility into. The best way to avoid problems is to test everything at the expected real user load. In our case, when we were planning a change that would dramatically increase the number of clients connecting and doing real work, we first used our experimentation system to have a set of clients establish a dummy connection, then gradually ramped up that number of clients in the experiment as we worked through issues.
- anildigital 7y agoI don't get point of using Node.js when compared to something like Elixir. Elixir's Phoenix can handle more numbers of concurrent connections as well as provide reliability with better programming abstractions, distribution, pretty good language.
- np_tedious 7y agoI don't understand people eating hot dogs. Hamburgers have...
- runj__ 7y agoBecause javascript is one of the most popular programming languages in the world and the pool of Elixir programmers is almost non-existent?
- anildigital 7y agoFrom this https://www.ycombinator.com/topcompanies/ https://www.ycombinator.com/topcompanies/ Brex, Podium, PagerDuty use Elixir. They are in top YC companies, not a surprise.
- IggleSniggle 7y agoThanks for pointing those out. I would really like to switch to Elixir in my day job.
- truth_seeker 7y agoThere is more performant web-socket implementation than the one mentioned in the blog. It can handle 6X more connections and much less memory https://github.com/uNetworking/uWebSockets.js https://github.com/uNetworking/uWebSockets.js EDIT: Note that the blog post is from 2015. There are many optimization (Ignition and TurboFan pipeline) has been done in V8 since then, especially offloading GC activity to separate thread than NodeJS Main thread.
- gbuk2013 7y agoUnfortunately it is written by someone whose technical ability far exceeds his people skills, which are essential for a library module that developers can safely depend on. This is doubly unfortunate because I very much share his views on bloated frameworks etc. :(
- Klathmon 7y agoYeah that's the dev who pushed a purposefully broken version of their library to npm because he was upset that they wouldn't let him delete it.
- 1drr 7y agoHow do you test something like this internally?
- verttii 7y agoLoad testing tools like Tsung.
- mcintyre1994 7y agoI thought this was going to be about AWS-managed websockets using API Gateway. I've been using that at a really small scale and it's got a great API but other than almost certainly being much more expensive than the EC2 machine used here I wonder how well it works with that sort of scale.
- throwaway_bad 7y agoDoes anyone have experience doing the same on GCP? In particular right now I am trying to add live reloading to my App Engine Standard app but Standard doesn't support long lived connections (so no websockets) and App Engine Flexible seems like it will be pricy. I think I can set up a single separate websocket instance which is only responsible for doing listen/notify on postgres and telling the client when it's time to refetch from the main webserver again. Does this sound approximately workable? Will I actually be able to reach the connection numbers like in this article?
- m3kw9 7y agoBut once they all start doing stuff then what happens? 600k is more of a function of memory right?
- choffee 7y agoI wonder how it would compare in terms of cost with doing it via the api-gateway. It would depend on how your app and user base scales and what the sockets are being used for I suppose. https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api-overview.html https://docs.aws.amazon.com/apigateway/latest/developerguide...
- iends 7y agoApi gateway is very expensive once you get sustained load. EC2 much cheaper.
- desireco42 7y agoThere is that Phoenix thing when they famously did 2 Million. https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://phoenixframework.org/blog/the-road-to-2-million-webs... I agree with the suggestion that smaller instances that can be scaled is not a bad idea.
- axismundi 7y agoDoes anyone have a more recent experience? I currently use socket.io 2.2 with node v10.16, no v8 tweaks in a docker container. At ~1000 sockets, sometimes the server receives spikes of 8000 HTTP reqs/sec, which it has to distribute to the websockets, up to 100 msgs/sec, ~1kb/msg to each socket. These spikes are making the server unstable, socket.io switches most of the clients from websockets to xhr polling.
- dgelks 7y agoI found this article very useful to resolve connections moving to xhr polling on our Node.js 10.16 server https://medium.com/@k1d_bl4ck/a-quick-story-about-node-js-socket-io-and-the-linux-tcp-stack-bf1e8318b20e https://medium.com/@k1d_bl4ck/a-quick-story-about-node-js-so...
- iends 7y agoI don't have any experiments to share, but you can go father if you stop using socket.io, but I guess you need something to deal with long polling. You should consider tweaking --max_old_space_size, we got a lot of mileage giving node more memory.
- axismundi 7y agoI make heavy use of socket.io chatrooms and have to support older browsers. Haven't found any better solution yet, but I'll try the tweak, thanks. I'm also looking into load balancing via haproxy or nginx, since I need less than 10k concurrent clients.
- truth_seeker 7y agoHave you tried "sticky-cluster" ? https://github.com/uqee/sticky-cluster https://github.com/uqee/sticky-cluster
- ArchReaper 7y agoWhy are websites still hijacking scroll behavior in 2019? I can't even take the article seriously with my scrolling bouncing and glitching all over the place.
- verttii 7y agoSo what's the point of attempting to max out the count of idle connections on some cloud engine?
- ArtWomb 7y agoLots of wisdom on this page ;) Just want to add. Real-world, often the predominant use case is not optimizing for "max-conns". But <100,000 concurrent users, who instead need to be connected for a very long time. In this instance, I've found Caddy's websocket directive, inspired by Websocketd, to be quite robust and elegant. It's just a process per conn. Handling stdin, stout style messaging ;)
- jjtheblunt 7y agoGenuinely missing the point question: isn't the number of concurrent socket (websocket or otherwise) connections just a function of the underlying OS and number of instances thereof, not a function of Node.js ?
- nurettin 7y agonumber of connections ultimately depends on the OS limits assuming you have infinite RAM and CPU resources to start your sessions, do the handshakes, serve ping/pong packets, register events to event loops and fire timers.
- unilynx 7y agoEven if the OS can take it, it requires proper engineering in your stack (ie, node.js here). it wasn't long ago that even managing 10K connections on a server was considered quite a feat - see http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html
- nly 7y agoDoesn't sound so impressive. I've done close to a million on a single Digital Ocean droplet using nchan[0] before. Latency was reasonable even with that many connections, you just need to set your buffer sizes carefully. Handshakes are also expensive, so it's useful to to be able to control the client and build in some smart reconnect/back-off logic. [0] https://www.nginx.com/resources/wiki/modules/Nchan/ https://www.nginx.com/resources/wiki/modules/Nchan/
- sixplusone 7y agowould be nice to compare all providers with something like https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ (azure & aws)
- amelius 7y agoIt's not difficult to see that such amount of connections is only useful if your server performs a trivial task. Otherwise, the bottleneck will be your CPU(s).
- k__ 7y agoDoes anyone know what latency overhead switching to ECS/EKS would add?
- winrid 7y agoA better solution would be to use nchan+nginx and then your Node API is just a simple stateless REST service. Will scale better and be easier to maintain.
- sankha93 7y agoI have written and deployed message queues in Node.js that take data from Redis and push it out on websockets. It is a pain to deal with the GC in weird cases. This was about 5 years ago, so the details might not accurate. Things worked fine until some day some customer started sending multi-megabyte strings over the system. It is difficult to actually track down that it is GC that is halting the system and then figuring out ways to fix the issue. We ended up not using JavaScript strings and instead using Node.js buffers to do the transport - I don't recall the Node.js library for Redis supporting that out of the box.
- winrid 7y agoIt's better but still a pain. The GC is so much more fragile than the JVM. Would have used nchan for that probably :p
- sehugg 7y agoI did this with Netty back in 2010. Increasing nf_conntrack_max is a hard-learned lesson.