4 ms·
The general case to use Erlang for your reliability and scalability issues. Plenty of suggestions and time to switch. [0] Now after the fact of its popularity
by rvz 6y ago
The general case to use Erlang for your reliability and scalability issues. Plenty of suggestions and time to switch. [0]
Now after the fact of its popularity from the source of it all [1].
My response: Use Erlang.
Downvoters: So for a chat app that is still down for several hours and needs to be reliable to handle the scale of new users, what should they (Signal) have used instead? Java?
[0] https://mobile.twitter.com/ejpcmac/status/977192665489035265 https://mobile.twitter.com/ejpcmac/status/977192665489035265
[1] https://mobile.twitter.com/anildigital/status/1350274900737585152 https://mobile.twitter.com/anildigital/status/13502749007375...
- vips7L 6y agoHow does erlang fix the problem of not having enough physical servers?
- rvz 6y agoFirst of all, they’re on AWS. They don’t ‘have’ physical servers. Secondly, the point of Erlang is that you do not need that many physical/virtual servers to scale to these millions of users. Perhaps just a few or the same number of servers they have, but running Erlang instead. The BEAM VM and languages based on this is quite frankly designed for this scale. Still working well for WhatsApp and Discord.
- lsllc 6y agoI think what GP is referring to is that WhatsApp's Erlang based system was able to handle 2M connections per server allowing you to do more with less [0] (although I think I read since the FB acquisition, WhatsApp has moved off from Erlang). [0] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...
- darksaints 6y agoThere are lots of languages that can do this, and it requires deliberate engineering effort in all of them, including erlang. I mean, you have to have all of your ports open, with multiple IP addresses allocated to a single server, and the process has to have efficient routing algorithms to do so. Erlang doesn't make this job more efficient than other languages. Erlang has two killer use cases: fault tolerance and horizontal scalability. Computational resource efficiency has never been a goal and it shows. The grandparent comment is nothing more than a cargo cult. "If we use the same language as WhatsApp, we'll magically be just as scalable as WhatsApp". It's horrendously naive.
- toast0 6y ago> I mean, you have to have all of your ports open, with multiple IP addresses allocated to a single server Not sure what this section is about. IP connections are keyed on a 5-tuple of {PeerAIP, PeerBIP, Protocol, PeerAPort, PeerBPort} If you hold PeerAIP and PeerAPort and protocol fixed (single server IP, single port, TCP), you can accept 65535 connections from any other IP (so long as the system managing that IP can use all the ports; if it's CGNAT, it probably can; if it's a traditional OS, it will probably have soft limits, but easily thousands).
- darksaints 6y ago65,535 is substantially less than 2,000,000. At a minimum, you'd need 31 IP addresses allocated to a single box in order to have 2,000,000 active connections on that box. Unless you're arguing that the chat clients would be opening dozens of connections...but that's ridiculous. Whatsapp was built for actual usage, not research projects, and more than 1-2 connections per client is overkill.
- toast0 6y agoLet me be more concrete. If your server is 192.0.2.23, and it listens on port 5222. You can accept 65535 connections from 198.51.100.1. 198.51.100.1:X to 192.0.2.23:5222 for X in {1:65535} (port zero is weird) You can also accept 65535 connections from 198.51.100.2 198.51.100.2:X to 192.0.2.23:5222 for X in {1:65535} Pick any X, like say 12345; a connection between 198.51.100.1:12345 and 192.0.2.23:5222 does not preclude a connection between 198.51.100.2:12345 and 192.0.2.23:5222. Thus, this only gets to be an issue if you have a large number of potential clients behind a single (public) IP address; which is generally not the case (except when you're testing from a load generator)
- rvz 6y agoSo in my comment, it’s somewhat ‘naive’ to use Erlang for distributed systems like a messaging system and is ‘nothing more than a cargo cult’ all because I said WhatsApp is using it? Ha. In fact it is the other way round and especially for Signal’s use case. Signal is already seeking for Erlang expertise in their server developer roles so I’m sure they’re already looking into this. Perhaps using languages that are not suitable or designed for distributed systems like Java sounds more like a cargo-cult than using Erlang. They bought into the JVM hype until their server capsized on high traffic (again) and it is still down.
- vaduz 6y agoAh, but they are using Java [0] - and obviously they need a solution that works with the stack they currently have - proposing Erland at this time is asinine. Erlang would have been solution if they have predicted the brain-dead actions of Whatsapp/Facebook, and if they had ample time to prepare - and it would still have some hard limits on a number of messages it can handle. [0] https://github.com/signalapp/Signal-Server https://github.com/signalapp/Signal-Server
- rvz 6y ago> Ah, but they are using Java Exactly my point. It’s still down. Their Java servers doesn’t seem to handle this scale well and aren’t reliable for this. The fact that they have to “Add more servers” to handle this traffic tells you exactly why Erlang / Elixir or BEAM VM languages are better suited for this (And also can save them money instead of provisioning more servers on AWS). > obviously they need a solution that works with the stack they currently have It was down before even when they had less users. One user made this suggestion 3 years ago, and Signal looked at this. Plenty of time to switch. > proposing Erland at this time is asinine. You mean 3 years ago with their $60M in funding? Plenty of time and money to hire an Erlang / Elixir consultant.
- aaomidi 6y agoYou realize erlang isn't a silver bullet right?
- deleted 6y ago[deleted]
- darksaints 6y ago> The fact that they have to “Add more servers” to handle this traffic tells you exactly why Erlang / Elixir or BEAM VM languages are better suited for this (And also can save them money instead of provisioning more servers on AWS). You do realize that the major reason BeamVM is popular for this use case is because it makes it easy to add more servers, right? Erlang doesn't magically make it so you never have to add more servers. Actually, it's the opposite. Erlang is pretty computationally inefficient compared to most statically typed languages like Java. If there is any VM out there preventing your from needing more servers, it's the JVM. Generally speaking you can do more work with fewer resources with the JVM than you can with the BeamVM...it's just that the BeamVM makes it easier to add more resources.
- tyrust 6y agoThis post is about the app not gracefully handling server errors. Server errors are inevitable no matter what your stack is.
- KaiserPro 6y agoProblem: Client retries every n seconds when it receives a 5xx error. This causes outage amplification Proposed fix: Rebuild the entire backend in a different language. estimated time to outage mitigation: 6-12 months. Now colour me "inexperienced" but I'd personally push a fix for exponential backoff to the client.