4 ms·

Thanks for a terrific and informative response! > About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. Can you

Thanks for a terrific and informative response! > About half the early server team had used FreeBSD at Yahoo, and zero had any experience with Erlang. Can you say anything more about the choice of Erlang and experience with it? It seems pretty remarkable that you built such a successful business / service with no experience of Erlang.

toast0 4y ago
Hire smart people, give them Erlang books, tada. Erlang is basically a perfect fit for a chat server. ejabberd is out there and there's several high profile use cases where someone needed a chat server, picked ejabberd and became smitten with Erlang. Facebook did it before WhatsApp, but they rebuilt chat in C++ because they couldn't hire Erlang people? and the two people working on it wanted to have some time to do other things. Riot Games did it more recently. The nice thing about Erlang is it's really easy to make changes to your system; you don't need to move traffic and restart a server, you can just hotload the code changes. So, they started with ejabberd, and by the time I joined it was completely different. Erlang also has tremendous observability. Each Erlang process has detailed information available (heap size, message queue length, reduction count (cpu use, kind of), you can view the message queue if you want, etc). The BEAM is also observable, it's even got DTrace hooks. There's a way to build where you get stats on lock contention (lock counting), which pretty quickly helps you find the root of scaling issues. Once you find lock contention, almost always the answer is find a way to distribute the work over more locks so there's less contention; same thing you do when a process's message queue backs up. It is a small community, so if you're blessed with needing to scale, you are a bit on your own, but you probably could reach out to Erlang Solutions, if you get really stuck. Erlang will scale much farther than the community thinks though. WhatsApp routinely ran mnesia databases with way more data than people thought possible (just don't use disc_only_copies for anything other than the schema table) and dist clusters way larger than people thought possible (pg2 and global locks in general got iffy at times, but I think the new pg is better, and we didn't use global locks except through pg2). There is a general trend where there's no explicit limit on things, but the system will not work well if you go over the implicit limit, and you won't have much guidance. I hear WhatsApp now uses proper Erlang application packaging, wkth releases and relups, etc. But when I was there, we just did hotloading and it was good enough.
- klelatti 4y ago
  Really interesting, thanks again for sharing.

toast0 4y ago

Hire smart people, give them Erlang books, tada. Erlang is basically a perfect fit for a chat server. ejabberd is out there and there's several high profile use cases where someone needed a chat server, picked ejabberd and became smitten with Erlang. Facebook did it before WhatsApp, but they rebuilt chat in C++ because they couldn't hire Erlang people? and the two people working on it wanted to have some time to do other things. Riot Games did it more recently. The nice thing about Erlang is it's really easy to make changes to your system; you don't need to move traffic and restart a server, you can just hotload the code changes. So, they started with ejabberd, and by the time I joined it was completely different. Erlang also has tremendous observability. Each Erlang process has detailed information available (heap size, message queue length, reduction count (cpu use, kind of), you can view the message queue if you want, etc). The BEAM is also observable, it's even got DTrace hooks. There's a way to build where you get stats on lock contention (lock counting), which pretty quickly helps you find the root of scaling issues. Once you find lock contention, almost always the answer is find a way to distribute the work over more locks so there's less contention; same thing you do when a process's message queue backs up. It is a small community, so if you're blessed with needing to scale, you are a bit on your own, but you probably could reach out to Erlang Solutions, if you get really stuck. Erlang will scale much farther than the community thinks though. WhatsApp routinely ran mnesia databases with way more data than people thought possible (just don't use disc_only_copies for anything other than the schema table) and dist clusters way larger than people thought possible (pg2 and global locks in general got iffy at times, but I think the new pg is better, and we didn't use global locks except through pg2). There is a general trend where there's no explicit limit on things, but the system will not work well if you go over the implicit limit, and you won't have much guidance. I hear WhatsApp now uses proper Erlang application packaging, wkth releases and relups, etc. But when I was there, we just did hotloading and it was good enough.

klelatti 4y ago
Really interesting, thanks again for sharing.