8 ms·

Ericsson to WhatsApp: The Story of Erlang

270 points by tim_sw 3y ago

qxfys 3y ago
Met Joe Armstrong 7 years ago during his time at SICS (now RISE). Rest in peace, Joe! Definitely, the guy who became my role model in programming.
- mercer 3y ago
  I didn't know him personally, but I loved how he came across to me as both a cantankerous 'old' man AND an infectiously 'young' enthusiast in his talks.
  - macintux 3y ago
    I had the pleasure of escorting him around Chicago one day, and his enthusiasm was in full force. Such an amazing fellow, hearing the news of his death was a cruel blow.
    - mercer 3y ago
      Do you remember anything of what you talked about?
hoten 3y ago
I still recall a CS jobs fair at my college ~8 years ago, where some company representative took a very bad liking to Erlang on my CV (I took a "Programming Languages" course where Erlang was one of ~6 languages we chewed through. s/o venkat!) He kept asking "why use this over Java at my work X". He seemed to have the idea I wanted to join his company and rewrite their web administrative tools used by 5 people in Erlang. Not sure why he got so annoyed, and didn't ask me about anything other than a language he thought was silly!
- microtonal 3y ago
  Maybe experience with language enthusiasts? When a language has fewer users, the users that it has are more likely to be enthusiasts. Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp. I guess you’ll become allergic to minority programming languages if that happens often enough.
  - medo-bear 3y ago
    > Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp. Same story except it wasn't Lisp, but Rust
  - weatherlight 3y ago
    I frequently get told that the newest version of the JVM can do everything the Erlang VM can do and I should consider switching. Even here, in the comments of hacker news, I've witnessed JVM fanboys proclaim that Erlang, if "properly" implemented, for the JVM, would be faster and more performant then the Erlang VM. Theres zealots on both sides of the fence.
- Simon_O_Rourke 3y ago
  Seems to be a general knee-jerk reaction from typical CS-majors and other Java/C++ "bros" around Erlang. A more mature reflection could easily see lots of potential niches for Erlang, with the big caveat from a business perspective of the relative difficulty in finding devs. I had the pleasure to encounter Erlang professionally many many moons agos, in it's originally intended role within telecoms switching, and was nothing but impressed by it. The argument here might be less about Erlang and more about Maslow's hammer - i.e., if all you've got is a hammer then all you can see are nails.
  - danielscrubs 3y ago
    "CS majors"? Can you specify which universities you think have that mindset? At least for us, CS is just algorithms and maths, we had only one single course for a specific language in my five years (C) but you where required to just pick up whatever the professor chose. If you check the list of computer science awards they all focus on algos & maths so it feels weird if you could get a degree with a "language is what matters" mindset.
giraffe_lady 3y ago
I love erlang so much. It's just such a bizarre combination of things that should't mix as well as they do. Pattern match as assignment still feels futuristic; I know they didn't invent that but it must have taken serious vision & confidence to commit to it. Plus agents as the core abstraction, immutability semantics. It all feels very CS-theory elegant and then you get to OTP, the most grimy roughneck Système D ass standard library I've ever seen. But that has no actual problems anywhere in my experience other than that you'll go years not knowing something is in there bc you didn't guess what five letter abbreviation it would be under. Plus the syntax is like what you would get if you described a programming language to someone who had never seen one. Code from another planet. Flawless.
- bruce343434 3y ago
  For confused fellows: In the context of Erlang, OTP stands for "Open Telecom Platform." The key components of OTP include: OTP Behaviors: Behaviors are generic programming frameworks that define the structure and behavior of a particular type of process. OTP provides several predefined behaviors, such as gen_server, gen_fsm, and gen_event, which encapsulate common patterns of communication and state handling. Supervision Trees: OTP introduces a hierarchical structure called a supervision tree, which allows for the supervision and management of processes. Supervision trees help handle process failures and enable the system to automatically restart failed processes or take appropriate actions in response to failures. Application Framework: OTP provides an application framework that facilitates the organization and configuration of Erlang applications. The framework allows developers to define dependencies between applications, manage application startup and shutdown, and handle system-wide configuration. OTP Design Principles: OTP promotes the use of certain design principles, such as "Let It Crash" and "Fail Fast," which encourage building fault-tolerant systems that can recover from failures quickly and gracefully.
  - pessimizer 3y ago
    > OTP stands for "Open Telecom Platform." I don't think it stands for anything anymore, officially.
kragen 3y ago
you'd think that an article about how facebook bought the erlang-implemented whatsapp because facebook messenger was less popular would bother to mention that facebook messenger used bob ippolito's mochi media's mochiweb, an erlang web server, to handle the comet connections that allowed facebook messenger to deliver asynchronous notifications to web browsers that is, facebook messenger was also to a significant extent written in erlang, though i don't have any insight into how much of the internal facebook messenger system was erlang-based (i imagine probably not that much) also why does the article say > In 2022, it received a notable update that speeded up the execution of Erlang programs significantly through the introduction of a ‘just in time’ compiler. and not mention that the hipe jit compiler was added in 02000, 22 years earlier, with x86 support added the next year? there have literally been jit compilers for erlang since last millennium. https://www.researchgate.net/publication/225129672_The_HiPEx86_Erlang_Compiler_System_Description_and_Performance_Evaluation https://www.researchgate.net/publication/225129672_The_HiPEx...
- klelatti 3y ago
  Thanks for the feedback. The article does say that that Facebook Messenger used Erlang. I've amended the comment on the jit compiler to say 'updated jit compiler' - my mistake.
  - kragen 3y ago
    sorry, i guess i missed that, i'm a dope
- foldr 3y ago
  > and not mention that the hipe jit compiler was added in 02000 It’s notable because previous JITs were not very successful at increasing performance: https://www.erlang.org/blog/the-road-to-the-jit/ https://www.erlang.org/blog/the-road-to-the-jit/
- avsm 3y ago
  There was a Commercial Uses of Functional Programming workshop talk back in 2009 where the engineers behind Facebook Chat talked about their use of Erlang. http://cufp.org/2009/functional-programming-facebook.html http://cufp.org/2009/functional-programming-facebook.html The abstract covers the pros and cons behind using Erlang back then pretty well.
coldtea 3y ago
>Then suddenly, in 1998, Erlang was banned within the Ericsson. The reason? To avoid the costs of development of a language that was unique to Ericsson, and instead to access the shared effort being put into the development of other languages. Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).
- jabl 3y ago
  > Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS). AFAIU, nothing to do with FOSS, which wasn't really on the radar for a somewhat conservative telecom company at the time. It was about custom in-house stuff (PLEX, Erlang, etc.) vs. "industry standard" (C++, OO, UML, etc.). See also my other comment https://news.ycombinator.com/item?id=36570765 https://news.ycombinator.com/item?id=36570765
amadeuspagel 3y ago
> Firstly, it should be possible to change code “on the fly;” in other words, code change operations should be possible without stopping the system. > OTP or the ‘Open Telecoms Platform’ provided capabilities such as ‘hot code reloading’, databases (he ‘Mnesia’ package) and many other facilities. And yet most erlangvm users use it with phoenix as a better rails, deploying with git and docker, attaching a postgres database.
- rkachowski 3y ago
  In practice, hot code reloading and mnesia are pretty cool concepts but very far from trivial. The vm and OTP provide mechanisms to work with such advanced concepts, but it still requires an intense amount of discipline and process to execute effectively. Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture). For given hard requirements, e.g. upgrading the application without dropping TCP connections - this can be a killer feature that removes considerable development effort, but usually it's a very cool thing that you don't need. That being said, hot code reloading on the individual module level I have found to be a useful and dirty trick to change the behaviour of running applications (add log statements / telemetry where they weren't before to diagnose production issues). This still has all the caveats and danger of the above, but is more limited in scope.
  - dikei 3y ago
    Exactly, it's similar to Linux live kernel patching. It's certainly desirable, but with the support of modern orchestration platform, most of the time it's simpler to migrate services off the nodes, update, reboot then redeploy.
    - 3y ago
jabl 3y ago
My father worked at Ericsson for a long time back then, and while he wasn't involved in any of the AXE-N or Erlang stuff, he did hear water-cooler talk about the ongoing political battles. So this is like 3 layers removed from the people who were actually involved in it all, but anyway: There was a huge push to move on from the in-house PLEX to industry standard software engineering. Object orientation, component architecture, UML, design patterns, etc.; think of a software engineering kool-aid from that time period, they were drinking it. And C++ was the industrial strength vehicle to deliver all of this. AXE-N wasn't just a project that flopped, it was the crown jewel of this giant effort to switch to the new "industry standard" way to write telephony software. So the failure was a giant cultural shock. But even after the AXE-N failure, the phalanx behind it weren't done for, and apparently it was they who managed to push through a ban on Erlang, that embarrassing thorn in their side. As for what happened later, I don't know. Anybody who knows what language Ericsson is currently using to program their exchanges?
- swinglock 3y ago
  Ericsson is currently using Erlang where appropriate. Open sourcing Erlang and it finding some success outside, meant it's no longer an in-house only language.
- eproxus 3y ago
  I think all modern Ericsson mobile network base stations use Erlang (at least for the configuration part) so most likely just using your phone you come in contact with an Erlang system everyday (depending on what gear your ISP use, but Ericsson is one of the largest providers of telecom equipment).
- ojl 3y ago
  Lots of C++, but also Erlang and other languages depending on the product (and depending on which part of the product).
- jimmySixDOF 3y ago
  Funny there was just a story on HN about the Rise and Fall of CORBA which was in Telecom vendor slide decks at the time. https://news.ycombinator.com/item?id=36546954 https://news.ycombinator.com/item?id=36546954
Joel_Mckay 3y ago
I think the assumption Sun's Java could become something that eclipsed most languages including Erlang is over, as the prototype distributed JVMs just became another navel-gazing exercise. The Erlang/Elixir OTP has design constraints like any other solution, but has survived 4+ business cycles. One can only hope Julia ends up with something similar, but more user friendly. Distributed systems is hard, and training obstinate people is like trying to teach a Raccoon TCL. If a design really needs Erlang/Elixir, than there are few alternatives that won't degenerate into a polyglot dumpster-fire. Happy computing =)
amelius 3y ago
If only Twitter used Erlang ...
- Rovanion 3y ago
  It would still have Musk.
DeathArrow 3y ago
>Like WhatsApp, Facebook Messenger had used Erlang and ejabberd but moved away from it after finding it difficult to recruit engineers. Since WhatsApp handled much more users with just 35 engineers, I would have thought it wouldn't be so much of a deal to recruit some Erlang engineers. They didn't need to recruit thousands of engineers. What platform have Facebook moved Messenger to?
- toast0 3y ago
  Facebook Messenger moved to a C++ server, built like the rest of Facebook C++ servers. The story I heard was they were looking for people who already knew Erlang and were having trouble finding them, so they rewrote the thing in a language that they could hire for. From my recollection, when I was at WhatsApp, we only hired two server engineers who had used Erlang before. I was considered knowledgable about Erlang, because I vaguely remembered seeing the Slashdot post when Ericsson open sourced it. Pretty much we hired smart people and gave them an Erlang book. As a result, I don't think we did OTP applications properly, and we might have missed out on a couple other things, and our code might not be very idiomatic always, but I think we did ok. But, beyond hiring, Erlang is a weird fit inside Facebook's datacenter infrastructure. The Messenger server team was (and I believe still is) very small, and rewriting into something with less impedence mismatch for a small team to deal with probably made sense. For WhatsApp, we spent quite a lot of effort making our software fit at Facebook, and we had to give up FreeBSD and from what I've heard, hotloading, too.
tru1ock 3y ago
This one misses Discord? Elixir drives a lot of the backend as well?
- barkerja 3y ago
  I believe it does. Elixir with quite a few Rust NIFs is my understanding.
  - weatherlight 3y ago
    Just a couple. They have to do with sets and set operations.
deleted 3y ago
[deleted]
silverlake 3y ago
Motorola Labs had a similar-ish internal programming language around this time. It also died because devs didn’t want to use an internally-developed language. The important things were hiring, training, support, external tools, etc. Being a “good” language was at the bottom of the list. They went with C++. Even though I worked on that PL I knew they were right not to bet a giant business on a bunch of researchers.
jarek83 3y ago
Aside from the main article, I find it very convincing marketing strategy to announce a continuation of the subject in next chapter that will be paid. I could already see the quality of the authors and I feel like I want to pay now to read more. I'm all in for this kind of marketing.

270 points by tim_sw 3y ago

Met Joe Armstrong 7 years ago during his time at SICS (now RISE). Rest in peace, Joe! Definitely, the guy who became my role model in programming.

mercer 3y ago
I didn't know him personally, but I loved how he came across to me as both a cantankerous 'old' man AND an infectiously 'young' enthusiast in his talks.
- macintux 3y ago
  I had the pleasure of escorting him around Chicago one day, and his enthusiasm was in full force. Such an amazing fellow, hearing the news of his death was a cruel blow.
  - mercer 3y ago
    Do you remember anything of what you talked about?

hoten 3y ago

I still recall a CS jobs fair at my college ~8 years ago, where some company representative took a very bad liking to Erlang on my CV (I took a "Programming Languages" course where Erlang was one of ~6 languages we chewed through. s/o venkat!) He kept asking "why use this over Java at my work X". He seemed to have the idea I wanted to join his company and rewrite their web administrative tools used by 5 people in Erlang. Not sure why he got so annoyed, and didn't ask me about anything other than a language he thought was silly!

microtonal 3y ago
Maybe experience with language enthusiasts? When a language has fewer users, the users that it has are more likely to be enthusiasts. Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp. I guess you’ll become allergic to minority programming languages if that happens often enough.
- medo-bear 3y ago
  > Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp. Same story except it wasn't Lisp, but Rust
- weatherlight 3y ago
  I frequently get told that the newest version of the JVM can do everything the Erlang VM can do and I should consider switching. Even here, in the comments of hacker news, I've witnessed JVM fanboys proclaim that Erlang, if "properly" implemented, for the JVM, would be faster and more performant then the Erlang VM. Theres zealots on both sides of the fence.
Simon_O_Rourke 3y ago
Seems to be a general knee-jerk reaction from typical CS-majors and other Java/C++ "bros" around Erlang. A more mature reflection could easily see lots of potential niches for Erlang, with the big caveat from a business perspective of the relative difficulty in finding devs. I had the pleasure to encounter Erlang professionally many many moons agos, in it's originally intended role within telecoms switching, and was nothing but impressed by it. The argument here might be less about Erlang and more about Maslow's hammer - i.e., if all you've got is a hammer then all you can see are nails.
- danielscrubs 3y ago
  "CS majors"? Can you specify which universities you think have that mindset? At least for us, CS is just algorithms and maths, we had only one single course for a specific language in my five years (C) but you where required to just pick up whatever the professor chose. If you check the list of computer science awards they all focus on algos & maths so it feels weird if you could get a degree with a "language is what matters" mindset.

giraffe_lady 3y ago

I love erlang so much. It's just such a bizarre combination of things that should't mix as well as they do. Pattern match as assignment still feels futuristic; I know they didn't invent that but it must have taken serious vision & confidence to commit to it. Plus agents as the core abstraction, immutability semantics. It all feels very CS-theory elegant and then you get to OTP, the most grimy roughneck Système D ass standard library I've ever seen. But that has no actual problems anywhere in my experience other than that you'll go years not knowing something is in there bc you didn't guess what five letter abbreviation it would be under. Plus the syntax is like what you would get if you described a programming language to someone who had never seen one. Code from another planet. Flawless.

bruce343434 3y ago
For confused fellows: In the context of Erlang, OTP stands for "Open Telecom Platform." The key components of OTP include: OTP Behaviors: Behaviors are generic programming frameworks that define the structure and behavior of a particular type of process. OTP provides several predefined behaviors, such as gen_server, gen_fsm, and gen_event, which encapsulate common patterns of communication and state handling. Supervision Trees: OTP introduces a hierarchical structure called a supervision tree, which allows for the supervision and management of processes. Supervision trees help handle process failures and enable the system to automatically restart failed processes or take appropriate actions in response to failures. Application Framework: OTP provides an application framework that facilitates the organization and configuration of Erlang applications. The framework allows developers to define dependencies between applications, manage application startup and shutdown, and handle system-wide configuration. OTP Design Principles: OTP promotes the use of certain design principles, such as "Let It Crash" and "Fail Fast," which encourage building fault-tolerant systems that can recover from failures quickly and gracefully.
- pessimizer 3y ago
  > OTP stands for "Open Telecom Platform." I don't think it stands for anything anymore, officially.

kragen 3y ago

you'd think that an article about how facebook bought the erlang-implemented whatsapp because facebook messenger was less popular would bother to mention that facebook messenger used bob ippolito's mochi media's mochiweb, an erlang web server, to handle the comet connections that allowed facebook messenger to deliver asynchronous notifications to web browsers that is, facebook messenger was also to a significant extent written in erlang, though i don't have any insight into how much of the internal facebook messenger system was erlang-based (i imagine probably not that much) also why does the article say > In 2022, it received a notable update that speeded up the execution of Erlang programs significantly through the introduction of a ‘just in time’ compiler. and not mention that the hipe jit compiler was added in 02000, 22 years earlier, with x86 support added the next year? there have literally been jit compilers for erlang since last millennium. https://www.researchgate.net/publication/225129672_The_HiPEx86_Erlang_Compiler_System_Description_and_Performance_Evaluation https://www.researchgate.net/publication/225129672_The_HiPEx...

klelatti 3y ago
Thanks for the feedback. The article does say that that Facebook Messenger used Erlang. I've amended the comment on the jit compiler to say 'updated jit compiler' - my mistake.
- kragen 3y ago
  sorry, i guess i missed that, i'm a dope
foldr 3y ago
> and not mention that the hipe jit compiler was added in 02000 It’s notable because previous JITs were not very successful at increasing performance: https://www.erlang.org/blog/the-road-to-the-jit/ https://www.erlang.org/blog/the-road-to-the-jit/
avsm 3y ago
There was a Commercial Uses of Functional Programming workshop talk back in 2009 where the engineers behind Facebook Chat talked about their use of Erlang. http://cufp.org/2009/functional-programming-facebook.html http://cufp.org/2009/functional-programming-facebook.html The abstract covers the pros and cons behind using Erlang back then pretty well.

coldtea 3y ago

>Then suddenly, in 1998, Erlang was banned within the Ericsson. The reason? To avoid the costs of development of a language that was unique to Ericsson, and instead to access the shared effort being put into the development of other languages. Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).

jabl 3y ago
> Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS). AFAIU, nothing to do with FOSS, which wasn't really on the radar for a somewhat conservative telecom company at the time. It was about custom in-house stuff (PLEX, Erlang, etc.) vs. "industry standard" (C++, OO, UML, etc.). See also my other comment https://news.ycombinator.com/item?id=36570765 https://news.ycombinator.com/item?id=36570765

amadeuspagel 3y ago

> Firstly, it should be possible to change code “on the fly;” in other words, code change operations should be possible without stopping the system. > OTP or the ‘Open Telecoms Platform’ provided capabilities such as ‘hot code reloading’, databases (he ‘Mnesia’ package) and many other facilities. And yet most erlangvm users use it with phoenix as a better rails, deploying with git and docker, attaching a postgres database.

rkachowski 3y ago
In practice, hot code reloading and mnesia are pretty cool concepts but very far from trivial. The vm and OTP provide mechanisms to work with such advanced concepts, but it still requires an intense amount of discipline and process to execute effectively. Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture). For given hard requirements, e.g. upgrading the application without dropping TCP connections - this can be a killer feature that removes considerable development effort, but usually it's a very cool thing that you don't need. That being said, hot code reloading on the individual module level I have found to be a useful and dirty trick to change the behaviour of running applications (add log statements / telemetry where they weren't before to diagnose production issues). This still has all the caveats and danger of the above, but is more limited in scope.
- dikei 3y ago
  Exactly, it's similar to Linux live kernel patching. It's certainly desirable, but with the support of modern orchestration platform, most of the time it's simpler to migrate services off the nodes, update, reboot then redeploy.
  - 3y ago

jabl 3y ago

My father worked at Ericsson for a long time back then, and while he wasn't involved in any of the AXE-N or Erlang stuff, he did hear water-cooler talk about the ongoing political battles. So this is like 3 layers removed from the people who were actually involved in it all, but anyway: There was a huge push to move on from the in-house PLEX to industry standard software engineering. Object orientation, component architecture, UML, design patterns, etc.; think of a software engineering kool-aid from that time period, they were drinking it. And C++ was the industrial strength vehicle to deliver all of this. AXE-N wasn't just a project that flopped, it was the crown jewel of this giant effort to switch to the new "industry standard" way to write telephony software. So the failure was a giant cultural shock. But even after the AXE-N failure, the phalanx behind it weren't done for, and apparently it was they who managed to push through a ban on Erlang, that embarrassing thorn in their side. As for what happened later, I don't know. Anybody who knows what language Ericsson is currently using to program their exchanges?

swinglock 3y ago
Ericsson is currently using Erlang where appropriate. Open sourcing Erlang and it finding some success outside, meant it's no longer an in-house only language.
eproxus 3y ago
I think all modern Ericsson mobile network base stations use Erlang (at least for the configuration part) so most likely just using your phone you come in contact with an Erlang system everyday (depending on what gear your ISP use, but Ericsson is one of the largest providers of telecom equipment).
ojl 3y ago
Lots of C++, but also Erlang and other languages depending on the product (and depending on which part of the product).
jimmySixDOF 3y ago
Funny there was just a story on HN about the Rise and Fall of CORBA which was in Telecom vendor slide decks at the time. https://news.ycombinator.com/item?id=36546954 https://news.ycombinator.com/item?id=36546954

Joel_Mckay 3y ago

I think the assumption Sun's Java could become something that eclipsed most languages including Erlang is over, as the prototype distributed JVMs just became another navel-gazing exercise. The Erlang/Elixir OTP has design constraints like any other solution, but has survived 4+ business cycles. One can only hope Julia ends up with something similar, but more user friendly. Distributed systems is hard, and training obstinate people is like trying to teach a Raccoon TCL. If a design really needs Erlang/Elixir, than there are few alternatives that won't degenerate into a polyglot dumpster-fire. Happy computing =)

amelius 3y ago

If only Twitter used Erlang ...

Rovanion 3y ago
It would still have Musk.

DeathArrow 3y ago

>Like WhatsApp, Facebook Messenger had used Erlang and ejabberd but moved away from it after finding it difficult to recruit engineers. Since WhatsApp handled much more users with just 35 engineers, I would have thought it wouldn't be so much of a deal to recruit some Erlang engineers. They didn't need to recruit thousands of engineers. What platform have Facebook moved Messenger to?

toast0 3y ago
Facebook Messenger moved to a C++ server, built like the rest of Facebook C++ servers. The story I heard was they were looking for people who already knew Erlang and were having trouble finding them, so they rewrote the thing in a language that they could hire for. From my recollection, when I was at WhatsApp, we only hired two server engineers who had used Erlang before. I was considered knowledgable about Erlang, because I vaguely remembered seeing the Slashdot post when Ericsson open sourced it. Pretty much we hired smart people and gave them an Erlang book. As a result, I don't think we did OTP applications properly, and we might have missed out on a couple other things, and our code might not be very idiomatic always, but I think we did ok. But, beyond hiring, Erlang is a weird fit inside Facebook's datacenter infrastructure. The Messenger server team was (and I believe still is) very small, and rewriting into something with less impedence mismatch for a small team to deal with probably made sense. For WhatsApp, we spent quite a lot of effort making our software fit at Facebook, and we had to give up FreeBSD and from what I've heard, hotloading, too.

tru1ock 3y ago

This one misses Discord? Elixir drives a lot of the backend as well?

barkerja 3y ago
I believe it does. Elixir with quite a few Rust NIFs is my understanding.
- weatherlight 3y ago
  Just a couple. They have to do with sets and set operations.

deleted 3y ago

[deleted]

silverlake 3y ago

Motorola Labs had a similar-ish internal programming language around this time. It also died because devs didn’t want to use an internally-developed language. The important things were hiring, training, support, external tools, etc. Being a “good” language was at the bottom of the list. They went with C++. Even though I worked on that PL I knew they were right not to bet a giant business on a bunch of researchers.

jarek83 3y ago

Aside from the main article, I find it very convincing marketing strategy to announce a continuation of the subject in next chapter that will be paid. I could already see the quality of the authors and I feel like I want to pay now to read more. I'm all in for this kind of marketing.