13 ms·

Scaling Erlang Developer Experience at WhatsApp [pdf]

308 points by anuragsoni 6y ago

btbuildem 6y ago
> some {T} | undefined Can we please avoid TS pitfalls?
- phowat 6y ago
  Would you mind expanding a little bit on why this is bad ? I'm fairly new to TS and find myself doing it a lot.
  - angio 6y ago
    The advantage of the Option type is that you don't need to check for nullability, you simply pattern match on Some(_) or None. A type of `some{T} | undefined` means that the expression can have values `Some(_)` or `None` or `undefined`, which defeats the point of the Option type!
    - masklinn 6y ago
      Except Erlang doesn't have a none. Nor an undefined for that matter. So `some {T} | undefined` means exactly that, there is no underlying null pointer which could sneak up on you. `undefined` here is not built-in type or value of the language, it's a standard atom like any other. So are null, nil, value or false (although false does have a special status in that it's the customary output of boolean operations alongside true).
      - angio 6y ago
        I'm aware of that, I was explaining why `some{T} | undefined` is problematic in Typescript. If you have a signature with both option type and undefined it's a signal of a code smell.
        mbo 6y ago
        But `some{T}` isn't an option type, it's a member of the option type!
    - mbo 6y ago
      > A type of `some{T} | undefined` means that the expression can have values `Some(_)` or `None` or `undefined` Can you please explain this further? Why is `None` an inhabitant of this union?
  - btbuildem 6y ago
    It can lead to cascading checks for undefined; best case scenario is a lot of boilerplate type code.
    - masklinn 6y ago
      > It can lead to cascading checks for undefined Do you know anything about Erlang? Because unlike typescript, `undefined` in erlang is not a special value, it's just an atom like any other. > best case scenario is a lot of boilerplate type code. Do you know anything about Erlang(2)? Because unlike typescript erlang has pattern matching, both faillible and not. I'm not talking "it can kinda unpack objects", erlang has actual pattern matching (though not unification despite the prolog-inspired syntax), and a simple-looking assignment is a pattern match (and a potential assertion).
      - airstrike 6y ago
        Couple things: personal attacks are not cool, and "please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
        masklinn 6y ago
        > Couple things: personal attacks are not cool, and "please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." Couple things: asking somebody if they know anything about a subject is not a personal attack, and that GP doesn't know anything about Erlang and completely miunderstands the snippet is the strongest plausible interpretation of their comments which assumes good faith. The alternative interpretations which do not are that they're actively lying, or that they can't be trusted with any device more complicated than a rock or a small stick.
        dang 6y ago
        It sounds like you didn't intend it as a personal attack, but inserting rhetorical questions like "Do you know anything about $foo" in an argument about $foo definitely comes across that way on the internet, and isn't allowed here. The fact that you repeated it twice makes it much worse—this is an attempt to expose the other commenter as "someone who doesn't know anything", which is needlessly personal. If you know more than someone else, that's great, but please just share some of what you know so the rest of us can learn. Putting others down (even implicitly) doesn't serve learning, it's just distracting and builds up toxins in the ecosystem. Consider how much better your comment would read without those bits. The best way to react to bad information is respectfully with better information.
amgreg 6y ago
What I like most about some experienced engineers who have been at companies from the beginning is that they realize the needs of a small-team, cash-constrained, rapidly growing company are different from that of a dozens-of-teams, complex, but also rapidly scaling enterprise. The devil is in understanding when and how to transition your tooling, architecture, and — dare I say it — attitudes, from the one to the other. I am curious to hear more from these engineers; how decisions were made about transitioning/scaling; and what decisions were made.
- jacquesm 6y ago
  This comment is gold. Yes, that's exactly it. Trying to use the tooling and processes that are a bad match for your situation is the way to waste a fortune or to end up being overtaken on the right by a competitor that made smarter choices.
- thdrdt 6y ago
  There are things that could (should?) be done from the start. Most has to do with automation. For example automated build tools. Maybe you don't want automated unit tests in the beginning but those parts can be added later without changing the company's culture and stucture too much. Or automated invoices. It is much easier to change this than changing a co-worker who manually created all the invoices.
  - outworlder 6y ago
    > Most has to do with automation. This. Don't ever put anything into production that's not automated. Ever. "Oh it's a simple HTTP server that's going to be used by 5 people". Cool. Automate it. "Oh but it's going to take time!". I thought you said it was simple... When things are not automated, we treat them differently. If a server is not automated, now it becomes time-consuming to setup another, which means it's now a pet, not cattle. But if it is automated and is misbehaving, we can easily replace it (and save the current for troubleshooting later if necessary). Or we can spin up 100 of them. Same goes for development. If builds are not automated, they become more difficult to do, more error prone, which means people will avoid making contributions unless they are forced to do so (by some ticket). When contributing to a code base that has no guardrails becomes a lose/neutral proposition, people won't want to. And now it's been 5 years and it's difficult to automate things because there's a 100 different workflows that people use and you are going to break them.
AzzieElbab 6y ago
I am curios why they need 1K people instead of 10? What changed on the server side of WhatsApp?
- abrolhos 6y ago
  I understand that the 1K+ engineer number comes from Facebook as a whole, when mentioning the use of Hack as a typed PHP. More like to give a perspective on the transition on how to come from a small company into a huge one.
  - deleted 6y ago
    [deleted]
  - user5994461 6y ago
    WhatApps got audio and video calls after the Facebook acquisition. Usage increased by a few orders of magnitude too.
    - stjohnswarts 6y ago
      Yeah I think people get usage confused with number of users.
    - AzzieElbab 6y ago
      That explains it. I haven’t used WhatsApp until few months ago
- user5994461 6y ago
  No global service can run with 10 people. Just to have a good coverage 24/7 you're looking at a 50 people company.
  - oblio 6y ago
    Yeah, just ops would be 3 timezones x 8 hours x 2 (ideally 3) people per timezone. So that's 6-9 people just for a barebones ops team.
stevencorona 6y ago
I absolutely love the BEAM and Elixir, but one of my common complaints and source of errors is the lack of compile-time type sanity checking, so I'm excited to see movement in this direction. Dialyzer is sorta okay, but the speed, cryptic errors, and syntax have kept me from fully embracing it.
- findjashua 6y ago
  Gleam may be what you're looking for: https://gleam.run/ https://gleam.run/
  - faitswulff 6y ago
    I'd be curious how WhatsApp's internal statically typed Erlang solution looks compared to Gleam. If I had to guess based on current trends, I'd guess that it's a gradually typed, opt-in toolchain a la Sorbet or Typescript, in order to maintain backwards compatibility and gradually update their legacy codebase as opposed to a whole different language.
  - callamdelaney 6y ago
    This isn't ready for prime time yet iirc.
    - sodapopcan 6y ago
      It's not but is in constant development. The author is very active on elixirforum.com
fbn79 6y ago
Working in a small Company always let me astonished learning things about unicorn apps. But really an application like Whatsapp need 1000+ developers!? I can think about teams about android, ios and web app. X2 if you think about teams for business oriented services. But 1000+ developers for what?
- jamestimmins 6y ago
  Just some of the needs at that scale: Fraud Security Integrations Infrastructure Data Accessibility Growth I18n API In addition, there won't be individual teams for android, ios, etc. Those domains will be broken across many task-specific teams.
- gt565k 6y ago
  Whatsapp had a total of 55 employees when it was acquired by facebook for $16B.
- MiroF 6y ago
  > But 1000+ developers for what? Managerial prestige
- ramraj07 6y ago
  Even whatsapp the app is deceptively simple. It works seamlessly in all the dialects around the world, on a swathe of devices quite possibly matched by no other app; and it manages to handle every actually useful feature anyone wants in a messaging app with amazing aplomb. In India for the majority of the people whatsapp is not just their communication platform, it's also their photo album, dropbox, task manager, social networking platform and contacts manager. All within the simplest UIs I've ever seen.
derefr 6y ago
Last I heard (years ago), I though that WhatsApp's backend was being rewritten in some other language, because Facebook didn't think maintaining an Erlang codebase was tenable. Did that change?
- ramchip 6y ago
  Perhaps you're thinking of Facebook chat? https://www.quora.com/Why-was-Erlang-chosen-for-use-in-Facebook-chat https://www.quora.com/Why-was-Erlang-chosen-for-use-in-Faceb... https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-for-Facebook-Chat-What-was-the-reason-for-it-What-did-they-replace-it-with https://www.quora.com/When-did-Facebook-switch-away-from-usi...
- deleted 6y ago
  [deleted]
tiffanyh 6y ago
> "We are working on a prototype, open-sourcing in November" I'm curious to see how Elixir integration with such static type system would work.
- seanclayton 6y ago
  I absolutely love Elixir and use it all the time, I really just wish it had a type system (that's not the slow behemoth Dialyzer). I understand the difficulty of typing processes and I don't envy the task, but as a user of the language it would make it that much better to work with! Just imagine working with a language like Elixir that has the type system of F#!
  - acjohnson55 6y ago
    Sounds like Scala / Akka to me
- rch 6y ago
  I think it's more natural to gradually implement components as Erlang NIFs in Rust via https://github.com/rusterlium/rustler https://github.com/rusterlium/rustler.
gautamcgoel 6y ago
Can someone explain what Erlang offers that Go does not? Both have great support for concurrency, but Go has type checking and a more familiar syntax.
- emit_time 6y ago
  Doesn’t erlang have a REPL, allows you to do live deployments. Individual processes crashing doesn’t bring everything down, you can tell it how to handle those. Dynamic typing. Functional programming. Concise syntax, with “let it fail” mentality.
- epidemian 6y ago
  There's this talk, The Sould of Erlang and Elixir, that opened my eyes to what makes this platform great: https://youtu.be/JvBT4XBdoUE https://youtu.be/JvBT4XBdoUE It's not so much about what Erlang, as a language, "offers that Go does not". It's about the BEAM VM that runs it that makes all the difference. I very much recommend watching the talk if you're interested in the topic :)
  - coder543 6y ago
    I watched that talk just now, and it was basically a retelling of everything I really like about Go. Except, Go is statically typed, and a lot faster than BEAM in benchmarks that I've seen. Go is also a very "normal" language, so it's usually easy to onboard developers into it, no matter what their background is. JetBrains GoLand makes the IDE experience essentially impeccable. I can refactor code with a few clicks, and GoLand precisely updates all references thanks to the static type system. Go can run massive numbers of Goroutines on a single system, and it preemptively timeslices individual Goroutines, so I'm never worried about any single task preventing other tasks from making progress, which was like 75% of that talk. Go has a fantastic production runtime debuggability story, which was the other 25% of that talk. Go's pprof support feels magical: https://golang.org/pkg/net/http/pprof/ https://golang.org/pkg/net/http/pprof/ You can connect to running systems and see flamegraphs of where CPU time is being spent, you can take traces of the goroutines to understand why the system is idle when you think it shouldn't be, you can see how much memory different parts of the system are using, and so much more. It's not mentioned as prominently as I think it should be, but `go tool pprof` has this `-http=:` option that will open a web browser to let you just see everything graphically and click around, and it's an amazing experience. There were really only three things in there that Go doesn't offer: 1. Forcefully terminating processes Go doesn't allow anyone to forcefully terminate a Goroutine except the Goroutine itself. Personally, if I identified a buggy Goroutine, I would just focus on writing a patch and deploying it. Manually terminating a process is cool, but then the user types "-1" again and again, and you're just sitting there playing whack-a-mole instead of fixing the problem and deploying the fix. 2. Patching the running system without restarting it The advantages of this are difficult to enumerate. It sounds nice and clean conceptually, but I would worry about existing processes / goroutines being in unexpected states because of erroneous code that they had previously run. Having a way to gracefully shut down a system and restart it without abruptly terminating responsive client connections is much more predictable to me. 3. Seamlessly distributed execution He briefly talks about distributed Erlang towards the end of the talk. but mentions that it has serious problems and that a number of people don't recommend using it in production. Personally, even if it were nearly perfect, I think the benefits of it are limited. It's not difficult to have nodes communicate with each other using any number of other things, including proper RPC libraries all the way down to just raw TCP connections. The hard thing is service discovery, and I don't think distributed erlang meaningfully solves the problem different than other solutions. If everything is running homogeneous code, why bother your nodes with knowing about the existence of other nodes? Just use a load balancer, and then do everything internally on each node. If your systems are heterogeneous, you need a way to send the right messages to the right nodes, and that's where a named load balancer for each type of node can do just fine.
an_opabinia 6y ago
Erlang is the Actors-model version of Greenspun's tenth rule of programming: "Any sufficiently complicated server backend contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang." Is it really the end-all-be-all model of server architecture? What are microservices, if not Actors Clunkified? "Serverless" functions, if not the simplest actors in disguise? Docker, actors except over IP addresses and clunky APIs? The real takeaway: Valuable stuff is stateful. You know you're doing valuable work for a real human end-user if you are dealing with tricky state problems. Extracting money out of that is a bunch of developer-productivity-limited business concerns. It seems actors are a good model to solve that.
- davidw 6y ago
  That quote regarding Erlang is attributed to Robert Virding.
- acjohnson55 6y ago
  I agree that serverless / cloud native feels a lot like what you would build in a distributed monolithic codebase using Erlang or Akka. I guess you get something more polyglot and modular for your trouble, but I think a lot of companies have skipped right over much less complex setups for building resilient, scalable distributed systems.
kornish 6y ago
One of my favorite talks is by Rick Reed about scaling Erlang at WhatsApp. What an absolute savage. He flies through an articulate and in-depth curriculum on system performance and bottleneck mitigation. The talk is called "That's 'Billion' with a 'B'" and makes for a great lunchtime watch: https://www.youtube.com/watch?v=c12cYAUTXXs https://www.youtube.com/watch?v=c12cYAUTXXs
- strmpnk 6y ago
  I remember that year. I was giving a talk at EF during the same time slot but the schedule originally had me in the large room and they had a much smaller one. When the news of the acquisition hit, everyone wanted to see the WhatsApp talk. The organizers knew this so we swapped rooms. So, I started my talk by asking if anyone in the room was here for the WhatsApp talk and told them they could quietly leave and I wouldn't mind and a bunch of people got up. Heheh. I don't blame them. I didn't really like my talk and Rick Reed is very good at what he does and the talk is no exception.
- secondcoming 6y ago
  I know nothing about Erlang, but I do know high-volume systems. Needing 550 beefy servers to handle all that load is - in my mind - not that impressive.
- staticassertion 6y ago
  Yeah, not quite related to the linked slides, which are about scaling in terms of productivity, but still a great talk. I really found that "isolation" is just a key optimization and one of the most useful properties of a system, which they call out in that talk. I wrote about it a bit more in depth here: https://www.graplsecurity.com/post/architecting-for-performance-and-security-at-the-same-time https://www.graplsecurity.com/post/architecting-for-performa... I've written our data processing layer and our event orchestration layer as basically an actor oriented system, with push/pull systems in a couple of key places where it makes sense. It's incredible what you can do with strong isolated infrastructure in terms of performance, security, reliability, and quality.
AshamedCaptain 6y ago
I find it funny they say Whatsapp "chose" Erlang. They "chose" Erlang because ejabberd was written on it, and they chose ejabberd because of a Jabber mailing list suggestion. That's about it...
- chii 6y ago
  are you saying that the choice was purely due to luck? Would whatsapp have been as popular and successful had it not been in erlang but some other language/stack?
  - krn 6y ago
    > are you saying that the choice was purely due to luck? What he is saying is, that they chose the best open source tool available for their use case at that time, and not a programming language. > Would whatsapp have been as popular and successful had it not been in erlang but some other language/stack? I would argue, that they might not have been as efficient in terms of computational resources and the number of technical employees needed, but they would still have succeeded even if the back-end was implemented in Java. Because what all WhatsApp users actually interacted with were the clients – and they were not written in Erlang.
pjmlp 6y ago
> shift to modern languages with integrated tooling, e.g. Erlang competition: > > C++, Java => Go, Rust, Kotlin My first C++ IDE was Turbo C++ 1.0 for MS-DOS in 1993, and Borland, Zortech, Symatec, Microsoft, IBM, and plenty of others had well known C++ IDEs. Many of the ideas of modern C++ IDEs is basically rediscovering the work done by Lucid and IBM with their Smalltalk like capabilities for C++ tooling in the late 80's/early 90's. Java is well known for being one of the languages with best IDE support out of box. It didn't start like that in 1996, but the last 25 years have been good to it. Go and Rust are yet to have similar offerings and Kotlin, well it is a JVM language and profits from the Java eco-system and Google's push to replace it on Android due to non-technical reasons. Maybe better examples should have been provided.
- saagarjha 6y ago
  Java is interesting because writing code in it is generally absolutely miserable without an IDE.
ramon 6y ago
I think the concepts are very interesting of Erlang but I still don't understand it fully. And also about the multi-node os only an internal network scenario is not like a cloud mesh thing. You would need docker.

308 points by anuragsoni 6y ago

btbuildem 6y ago

> some {T} | undefined Can we please avoid TS pitfalls?

phowat 6y ago
Would you mind expanding a little bit on why this is bad ? I'm fairly new to TS and find myself doing it a lot.
- angio 6y ago
  The advantage of the Option type is that you don't need to check for nullability, you simply pattern match on Some(_) or None. A type of `some{T} | undefined` means that the expression can have values `Some(_)` or `None` or `undefined`, which defeats the point of the Option type!
  - masklinn 6y ago
    Except Erlang doesn't have a none. Nor an undefined for that matter. So `some {T} | undefined` means exactly that, there is no underlying null pointer which could sneak up on you. `undefined` here is not built-in type or value of the language, it's a standard atom like any other. So are null, nil, value or false (although false does have a special status in that it's the customary output of boolean operations alongside true).
    - angio 6y ago
      I'm aware of that, I was explaining why `some{T} | undefined` is problematic in Typescript. If you have a signature with both option type and undefined it's a signal of a code smell.
      - mbo 6y ago
        But `some{T}` isn't an option type, it's a member of the option type!
  - mbo 6y ago
    > A type of `some{T} | undefined` means that the expression can have values `Some(_)` or `None` or `undefined` Can you please explain this further? Why is `None` an inhabitant of this union?
- btbuildem 6y ago
  It can lead to cascading checks for undefined; best case scenario is a lot of boilerplate type code.
  - masklinn 6y ago
    > It can lead to cascading checks for undefined Do you know anything about Erlang? Because unlike typescript, `undefined` in erlang is not a special value, it's just an atom like any other. > best case scenario is a lot of boilerplate type code. Do you know anything about Erlang(2)? Because unlike typescript erlang has pattern matching, both faillible and not. I'm not talking "it can kinda unpack objects", erlang has actual pattern matching (though not unification despite the prolog-inspired syntax), and a simple-looking assignment is a pattern match (and a potential assertion).
    - airstrike 6y ago
      Couple things: personal attacks are not cool, and "please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
      - masklinn 6y ago
        > Couple things: personal attacks are not cool, and "please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." Couple things: asking somebody if they know anything about a subject is not a personal attack, and that GP doesn't know anything about Erlang and completely miunderstands the snippet is the strongest plausible interpretation of their comments which assumes good faith. The alternative interpretations which do not are that they're actively lying, or that they can't be trusted with any device more complicated than a rock or a small stick.
        dang 6y ago
        It sounds like you didn't intend it as a personal attack, but inserting rhetorical questions like "Do you know anything about $foo" in an argument about $foo definitely comes across that way on the internet, and isn't allowed here. The fact that you repeated it twice makes it much worse—this is an attempt to expose the other commenter as "someone who doesn't know anything", which is needlessly personal. If you know more than someone else, that's great, but please just share some of what you know so the rest of us can learn. Putting others down (even implicitly) doesn't serve learning, it's just distracting and builds up toxins in the ecosystem. Consider how much better your comment would read without those bits. The best way to react to bad information is respectfully with better information.

amgreg 6y ago

What I like most about some experienced engineers who have been at companies from the beginning is that they realize the needs of a small-team, cash-constrained, rapidly growing company are different from that of a dozens-of-teams, complex, but also rapidly scaling enterprise. The devil is in understanding when and how to transition your tooling, architecture, and — dare I say it — attitudes, from the one to the other. I am curious to hear more from these engineers; how decisions were made about transitioning/scaling; and what decisions were made.

jacquesm 6y ago
This comment is gold. Yes, that's exactly it. Trying to use the tooling and processes that are a bad match for your situation is the way to waste a fortune or to end up being overtaken on the right by a competitor that made smarter choices.
thdrdt 6y ago
There are things that could (should?) be done from the start. Most has to do with automation. For example automated build tools. Maybe you don't want automated unit tests in the beginning but those parts can be added later without changing the company's culture and stucture too much. Or automated invoices. It is much easier to change this than changing a co-worker who manually created all the invoices.
- outworlder 6y ago
  > Most has to do with automation. This. Don't ever put anything into production that's not automated. Ever. "Oh it's a simple HTTP server that's going to be used by 5 people". Cool. Automate it. "Oh but it's going to take time!". I thought you said it was simple... When things are not automated, we treat them differently. If a server is not automated, now it becomes time-consuming to setup another, which means it's now a pet, not cattle. But if it is automated and is misbehaving, we can easily replace it (and save the current for troubleshooting later if necessary). Or we can spin up 100 of them. Same goes for development. If builds are not automated, they become more difficult to do, more error prone, which means people will avoid making contributions unless they are forced to do so (by some ticket). When contributing to a code base that has no guardrails becomes a lose/neutral proposition, people won't want to. And now it's been 5 years and it's difficult to automate things because there's a 100 different workflows that people use and you are going to break them.

AzzieElbab 6y ago

I am curios why they need 1K people instead of 10? What changed on the server side of WhatsApp?

abrolhos 6y ago
I understand that the 1K+ engineer number comes from Facebook as a whole, when mentioning the use of Hack as a typed PHP. More like to give a perspective on the transition on how to come from a small company into a huge one.
- deleted 6y ago
  [deleted]
- user5994461 6y ago
  WhatApps got audio and video calls after the Facebook acquisition. Usage increased by a few orders of magnitude too.
  - stjohnswarts 6y ago
    Yeah I think people get usage confused with number of users.
  - AzzieElbab 6y ago
    That explains it. I haven’t used WhatsApp until few months ago
user5994461 6y ago
No global service can run with 10 people. Just to have a good coverage 24/7 you're looking at a 50 people company.
- oblio 6y ago
  Yeah, just ops would be 3 timezones x 8 hours x 2 (ideally 3) people per timezone. So that's 6-9 people just for a barebones ops team.

stevencorona 6y ago

I absolutely love the BEAM and Elixir, but one of my common complaints and source of errors is the lack of compile-time type sanity checking, so I'm excited to see movement in this direction. Dialyzer is sorta okay, but the speed, cryptic errors, and syntax have kept me from fully embracing it.

findjashua 6y ago
Gleam may be what you're looking for: https://gleam.run/ https://gleam.run/
- faitswulff 6y ago
  I'd be curious how WhatsApp's internal statically typed Erlang solution looks compared to Gleam. If I had to guess based on current trends, I'd guess that it's a gradually typed, opt-in toolchain a la Sorbet or Typescript, in order to maintain backwards compatibility and gradually update their legacy codebase as opposed to a whole different language.
- callamdelaney 6y ago
  This isn't ready for prime time yet iirc.
  - sodapopcan 6y ago
    It's not but is in constant development. The author is very active on elixirforum.com

fbn79 6y ago

Working in a small Company always let me astonished learning things about unicorn apps. But really an application like Whatsapp need 1000+ developers!? I can think about teams about android, ios and web app. X2 if you think about teams for business oriented services. But 1000+ developers for what?

jamestimmins 6y ago
Just some of the needs at that scale: Fraud Security Integrations Infrastructure Data Accessibility Growth I18n API In addition, there won't be individual teams for android, ios, etc. Those domains will be broken across many task-specific teams.
gt565k 6y ago
Whatsapp had a total of 55 employees when it was acquired by facebook for $16B.
MiroF 6y ago
> But 1000+ developers for what? Managerial prestige
ramraj07 6y ago
Even whatsapp the app is deceptively simple. It works seamlessly in all the dialects around the world, on a swathe of devices quite possibly matched by no other app; and it manages to handle every actually useful feature anyone wants in a messaging app with amazing aplomb. In India for the majority of the people whatsapp is not just their communication platform, it's also their photo album, dropbox, task manager, social networking platform and contacts manager. All within the simplest UIs I've ever seen.

derefr 6y ago

Last I heard (years ago), I though that WhatsApp's backend was being rewritten in some other language, because Facebook didn't think maintaining an Erlang codebase was tenable. Did that change?

ramchip 6y ago
Perhaps you're thinking of Facebook chat? https://www.quora.com/Why-was-Erlang-chosen-for-use-in-Facebook-chat https://www.quora.com/Why-was-Erlang-chosen-for-use-in-Faceb... https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-for-Facebook-Chat-What-was-the-reason-for-it-What-did-they-replace-it-with https://www.quora.com/When-did-Facebook-switch-away-from-usi...
deleted 6y ago
[deleted]

tiffanyh 6y ago

> "We are working on a prototype, open-sourcing in November" I'm curious to see how Elixir integration with such static type system would work.

seanclayton 6y ago
I absolutely love Elixir and use it all the time, I really just wish it had a type system (that's not the slow behemoth Dialyzer). I understand the difficulty of typing processes and I don't envy the task, but as a user of the language it would make it that much better to work with! Just imagine working with a language like Elixir that has the type system of F#!
- acjohnson55 6y ago
  Sounds like Scala / Akka to me
rch 6y ago
I think it's more natural to gradually implement components as Erlang NIFs in Rust via https://github.com/rusterlium/rustler https://github.com/rusterlium/rustler.

gautamcgoel 6y ago

Can someone explain what Erlang offers that Go does not? Both have great support for concurrency, but Go has type checking and a more familiar syntax.

emit_time 6y ago
Doesn’t erlang have a REPL, allows you to do live deployments. Individual processes crashing doesn’t bring everything down, you can tell it how to handle those. Dynamic typing. Functional programming. Concise syntax, with “let it fail” mentality.
epidemian 6y ago
There's this talk, The Sould of Erlang and Elixir, that opened my eyes to what makes this platform great: https://youtu.be/JvBT4XBdoUE https://youtu.be/JvBT4XBdoUE It's not so much about what Erlang, as a language, "offers that Go does not". It's about the BEAM VM that runs it that makes all the difference. I very much recommend watching the talk if you're interested in the topic :)
- coder543 6y ago
  I watched that talk just now, and it was basically a retelling of everything I really like about Go. Except, Go is statically typed, and a lot faster than BEAM in benchmarks that I've seen. Go is also a very "normal" language, so it's usually easy to onboard developers into it, no matter what their background is. JetBrains GoLand makes the IDE experience essentially impeccable. I can refactor code with a few clicks, and GoLand precisely updates all references thanks to the static type system. Go can run massive numbers of Goroutines on a single system, and it preemptively timeslices individual Goroutines, so I'm never worried about any single task preventing other tasks from making progress, which was like 75% of that talk. Go has a fantastic production runtime debuggability story, which was the other 25% of that talk. Go's pprof support feels magical: https://golang.org/pkg/net/http/pprof/ https://golang.org/pkg/net/http/pprof/ You can connect to running systems and see flamegraphs of where CPU time is being spent, you can take traces of the goroutines to understand why the system is idle when you think it shouldn't be, you can see how much memory different parts of the system are using, and so much more. It's not mentioned as prominently as I think it should be, but `go tool pprof` has this `-http=:` option that will open a web browser to let you just see everything graphically and click around, and it's an amazing experience. There were really only three things in there that Go doesn't offer: 1. Forcefully terminating processes Go doesn't allow anyone to forcefully terminate a Goroutine except the Goroutine itself. Personally, if I identified a buggy Goroutine, I would just focus on writing a patch and deploying it. Manually terminating a process is cool, but then the user types "-1" again and again, and you're just sitting there playing whack-a-mole instead of fixing the problem and deploying the fix. 2. Patching the running system without restarting it The advantages of this are difficult to enumerate. It sounds nice and clean conceptually, but I would worry about existing processes / goroutines being in unexpected states because of erroneous code that they had previously run. Having a way to gracefully shut down a system and restart it without abruptly terminating responsive client connections is much more predictable to me. 3. Seamlessly distributed execution He briefly talks about distributed Erlang towards the end of the talk. but mentions that it has serious problems and that a number of people don't recommend using it in production. Personally, even if it were nearly perfect, I think the benefits of it are limited. It's not difficult to have nodes communicate with each other using any number of other things, including proper RPC libraries all the way down to just raw TCP connections. The hard thing is service discovery, and I don't think distributed erlang meaningfully solves the problem different than other solutions. If everything is running homogeneous code, why bother your nodes with knowing about the existence of other nodes? Just use a load balancer, and then do everything internally on each node. If your systems are heterogeneous, you need a way to send the right messages to the right nodes, and that's where a named load balancer for each type of node can do just fine.

an_opabinia 6y ago

Erlang is the Actors-model version of Greenspun's tenth rule of programming: "Any sufficiently complicated server backend contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Erlang." Is it really the end-all-be-all model of server architecture? What are microservices, if not Actors Clunkified? "Serverless" functions, if not the simplest actors in disguise? Docker, actors except over IP addresses and clunky APIs? The real takeaway: Valuable stuff is stateful. You know you're doing valuable work for a real human end-user if you are dealing with tricky state problems. Extracting money out of that is a bunch of developer-productivity-limited business concerns. It seems actors are a good model to solve that.

davidw 6y ago
That quote regarding Erlang is attributed to Robert Virding.
acjohnson55 6y ago
I agree that serverless / cloud native feels a lot like what you would build in a distributed monolithic codebase using Erlang or Akka. I guess you get something more polyglot and modular for your trouble, but I think a lot of companies have skipped right over much less complex setups for building resilient, scalable distributed systems.

kornish 6y ago

One of my favorite talks is by Rick Reed about scaling Erlang at WhatsApp. What an absolute savage. He flies through an articulate and in-depth curriculum on system performance and bottleneck mitigation. The talk is called "That's 'Billion' with a 'B'" and makes for a great lunchtime watch: https://www.youtube.com/watch?v=c12cYAUTXXs https://www.youtube.com/watch?v=c12cYAUTXXs

strmpnk 6y ago
I remember that year. I was giving a talk at EF during the same time slot but the schedule originally had me in the large room and they had a much smaller one. When the news of the acquisition hit, everyone wanted to see the WhatsApp talk. The organizers knew this so we swapped rooms. So, I started my talk by asking if anyone in the room was here for the WhatsApp talk and told them they could quietly leave and I wouldn't mind and a bunch of people got up. Heheh. I don't blame them. I didn't really like my talk and Rick Reed is very good at what he does and the talk is no exception.
secondcoming 6y ago
I know nothing about Erlang, but I do know high-volume systems. Needing 550 beefy servers to handle all that load is - in my mind - not that impressive.
staticassertion 6y ago
Yeah, not quite related to the linked slides, which are about scaling in terms of productivity, but still a great talk. I really found that "isolation" is just a key optimization and one of the most useful properties of a system, which they call out in that talk. I wrote about it a bit more in depth here: https://www.graplsecurity.com/post/architecting-for-performance-and-security-at-the-same-time https://www.graplsecurity.com/post/architecting-for-performa... I've written our data processing layer and our event orchestration layer as basically an actor oriented system, with push/pull systems in a couple of key places where it makes sense. It's incredible what you can do with strong isolated infrastructure in terms of performance, security, reliability, and quality.

AshamedCaptain 6y ago

I find it funny they say Whatsapp "chose" Erlang. They "chose" Erlang because ejabberd was written on it, and they chose ejabberd because of a Jabber mailing list suggestion. That's about it...

chii 6y ago
are you saying that the choice was purely due to luck? Would whatsapp have been as popular and successful had it not been in erlang but some other language/stack?
- krn 6y ago
  > are you saying that the choice was purely due to luck? What he is saying is, that they chose the best open source tool available for their use case at that time, and not a programming language. > Would whatsapp have been as popular and successful had it not been in erlang but some other language/stack? I would argue, that they might not have been as efficient in terms of computational resources and the number of technical employees needed, but they would still have succeeded even if the back-end was implemented in Java. Because what all WhatsApp users actually interacted with were the clients – and they were not written in Erlang.

pjmlp 6y ago

> shift to modern languages with integrated tooling, e.g. Erlang competition: > > C++, Java => Go, Rust, Kotlin My first C++ IDE was Turbo C++ 1.0 for MS-DOS in 1993, and Borland, Zortech, Symatec, Microsoft, IBM, and plenty of others had well known C++ IDEs. Many of the ideas of modern C++ IDEs is basically rediscovering the work done by Lucid and IBM with their Smalltalk like capabilities for C++ tooling in the late 80's/early 90's. Java is well known for being one of the languages with best IDE support out of box. It didn't start like that in 1996, but the last 25 years have been good to it. Go and Rust are yet to have similar offerings and Kotlin, well it is a JVM language and profits from the Java eco-system and Google's push to replace it on Android due to non-technical reasons. Maybe better examples should have been provided.

saagarjha 6y ago
Java is interesting because writing code in it is generally absolutely miserable without an IDE.

ramon 6y ago

I think the concepts are very interesting of Erlang but I still don't understand it fully. And also about the multi-node os only an internal network scenario is not like a cloud mesh thing. You would need docker.