17 ms·

Which companies are using Erlang, and why?

394 points by lelf 7y ago

theamk 7y ago
At least some of those examples are highly misleading: > There is no better example of Erlang’s reliability than the English National Health Service (NHS). [...] Using Riak (written in Erlang), the NHS has managed 99.999% availability for over five years. From what I could find out, NHS's Spine2 is mostly Python, with "a bit" of Erlang and Javascript. They do use Riak, but they also use Redis, RabbitMQ, Python/Tornado, Ubuntu and lots of other components. This sentence could have said "Using redis (written in C)", or "Using Tornado (written in Python)", and still be fully correct.
- i_feel_great 7y ago
  Post from the implementing engineer himself: http://erlang.org/pipermail/erlang-questions/2017-December/094421.html http://erlang.org/pipermail/erlang-questions/2017-December/0...
  - theamk 7y ago
    > Almost all of the bespoke code is written in Python, but the message-passing architecture of the system and the approach to managing availability was directly inspired by reading about Erlang (in particular your book). This approach is pretty interesting - I’d love to read about implementing Erlang-style availability in other languages.
    - frobozz 7y ago
      In my experience, the message passing architecture of the NHS system almost exclusively uses Paper.
      - bvm 7y ago
        and fax.
    - i_feel_great 7y ago
      I have been watching this for a while: the Chez Scheme concurrency engine https://github.com/becls/swish https://github.com/becls/swish Another in Gambit Scheme, but it hasn't been touched in a while: https://github.com/mindpool/gambit-termite https://github.com/mindpool/gambit-termite
- ailideex 7y ago
  > the NHS has managed 99.999% availability Really? I am not sure I trust this claim.
  - sealjam 7y ago
    The lead engineer/designer on the project makes this claim in these slides. I don't know whether this gives the claim more veracity in your eyes, but it's there. [1]: http://martinsumner.github.io/presentations/spine2_erlang#/22 http://martinsumner.github.io/presentations/spine2_erlang#/2...
  - masleeds 7y ago
    The claim relates specifically to the spine 2 project, and is as measured over the 5 years since it went live. There are a lot of systems which depend on Spine 2 for their own availability, it is not the sort of thing that can go down at any time without a lot of people noticing. I'm biased in that I've been heavily involved in the project, but I will stand by the five years of five nines availability claim.
  - robertony 7y ago
    My NHS trust doesn't even use Spine 2. That would require reliable Internet.
- melicerte 7y ago
  Note that rabbitmq is also written in Erlang.Furthermore, NHS is also involved in the maintenance of Riak since the collapse of Basho.
  - cies 7y ago
    Cool. I thought it was only some online gambling shop taking up maintenance. Having NHS on that list as well gives me more confidence. Some more info on riak.org about the current story regarding who maintains would help me a lot in choosing it. To me Riak was the only noSQL scale-out-able data store product that really made sense.
- dmitriid 7y ago
  There are two areas where Erlang really works well if you know what you're doing: - telecoms, as it was specifically designed for telecoms. - "glue" in networked applications: to orchestrate, pass messages around and the like. See GitHub [1] or CoD [2] etc. In a sense, chats are a mix of both. Erlang is rarely a good fit in pretty much anything else. [1] https://www.infoq.com/interviews/erlang-and-github/ https://www.infoq.com/interviews/erlang-and-github/ [2] https://www.erlang-factory.com/upload/presentations/395/ErlangandFirst-PersonShooters.pdf https://www.erlang-factory.com/upload/presentations/395/Erla...
  - ahoka 7y ago
    Yes, Erlang was designed for the control plane, not the data plane.
aloknnikhil 7y ago
WhatsApp used to use Erlang, not anymore. Facebook rewrote it completely in C++. https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-for-Facebook-Chat-What-was-the-reason-for-it-What-did-they-replace-it-with https://www.quora.com/When-did-Facebook-switch-away-from-usi...
- i_feel_great 7y ago
  That Quora post refers to Facebook Chat - not WhatsApp. As far as I am aware, it is still implemented in Erlang.
- elgenie 7y ago
  That system was the backend of the functionality currently known as Facebook Messenger, not WhatsApp. One might note, snidely [1], that three-four years after freezing development of an Erlang-based messaging system, starving maintenance work on that system of engineering effort, devoting massive engineering resources to a from-scratch C++ rewrite, and in the meantime blaming the language for relatively minor system design issues that could have been improved with a fraction of the effort (including, but not limited to, Erlang's ability to wrap allegedly critical C++ components) … Facebook plowed $19B into the acquisition of an Erlang-based messaging system. [1] as a main author of the Facebook Chat version written in Erlang
  - dmitriid 7y ago
    > after freezing development of an Erlang-based messaging system... > Facebook plowed $19B into the acquisition of an Erlang-based messaging system. They didn't care what it was written in. They would've spent it on Whatsapp even it was written in PHP (like Slack). They paid for users and market penetration.
    - rdtsc 7y ago
      The irony is still there though
- bdd 7y ago
latentpot 7y ago
IIRC a clearing co. CLS used erlang a few years ago. LinkedIn still has the erlang architect JDs somewhere.
atupis 7y ago
I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages.
- ailideex 7y ago
  > It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages. Like what ?
  - quaunaut 7y ago
    The fact that the entire ecosystem is based around the actor model is one of the primary ones. It's allowed for code that isn't simply a library, but effectively an entire application, to be used as open source to be dumped into your system, because it has tools like Erlang Term Storage, Mnesia, and other devices to make it easy.
    - hota_mazi 7y ago
      You're still not explaining why this allows to do things that can't be done in other languages. Probably because that claim makes no sense.
      - nathanappere 7y ago
        Reductions. How do you ensure in other languages (or in other VMs) that every actor will get the same cpu time and that none can block the others? That's a tricky problem to solve as a lib.
  - atupis 7y ago
    I think Whatsapp served "at peak" like 450 million users with only 32 engineers.
    - dmitriid 7y ago
jhgg 7y ago
Although not in Erlang, Discord uses Elixir which runs on the BEAM VM to much success as well for our real time distributed system as well as VOIP signaling components. Looking back years later - it’s safe to say we wouldn’t have chosen anything else.
- erk__ 7y ago
  What was the reason to choose Elixir over Erlang? Just familiarity with the language?
  - superhawk610 7y ago
    Not OP, but have used Elixir pretty extensively; the major selling points for me are the friendlier syntax and more active community support, plus with rebar3 + Elixir's seamless Erlang interop you really don't give up anything, you can use any existing Erlang/Elixir libraries with a single syntax. It's pretty great.
    - alskdj21 7y ago
      Haven't touched Erlang but has an entry level knowledge on Elixir. I often heard how easy it is to write macros on Elixir. Can you elaborate on this?
      - arjan_sch 7y ago
        One of the big differences between Erlang and Elixir is indeed that Elixir allows you to write macros in the language itself, using quote / unquote constructs just like lisps. This makes it easy to generate code on compile time, which is then executed in runtime without any performance penalty. A large part of Elixir itself is actually implemented as macros. For instance, the "unless" construct: defmacro unless(condition, do: do_clause, else: else_clause) do quote do if(unquote(condition), do: unquote(else_clause), else: unquote(do_clause)) end end (code simplified for clarity)
        7y ago
MetalMASK 7y ago
FWIW, wechat's messaging server is in erlang. In terms of scale, it's probably bigger than whatsapp.
- kgc 7y ago
  Interesting. Source?
- mcqueenjordan 7y ago
  The opposite is true by a non trivial margin. (It doesn’t change the fact that WeChat is massive scale.) https://www.statista.com/statistics/258749/most-popular-global-mobile-messenger-apps/ https://www.statista.com/statistics/258749/most-popular-glob...
  - erk__ 7y ago
    Then the question comes if wechat and qq uses the same backend as they are owned by the same company.
  - mappu 7y ago
    The Discord number is inflated - 250mm is registered users, not MAUs. When they announced 250mm registered, they had 56mm MAUs. Telegram's 200mm is MAUs, and that number is over 1 year older than Discord's information.
- yashap 7y ago
  WhatsApp’s servers are largely written in Erlang too, right?
  - fock 7y ago
    they were ejabberd when they started up ;)
- tulunsutao 7y ago
  No, wechat's infra is most in C++[0], e.g phx*[1] [0] https://github.com/Tencent?q=wechat https://github.com/Tencent?q=wechat [1] https://github.com/Tencent?q=phx https://github.com/Tencent?q=phx
known 7y ago
Telecom companies
- melicerte 7y ago
  So you think gaming, online bets, financial institutions or NHS are telecom companies?
  - deleted 7y ago
    [deleted]
  - hsson 7y ago
    I'm fairly certain he/she refers to Ericsson, the creator of Erlang.
vasilakisfil 7y ago
worked with Erlang lately in a side project, I think the actor model is really cool (and the FP style), but the error messages are not helpful at all (at least for a beginner), I wonder how other people manage.
- rapsey 7y ago
  Runtime error messages or compilation error messages? I assume compilation. This is true. Compilation messages are bare bones. But on the other hand the language compared to other modern languages is tiny and figuring out why something does not compile is a non-issue beyond an absolute beginner.
- samvher 7y ago
  I'm using Erlang for a production system at work and I completely agree, the error messages tend to be terrible. Once you've run into many errors you start recognizing patterns though (part of the problem is that the errors are not English sentences but data dumps with indicators like `badfun`, these you just have to learn to interpret) and Erlang makes up for the bad error messages in other ways. But yes, often I end up reasoning through the code to solve the problem, with the error message only roughly pointing me in the right direction, to a larger extent than in other languages.
- joekinley 7y ago
  More often than not after I did figure out what the error was, looking back at the initial error message it told me exactly what I should have looked at to begin with. So, if you can read them right, they are pretty good. I use Elixir in production and Erlang in side projects.
- fyfy18 7y ago
  Elixir gives you much more friendly error messages at runtime (I believe part of this feature was contributed to the Erlang VM, so improved it for all languages): 10:21:21.525 mid=1 [error] GenServer {Registry.WatcherManagerRegistry, 1} terminating ** (CaseClauseError) no case clause matching: {:ok, <snip>} (core) lib/core/http.ex:64: Core.Http.request/7 (core) lib/core/monitors/built_in_monitor.ex:99: Core.Monitors.BuiltInMonitor.perform_request/1 (core) lib/core/monitors/built_in_monitor.ex:76: Core.Monitors.BuiltInMonitor.perform/1 (core) lib/core/watcher.ex:40: Core.Watcher.perform_check/1 (core) lib/core/watcher_server.ex:91: Core.WatcherServer.handle_info/2
z3t4 7y ago
Something I find interesting is that the tech stack is often chosen by non-engineers/founders. So languages and frameworks are marketed to business people. Freelance work is often weak on the requirements, but they have strong opinions on what language and framework to use.
- lagadu 7y ago
  That's often because most clients will already have some sort of environment set up, running some OS and with certain features installed and most importantly: they don't want to have 5 different projects working on 5 different frameworks because that'd be a nightmare to maintain. Once a company gets into a tech stack, they'll want to stay within it as much as possible and that makes plenty of sense.
  - z3t4 7y ago
    You dont want to put all your eggs in the same basket, not in this age when a framework can become obsolete in just two years. It only make sense if you already have a small team - you want to use what they know best. But if you are growing you can have many small teams. You will also exhaust the talent supply if you lock yourself down to only one platform/language/framework.
    - oblio 7y ago
      Yeah, tell that to Java/Spring or C#/ASP :-)
atoav 7y ago
I have seen that Erlang is used in RabbitMQ, if I recall right their argument was that Erlang is particularily well suited for that kind of asynchonous message queue stuff. I donlt know all too much about Erlang, so I cannot judge if that claim makes sense.
- zbentley 7y ago
  I'm an Erlang novice, so take this with a grain of salt: Erlang's use in RMQ seems to have helped them immensely through the first few years of the system's life. They got an "all in one box" performant message broker off the ground, and relatively reliable, quite quickly. However, RMQ did hit numerous issues that they had to work around on the Erlang platform. They had to engineer what was effectively their own scheduler on top of parts of the native Erlang one in order to prevent some starvation cases [1], and I have also heard that issues with Mnesia are at the root of some of the pathological behaviors (data loss) of some versions of RabbitMQ when restarting from a crash (this is admittedly anecdotal, so take it with a grain of salt). [1] http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-f...
dominicl 7y ago
Having worked extensively with both Erlang and Elixir in the last 7 years, the main advantage for me has always been the smaller code base that those projects tend to produce when compared to other languages. Especially for projects that have some long-running components to them the Erlang built-in features allow writing really short yet fault resistant systems. The Supervisor system, the per process garbage collection (no stop the world), 'start_link', "Let it crash", and pattern matching are for me the superpowers of beam. Elixir is contributing a great package ecosystem, 'mix format', and better features for the new live web of websockets
- verttii 7y ago
  Interesting. Any input on erlang vs elixir debate? Which so you prefer as your primary tool personally?
  - fouc 7y ago
    You can easily drop into erlang from elixir, so you can have you cake and eat it too.
    - sntran 7y ago
      Unfortunately the opposite way is not possible,yet.
mesid 7y ago
How does Erlang compare with Go?
wprapido 7y ago
teamweek.com uses Elixir
truth_seeker 7y ago
Is there any official source which talks about the number of servers or total CPU cores used in production by the companies mentioned in the article?
ihuk 7y ago
We developed Online Charging System in Erlang that served couple million subscribers for close to three years. I found the whole experience fairly terrible. Erlang is nice enough language and I don't mind the syntax but it's also kinda cumbersome and verbose at times. For example, adding `true -> ok` to every if statement gets old fast. Similarly, Erlang/OPT is a nice platform but some parts are fairly bad. Looking at you `inets/httpc`. But the really big problem was the whole ecosystem. Build process was needlessly complicated. Surprisingly enough multicore support was not great. Performance was not that great. It seemed like a lot of libraries were abandoned after 2013. I could go on. We ended up rewriting the whole thing in Java and, so far, it worked out great. And after Java 8, Java the language is not so bad. I still have somewhat found memories of Erlang but I don't miss it.
- mapcars 7y ago
  Interesting, do you think Elixir would be better in your case? Also what was the amount of requests per second? Looks like you didn't really need Erlang's scalability.
  - ihuk 7y ago
    No; as I understand it Elixir's only advantage is, arguably, better syntax. I don't mind Erlang's syntax that much. We were handling, in peek hours, over 100k requests per second spread over 3 servers. Funny you should mention scalability as something that somehow justifies Erlang's other shortcomings. I don't think scalability is something Erlang/OTP does out of the bag. One thing we learned the hard way is that Erlang does not do overload well at all. Erlang mailboxes are unbounded and it can get very messy very fast. I always found that design choice odd given Erlang's origins in telecommunications. Now this is not to say you can't do massively scalable systems in Erlang, it's just that there's no silver bullet when it comes to scalability.
    - mac01021 7y ago
      > We were handling, in peek hours, over 100k requests per second spread over 3 servers. How big was one server and what does one request entail?
aloer 7y ago
What would be a good use case to learn erlang/elixir with that is not just a chat app? Online gaming? Collaborative tools?
- chrisbro 7y ago
  RabbitMQ is written in erlang and is popular, so perhaps a queueing system? https://en.wikipedia.org/wiki/RabbitMQ https://en.wikipedia.org/wiki/RabbitMQ
- bobwaycott 7y ago
  If you really want to find yourself digging into a lot of different features in the Elixir ecosystem, you can’t go wrong with collaborative tools. If you really want to play with the latest hotness in Elixir, and dig into a challenge or two, I’d suggest building a light multi-user blogging system that leverages Phoenix LiveView for real-time Markdown->HTML previews, while allowing simultaneous editing/previewing by two users. Put a DB behind it so you can also play with Ecto. Outside of the 3 lines of JS you need to wire up LiveView, limit yourself to writing 0 additional lines of JS. Do everything in Elixir and eex/leex templates only. Since you’re focused on learning, skip dealing with user auth stuff. There will be some challenge to deal with handling multiple edits at the same time. May as well play with doing DB persistence in your LiveView, without any refresh. :) Online gaming can be a good option—if building games is your thing. Personally, I find game-based demos/exercises with certain tools to be a bit too far removed from my daily work to help certain concepts really take root.
- kerkeslager 7y ago
  A Bittorrent client is a pretty good one. The protocol is fairly well-documented, and the domain is very well-suited to the language.
  - vonseel 7y ago
    I'd use this if someone made a good one! I'm currently using Transmission on Mac, which has more than its own fair share of issues, most notably the UI freezes in any action that takes time, like moving a file across hard disks or to a network drive, but IIRC all the others I have tried (Deluge, qBittorrent) also have multiple similar issues.
HeXetic 7y ago
My only exposure to Erlang has been through RabbitMQ and while I can't comment much on the language itself, managing the packages is a dependency nightmare. Like some twisted, mutant, supervillain version of Python, a given version of RabbitMQ will only work with a certain version of Erlang (with horrible, hard-to-understand error messages on incorrect versions), and on RedHat/CentOS, the Erlang packages all misadvertise what exact versions they supply, necessitating awkward forcing of the dependencies or else Yum screws everything up.
- not_a_cop75 7y ago
  You might be right, but what you say sounds like it's coming from a true outsider. What would someone within the community say?
- liveoneggs 7y ago
  why wouldn't you just use rabbit rpms that depend on the exact correct versions of erlang? aka els-erlang https://www.erlang-solutions.com/resources/download.html https://www.erlang-solutions.com/resources/download.html Linux distros have always been terrible places to find reliable packages.
  - oblio 7y ago
    He he. And then you have people complaining that you're a "bad OSS citizen"...
    - liveoneggs 7y ago
      me, ELS, or redhat?
aloukissas 7y ago
Awesome write-up, Francesco! To add to the success stories, our team's previous startup (Bugsense) was a huge user of Erlang, which allowed to scale to hundreds of thousands of concurrent devices with just a handful of servers [1]. It was a no brainer that we would now use Elixir for our current startup (AgentRisk [2,3]) and we wouldn't have been happier with our choice, especially paired with Phoenix, which is hands-down the best web framework we've ever worked with. [1] http://www.erlang-factory.com/static/upload/media/1394234889981861bugsenseldbefsf2014.pdf http://www.erlang-factory.com/static/upload/media/1394234889... [2] https://agentrisk.com https://agentrisk.com [3] https://blog.agentrisk.com https://blog.agentrisk.com
- interactivecode 7y ago
  What would be a good resource to get started with Phoenix / Elixir, coming from Node/EJS/React it seems very daunting
kitplummer 7y ago
Fairly blind question: does Erlang|Elixir and BEAM make any sense in a pseudo-embedded, edge compute or IoT environment with constrained resources (CPU and memory), or even ARM architecture? Aside from the developer-side things, if AMQP were to the centralized communications hub (e.g. RabbitMQ) does using Nerves on devices make any performance benefit - meaning BEAM and app(s) there? The intent would be pre-process data and stream inward, or simply be a command-and-control interface -> process and provide response.
- Globz 7y ago
  Yes, take a look at nerves : https://nerves-project.org/ https://nerves-project.org/
  - kitplummer 7y ago
    Yep, know Nerves. Like the toolset and the model. Really asking about the underlying Erlang/Elixir that Nerves provides. The delivery flow (think image to SD card) fits the more tradition embedded/realtime model - versus OS, and installing app.
    - Globz 7y ago
      Well a Nerves image is a Unix Kernel + Erlang/Elixir & BEAM and whatever else you need, so its a completely stripped down image. It uses the Whitelist model so you add what you need instead of removing what you don't need. This is both more secure and more performant since you don't have 100's of apps/services running in the background, you only run what you need. You can control everything remotely with nerves-hub, push update, debug etc. Remember you only have to burn the image once, then its all update from here. You can build a poncho application where everything is side-by-side, lets say you have your app-firmware (you build image from this), then you build separate app-ui, app-logic, app-whatever etc. and simply hard code the path as dependencies in app-firmware and each app has their own configuration. Everything will be bundled into a single image and protects you from things leaking in from the sides rather than from above.
- derpherpsson 7y ago
mc_ 7y ago
We're happily plugging along on KAZOO [0] since 2010 at 2600Hz. Erlang is a force-multiplier for us. Approaching 300K LOC and ~10 Erlang engineers (depends if Karl is writing code or CTO-ing that day) today. [0] https://github.com/2600hz/kazoo/ https://github.com/2600hz/kazoo/
tickbw 7y ago
I have been building grouper.ai to integrate voice ai systems with 2600hz's kazoo apis, and have enjoyed doing it as a Greenfield side project. Nerves for hardware on embedded Linux, c/c++ for the voice apis, ports in erlang/elixir to talk the low level calls, using elixir Phoenix in the cloud on the beam. Have used erlang also in day job for custom industrial M2M protocol translation (serial / Ethernet) very successfully also.
miki123211 7y ago
> Many countries rely on Erlang for their immediate payment switches, allowing instant bank transfers Just wondering if such a system used in Poland, actually called Elixir[1], has a line of Erlang/Elixir in its codebase. [1] https://pl.wikipedia.org/wiki/Elixir_(system_rozliczeń_międzybankowych) https://pl.wikipedia.org/wiki/Elixir_(system_rozliczeń_międz...
fastbmk 7y ago
Observed facts of Google Trends: 1. Erlang/Elixir are less than 1% of programming languages market. 2. Erlang/Elixir popularity is trending down.
- therockhead 7y ago
  Elixirforums have reported year on years increased traffic, so I’m surprised to hear that.

394 points by lelf 7y ago

At least some of those examples are highly misleading: > There is no better example of Erlang’s reliability than the English National Health Service (NHS). [...] Using Riak (written in Erlang), the NHS has managed 99.999% availability for over five years. From what I could find out, NHS's Spine2 is mostly Python, with "a bit" of Erlang and Javascript. They do use Riak, but they also use Redis, RabbitMQ, Python/Tornado, Ubuntu and lots of other components. This sentence could have said "Using redis (written in C)", or "Using Tornado (written in Python)", and still be fully correct.

i_feel_great 7y ago
Post from the implementing engineer himself: http://erlang.org/pipermail/erlang-questions/2017-December/094421.html http://erlang.org/pipermail/erlang-questions/2017-December/0...
- theamk 7y ago
  > Almost all of the bespoke code is written in Python, but the message-passing architecture of the system and the approach to managing availability was directly inspired by reading about Erlang (in particular your book). This approach is pretty interesting - I’d love to read about implementing Erlang-style availability in other languages.
  - frobozz 7y ago
    In my experience, the message passing architecture of the NHS system almost exclusively uses Paper.
    - bvm 7y ago
      and fax.
  - i_feel_great 7y ago
    I have been watching this for a while: the Chez Scheme concurrency engine https://github.com/becls/swish https://github.com/becls/swish Another in Gambit Scheme, but it hasn't been touched in a while: https://github.com/mindpool/gambit-termite https://github.com/mindpool/gambit-termite
ailideex 7y ago
> the NHS has managed 99.999% availability Really? I am not sure I trust this claim.
- sealjam 7y ago
  The lead engineer/designer on the project makes this claim in these slides. I don't know whether this gives the claim more veracity in your eyes, but it's there. [1]: http://martinsumner.github.io/presentations/spine2_erlang#/22 http://martinsumner.github.io/presentations/spine2_erlang#/2...
- masleeds 7y ago
  The claim relates specifically to the spine 2 project, and is as measured over the 5 years since it went live. There are a lot of systems which depend on Spine 2 for their own availability, it is not the sort of thing that can go down at any time without a lot of people noticing. I'm biased in that I've been heavily involved in the project, but I will stand by the five years of five nines availability claim.
- robertony 7y ago
  My NHS trust doesn't even use Spine 2. That would require reliable Internet.
melicerte 7y ago
Note that rabbitmq is also written in Erlang.Furthermore, NHS is also involved in the maintenance of Riak since the collapse of Basho.
- cies 7y ago
  Cool. I thought it was only some online gambling shop taking up maintenance. Having NHS on that list as well gives me more confidence. Some more info on riak.org about the current story regarding who maintains would help me a lot in choosing it. To me Riak was the only noSQL scale-out-able data store product that really made sense.
dmitriid 7y ago
There are two areas where Erlang really works well if you know what you're doing: - telecoms, as it was specifically designed for telecoms. - "glue" in networked applications: to orchestrate, pass messages around and the like. See GitHub [1] or CoD [2] etc. In a sense, chats are a mix of both. Erlang is rarely a good fit in pretty much anything else. [1] https://www.infoq.com/interviews/erlang-and-github/ https://www.infoq.com/interviews/erlang-and-github/ [2] https://www.erlang-factory.com/upload/presentations/395/ErlangandFirst-PersonShooters.pdf https://www.erlang-factory.com/upload/presentations/395/Erla...
- ahoka 7y ago
  Yes, Erlang was designed for the control plane, not the data plane.

aloknnikhil 7y ago

WhatsApp used to use Erlang, not anymore. Facebook rewrote it completely in C++. https://www.quora.com/When-did-Facebook-switch-away-from-using-Erlang-for-Facebook-Chat-What-was-the-reason-for-it-What-did-they-replace-it-with https://www.quora.com/When-did-Facebook-switch-away-from-usi...

i_feel_great 7y ago
That Quora post refers to Facebook Chat - not WhatsApp. As far as I am aware, it is still implemented in Erlang.
elgenie 7y ago
That system was the backend of the functionality currently known as Facebook Messenger, not WhatsApp. One might note, snidely [1], that three-four years after freezing development of an Erlang-based messaging system, starving maintenance work on that system of engineering effort, devoting massive engineering resources to a from-scratch C++ rewrite, and in the meantime blaming the language for relatively minor system design issues that could have been improved with a fraction of the effort (including, but not limited to, Erlang's ability to wrap allegedly critical C++ components) … Facebook plowed $19B into the acquisition of an Erlang-based messaging system. [1] as a main author of the Facebook Chat version written in Erlang
- dmitriid 7y ago
  > after freezing development of an Erlang-based messaging system... > Facebook plowed $19B into the acquisition of an Erlang-based messaging system. They didn't care what it was written in. They would've spent it on Whatsapp even it was written in PHP (like Slack). They paid for users and market penetration.
  - rdtsc 7y ago
    The irony is still there though
bdd 7y ago

latentpot 7y ago

IIRC a clearing co. CLS used erlang a few years ago. LinkedIn still has the erlang architect JDs somewhere.

atupis 7y ago

I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages.

ailideex 7y ago
> It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages. Like what ?
- quaunaut 7y ago
  The fact that the entire ecosystem is based around the actor model is one of the primary ones. It's allowed for code that isn't simply a library, but effectively an entire application, to be used as open source to be dumped into your system, because it has tools like Erlang Term Storage, Mnesia, and other devices to make it easy.
  - hota_mazi 7y ago
    You're still not explaining why this allows to do things that can't be done in other languages. Probably because that claim makes no sense.
    - nathanappere 7y ago
      Reductions. How do you ensure in other languages (or in other VMs) that every actor will get the same cpu time and that none can block the others? That's a tricky problem to solve as a lib.
- atupis 7y ago
  I think Whatsapp served "at peak" like 450 million users with only 32 engineers.
  - dmitriid 7y ago

jhgg 7y ago

Although not in Erlang, Discord uses Elixir which runs on the BEAM VM to much success as well for our real time distributed system as well as VOIP signaling components. Looking back years later - it’s safe to say we wouldn’t have chosen anything else.

erk__ 7y ago
What was the reason to choose Elixir over Erlang? Just familiarity with the language?
- superhawk610 7y ago
  Not OP, but have used Elixir pretty extensively; the major selling points for me are the friendlier syntax and more active community support, plus with rebar3 + Elixir's seamless Erlang interop you really don't give up anything, you can use any existing Erlang/Elixir libraries with a single syntax. It's pretty great.
  - alskdj21 7y ago
    Haven't touched Erlang but has an entry level knowledge on Elixir. I often heard how easy it is to write macros on Elixir. Can you elaborate on this?
    - arjan_sch 7y ago
      One of the big differences between Erlang and Elixir is indeed that Elixir allows you to write macros in the language itself, using quote / unquote constructs just like lisps. This makes it easy to generate code on compile time, which is then executed in runtime without any performance penalty. A large part of Elixir itself is actually implemented as macros. For instance, the "unless" construct: defmacro unless(condition, do: do_clause, else: else_clause) do quote do if(unquote(condition), do: unquote(else_clause), else: unquote(do_clause)) end end (code simplified for clarity)
      - 7y ago

MetalMASK 7y ago

FWIW, wechat's messaging server is in erlang. In terms of scale, it's probably bigger than whatsapp.

kgc 7y ago
Interesting. Source?
mcqueenjordan 7y ago
The opposite is true by a non trivial margin. (It doesn’t change the fact that WeChat is massive scale.) https://www.statista.com/statistics/258749/most-popular-global-mobile-messenger-apps/ https://www.statista.com/statistics/258749/most-popular-glob...
- erk__ 7y ago
  Then the question comes if wechat and qq uses the same backend as they are owned by the same company.
- mappu 7y ago
  The Discord number is inflated - 250mm is registered users, not MAUs. When they announced 250mm registered, they had 56mm MAUs. Telegram's 200mm is MAUs, and that number is over 1 year older than Discord's information.
yashap 7y ago
WhatsApp’s servers are largely written in Erlang too, right?
- fock 7y ago
  they were ejabberd when they started up ;)
tulunsutao 7y ago
No, wechat's infra is most in C++[0], e.g phx*[1] [0] https://github.com/Tencent?q=wechat https://github.com/Tencent?q=wechat [1] https://github.com/Tencent?q=phx https://github.com/Tencent?q=phx

known 7y ago

Telecom companies

melicerte 7y ago
So you think gaming, online bets, financial institutions or NHS are telecom companies?
- deleted 7y ago
  [deleted]
- hsson 7y ago
  I'm fairly certain he/she refers to Ericsson, the creator of Erlang.

vasilakisfil 7y ago

worked with Erlang lately in a side project, I think the actor model is really cool (and the FP style), but the error messages are not helpful at all (at least for a beginner), I wonder how other people manage.

rapsey 7y ago
Runtime error messages or compilation error messages? I assume compilation. This is true. Compilation messages are bare bones. But on the other hand the language compared to other modern languages is tiny and figuring out why something does not compile is a non-issue beyond an absolute beginner.
samvher 7y ago
I'm using Erlang for a production system at work and I completely agree, the error messages tend to be terrible. Once you've run into many errors you start recognizing patterns though (part of the problem is that the errors are not English sentences but data dumps with indicators like `badfun`, these you just have to learn to interpret) and Erlang makes up for the bad error messages in other ways. But yes, often I end up reasoning through the code to solve the problem, with the error message only roughly pointing me in the right direction, to a larger extent than in other languages.
joekinley 7y ago
More often than not after I did figure out what the error was, looking back at the initial error message it told me exactly what I should have looked at to begin with. So, if you can read them right, they are pretty good. I use Elixir in production and Erlang in side projects.
fyfy18 7y ago
Elixir gives you much more friendly error messages at runtime (I believe part of this feature was contributed to the Erlang VM, so improved it for all languages): 10:21:21.525 mid=1 [error] GenServer {Registry.WatcherManagerRegistry, 1} terminating ** (CaseClauseError) no case clause matching: {:ok, <snip>} (core) lib/core/http.ex:64: Core.Http.request/7 (core) lib/core/monitors/built_in_monitor.ex:99: Core.Monitors.BuiltInMonitor.perform_request/1 (core) lib/core/monitors/built_in_monitor.ex:76: Core.Monitors.BuiltInMonitor.perform/1 (core) lib/core/watcher.ex:40: Core.Watcher.perform_check/1 (core) lib/core/watcher_server.ex:91: Core.WatcherServer.handle_info/2

z3t4 7y ago

Something I find interesting is that the tech stack is often chosen by non-engineers/founders. So languages and frameworks are marketed to business people. Freelance work is often weak on the requirements, but they have strong opinions on what language and framework to use.

lagadu 7y ago
That's often because most clients will already have some sort of environment set up, running some OS and with certain features installed and most importantly: they don't want to have 5 different projects working on 5 different frameworks because that'd be a nightmare to maintain. Once a company gets into a tech stack, they'll want to stay within it as much as possible and that makes plenty of sense.
- z3t4 7y ago
  You dont want to put all your eggs in the same basket, not in this age when a framework can become obsolete in just two years. It only make sense if you already have a small team - you want to use what they know best. But if you are growing you can have many small teams. You will also exhaust the talent supply if you lock yourself down to only one platform/language/framework.
  - oblio 7y ago
    Yeah, tell that to Java/Spring or C#/ASP :-)

atoav 7y ago

I have seen that Erlang is used in RabbitMQ, if I recall right their argument was that Erlang is particularily well suited for that kind of asynchonous message queue stuff. I donlt know all too much about Erlang, so I cannot judge if that claim makes sense.

zbentley 7y ago
I'm an Erlang novice, so take this with a grain of salt: Erlang's use in RMQ seems to have helped them immensely through the first few years of the system's life. They got an "all in one box" performant message broker off the ground, and relatively reliable, quite quickly. However, RMQ did hit numerous issues that they had to work around on the Erlang platform. They had to engineer what was effectively their own scheduler on top of parts of the native Erlang one in order to prevent some starvation cases [1], and I have also heard that issues with Mnesia are at the root of some of the pathological behaviors (data loss) of some versions of RabbitMQ when restarting from a crash (this is admittedly anecdotal, so take it with a grain of salt). [1] http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-f...

dominicl 7y ago

Having worked extensively with both Erlang and Elixir in the last 7 years, the main advantage for me has always been the smaller code base that those projects tend to produce when compared to other languages. Especially for projects that have some long-running components to them the Erlang built-in features allow writing really short yet fault resistant systems. The Supervisor system, the per process garbage collection (no stop the world), 'start_link', "Let it crash", and pattern matching are for me the superpowers of beam. Elixir is contributing a great package ecosystem, 'mix format', and better features for the new live web of websockets

verttii 7y ago
Interesting. Any input on erlang vs elixir debate? Which so you prefer as your primary tool personally?
- fouc 7y ago
  You can easily drop into erlang from elixir, so you can have you cake and eat it too.
  - sntran 7y ago
    Unfortunately the opposite way is not possible,yet.

mesid 7y ago

How does Erlang compare with Go?

wprapido 7y ago

teamweek.com uses Elixir

truth_seeker 7y ago

Is there any official source which talks about the number of servers or total CPU cores used in production by the companies mentioned in the article?

ihuk 7y ago

We developed Online Charging System in Erlang that served couple million subscribers for close to three years. I found the whole experience fairly terrible. Erlang is nice enough language and I don't mind the syntax but it's also kinda cumbersome and verbose at times. For example, adding `true -> ok` to every if statement gets old fast. Similarly, Erlang/OPT is a nice platform but some parts are fairly bad. Looking at you `inets/httpc`. But the really big problem was the whole ecosystem. Build process was needlessly complicated. Surprisingly enough multicore support was not great. Performance was not that great. It seemed like a lot of libraries were abandoned after 2013. I could go on. We ended up rewriting the whole thing in Java and, so far, it worked out great. And after Java 8, Java the language is not so bad. I still have somewhat found memories of Erlang but I don't miss it.

mapcars 7y ago
Interesting, do you think Elixir would be better in your case? Also what was the amount of requests per second? Looks like you didn't really need Erlang's scalability.
- ihuk 7y ago
  No; as I understand it Elixir's only advantage is, arguably, better syntax. I don't mind Erlang's syntax that much. We were handling, in peek hours, over 100k requests per second spread over 3 servers. Funny you should mention scalability as something that somehow justifies Erlang's other shortcomings. I don't think scalability is something Erlang/OTP does out of the bag. One thing we learned the hard way is that Erlang does not do overload well at all. Erlang mailboxes are unbounded and it can get very messy very fast. I always found that design choice odd given Erlang's origins in telecommunications. Now this is not to say you can't do massively scalable systems in Erlang, it's just that there's no silver bullet when it comes to scalability.
  - mac01021 7y ago
    > We were handling, in peek hours, over 100k requests per second spread over 3 servers. How big was one server and what does one request entail?

aloer 7y ago

What would be a good use case to learn erlang/elixir with that is not just a chat app? Online gaming? Collaborative tools?

chrisbro 7y ago
RabbitMQ is written in erlang and is popular, so perhaps a queueing system? https://en.wikipedia.org/wiki/RabbitMQ https://en.wikipedia.org/wiki/RabbitMQ
bobwaycott 7y ago
If you really want to find yourself digging into a lot of different features in the Elixir ecosystem, you can’t go wrong with collaborative tools. If you really want to play with the latest hotness in Elixir, and dig into a challenge or two, I’d suggest building a light multi-user blogging system that leverages Phoenix LiveView for real-time Markdown->HTML previews, while allowing simultaneous editing/previewing by two users. Put a DB behind it so you can also play with Ecto. Outside of the 3 lines of JS you need to wire up LiveView, limit yourself to writing 0 additional lines of JS. Do everything in Elixir and eex/leex templates only. Since you’re focused on learning, skip dealing with user auth stuff. There will be some challenge to deal with handling multiple edits at the same time. May as well play with doing DB persistence in your LiveView, without any refresh. :) Online gaming can be a good option—if building games is your thing. Personally, I find game-based demos/exercises with certain tools to be a bit too far removed from my daily work to help certain concepts really take root.
kerkeslager 7y ago
A Bittorrent client is a pretty good one. The protocol is fairly well-documented, and the domain is very well-suited to the language.
- vonseel 7y ago
  I'd use this if someone made a good one! I'm currently using Transmission on Mac, which has more than its own fair share of issues, most notably the UI freezes in any action that takes time, like moving a file across hard disks or to a network drive, but IIRC all the others I have tried (Deluge, qBittorrent) also have multiple similar issues.

HeXetic 7y ago

My only exposure to Erlang has been through RabbitMQ and while I can't comment much on the language itself, managing the packages is a dependency nightmare. Like some twisted, mutant, supervillain version of Python, a given version of RabbitMQ will only work with a certain version of Erlang (with horrible, hard-to-understand error messages on incorrect versions), and on RedHat/CentOS, the Erlang packages all misadvertise what exact versions they supply, necessitating awkward forcing of the dependencies or else Yum screws everything up.

not_a_cop75 7y ago
You might be right, but what you say sounds like it's coming from a true outsider. What would someone within the community say?
liveoneggs 7y ago
why wouldn't you just use rabbit rpms that depend on the exact correct versions of erlang? aka els-erlang https://www.erlang-solutions.com/resources/download.html https://www.erlang-solutions.com/resources/download.html Linux distros have always been terrible places to find reliable packages.
- oblio 7y ago
  He he. And then you have people complaining that you're a "bad OSS citizen"...
  - liveoneggs 7y ago
    me, ELS, or redhat?

aloukissas 7y ago

Awesome write-up, Francesco! To add to the success stories, our team's previous startup (Bugsense) was a huge user of Erlang, which allowed to scale to hundreds of thousands of concurrent devices with just a handful of servers [1]. It was a no brainer that we would now use Elixir for our current startup (AgentRisk [2,3]) and we wouldn't have been happier with our choice, especially paired with Phoenix, which is hands-down the best web framework we've ever worked with. [1] http://www.erlang-factory.com/static/upload/media/1394234889981861bugsenseldbefsf2014.pdf http://www.erlang-factory.com/static/upload/media/1394234889... [2] https://agentrisk.com https://agentrisk.com [3] https://blog.agentrisk.com https://blog.agentrisk.com

interactivecode 7y ago
What would be a good resource to get started with Phoenix / Elixir, coming from Node/EJS/React it seems very daunting

kitplummer 7y ago

Fairly blind question: does Erlang|Elixir and BEAM make any sense in a pseudo-embedded, edge compute or IoT environment with constrained resources (CPU and memory), or even ARM architecture? Aside from the developer-side things, if AMQP were to the centralized communications hub (e.g. RabbitMQ) does using Nerves on devices make any performance benefit - meaning BEAM and app(s) there? The intent would be pre-process data and stream inward, or simply be a command-and-control interface -> process and provide response.

Globz 7y ago
Yes, take a look at nerves : https://nerves-project.org/ https://nerves-project.org/
- kitplummer 7y ago
  Yep, know Nerves. Like the toolset and the model. Really asking about the underlying Erlang/Elixir that Nerves provides. The delivery flow (think image to SD card) fits the more tradition embedded/realtime model - versus OS, and installing app.
  - Globz 7y ago
    Well a Nerves image is a Unix Kernel + Erlang/Elixir & BEAM and whatever else you need, so its a completely stripped down image. It uses the Whitelist model so you add what you need instead of removing what you don't need. This is both more secure and more performant since you don't have 100's of apps/services running in the background, you only run what you need. You can control everything remotely with nerves-hub, push update, debug etc. Remember you only have to burn the image once, then its all update from here. You can build a poncho application where everything is side-by-side, lets say you have your app-firmware (you build image from this), then you build separate app-ui, app-logic, app-whatever etc. and simply hard code the path as dependencies in app-firmware and each app has their own configuration. Everything will be bundled into a single image and protects you from things leaking in from the sides rather than from above.
derpherpsson 7y ago

mc_ 7y ago

We're happily plugging along on KAZOO [0] since 2010 at 2600Hz. Erlang is a force-multiplier for us. Approaching 300K LOC and ~10 Erlang engineers (depends if Karl is writing code or CTO-ing that day) today. [0] https://github.com/2600hz/kazoo/ https://github.com/2600hz/kazoo/

tickbw 7y ago

I have been building grouper.ai to integrate voice ai systems with 2600hz's kazoo apis, and have enjoyed doing it as a Greenfield side project. Nerves for hardware on embedded Linux, c/c++ for the voice apis, ports in erlang/elixir to talk the low level calls, using elixir Phoenix in the cloud on the beam. Have used erlang also in day job for custom industrial M2M protocol translation (serial / Ethernet) very successfully also.

miki123211 7y ago

> Many countries rely on Erlang for their immediate payment switches, allowing instant bank transfers Just wondering if such a system used in Poland, actually called Elixir[1], has a line of Erlang/Elixir in its codebase. [1] https://pl.wikipedia.org/wiki/Elixir_(system_rozliczeń_międzybankowych) https://pl.wikipedia.org/wiki/Elixir_(system_rozliczeń_międz...

fastbmk 7y ago

Observed facts of Google Trends: 1. Erlang/Elixir are less than 1% of programming languages market. 2. Erlang/Elixir popularity is trending down.

therockhead 7y ago
Elixirforums have reported year on years increased traffic, so I’m surprised to hear that.