5 ms·

Twitter considered migrating from Ruby on Rails to Erlang 11 years ago (but ultimately chose Scala due to the Java tie-in). https://news.ycombinator.com/item?i

Twitter considered migrating from Ruby on Rails to Erlang 11 years ago (but ultimately chose Scala due to the Java tie-in). https://news.ycombinator.com/item?id=2825689 https://news.ycombinator.com/item?id=2825689 I've always wonder how the Erlang ecosystem would be different today if a digital giant like a Twitter/FB/Google had used Erlang as part of their core business. Note: yes, I realize WhatsApp owned by FB is run on Erlang.

dmitriid 5y ago
Facebook chat was originally written in Erlang and then migrated to internal C/C++ framework. Giants use and experiment with a variety of languages.
- bcrosby95 5y ago
  This is why I'm always skeptical when someone says "$bigco uses $language! It's on the map!" Yeah, well, somewhere, in some department, $bigco probably uses any random language you can think of.
  - joelbluminator 5y ago
    Yes, this. They buy companies all the time. Also counting Erlang and Elixir together seems a bit odd, are they the same thing? Is a Clojure company actually a Java company?
    - josevalim 5y ago
      If someone makes the Java GC 3x faster, that will most likely positively impact all Java and Clojure programmers. Similarly, if someone wants to invest on Clojure performance, they most likely need to do so by investing on the JVM and core libraries, which will benefit the overall ecosystem. Plus one main reasons people use to pick JVM languages is the access to the whole Java ecosystem and related packages. Therefore, if you are considering the performance of the platform, improvements to the runtime and virtual machine, package ecosystem, etc, lumping them together makes total sense. On the other hand, Clojure and Java are quite different languages, with complete different paradigms, so there is probably a ticker layer specifically dedicated to Clojure. Erlang and Elixir are much closer to each other than Clojure and Java or even Scala and Java. The data structures are the same, the language constructs are mostly the same (different syntax but the same semantics), etc. You can call each other with no performance cost whatsoever. All of the companies investing on Erlang will directly impact Elixir and, if you want to make the Elixir compiler or runtime faster, odds are that this efforts goes to Erlang - it is one of the reasons I have ~100 PRs merged upstream, mostly focused on the compiler and/or performance. So, once again, if your goal is to assess the healthy of the ecosystem or how many companies are investing on the platform, then they must be considered together. The contributions done by Ericsson, Whatsapp, Klarna, Erlang Solutions, etc. benefit all languages running on the Erlang VM. In fact, if someone asks me how they can financially contribute to Elixir, I will 100% direct them to the Erlang Ecosystem Foundation (https://erlef.org/ https://erlef.org/), which focuses on domain areas around the ecosystem, rather than languages. Since you seem to use Ruby as a reference, I don't expect the dynamics to be much different. Most companies and contributors are likely working directly on the Ruby VM and the runtime, rather than the language itself. However, because Ruby is the only (main?) language running on it, we don't tend to separate them. But in practice the language is only a small surface into the complexity that is its underlying platform and working on said platforms requires a completely different set of skills and technologies. EDIT: this applies to the ecosystem too. For example, every Phoenix application is running a web server written in... Erlang. :) All Ecto database adapters are using the tcp modules provided by Erlang. The package manager that powers all languages running on the Erlang VM is written in Elixir. And so on. You don't need to know Erlang in order to write Elixir but, if for some reason you need to pick up, it is straightforward because of all of the above.
      - joelbluminator 5y ago
        Ruby runs on C, I guess if C gains in performance then Ruby (and Python, and JS(?) and PHP) all become faster as well. I know it's not a perfect analogy to Erlang/Elixir but there is a strong connection between C performance/memory management and Ruby. For instance the whole malloc/jemalloc memory issue which affects Ruby/Python/Node(?) apps. It could be that these dynamic languages misuse C or something and it's really not C's "fault", I'm not sure, it's above my pay grade :)
        josevalim 5y ago
        I am sorry but that's a very inaccurate analogy on this topic. The Ruby VM, in particular MRI, is implemented in C. The Erlang VM is also implemented in C. A better C compiler can likely benefit them. But what runs your Ruby code is the Ruby VM (or even the JVM). What runs Java and Clojure code is the JVM. And what runs Elixir code is the Erlang VM. The relationship between Elixir and Erlang is much closer to, say, Rails and Ruby (i.e. Rails is built on Ruby's abstractions and semantics) than Ruby and C. It is not a good example but it makes much more sense than the analogy above.
        joelbluminator 5y ago
        OK, gotcha thanks for explaining. Just for my general knowledge is Erlang/Elixir apps also affected with memory bloat (I mean memory that grows and grows and never properly garbage collection) which seems to be improved for certain apps with jemalloc 3? I mean if it's built with C I would imagine the same problem may occur for Erlang/Elixir.
        josevalim 5y ago
        I am not familiar with the issues affecting the referenced languages to make an efficient comparison, sorry. But I can add some context if you are interested in looking further. First of all, let's talk about the memory model. The Erlang VM manages memory per process (where a process is a lightweight thread of execution inside the VM that runs concurrently and you can literally have millions of them). So for example, when you have a web request, you spawn a separate process for this web request and once the process terminates, all the memory allocated to it is reclaimed by the VM. This means you can serve a web request without triggering a garbage collector even once. And if you need to garbage collect, it is local per process, so no system wide pauses. However, what happens if you want to share a large blob of memory between processes? If you copied it between each process it would be expensive, so in this case the VM employs reference counting. One curious point is that the Erlang VM did not give allocated memory back to the OS until Erlang/OTP 22 (roughly two years ago). This was not much of a problem in the Erlang VM because you typically run only one instance of the VM per node, regardless if you have 4 or 32 cores. Given the issues you mentioned in Python/Ruby/Node, could it partially be because you would rather start 4 or 32 instances per node, so if one of them accumulates too much memory and doesn't give back, it starves the other ones? Or perhaps you would have background workers fighting each other? If you have links to said issues, please share them! Implementation wise, the Erlang VM uses its own memory allocation library to fight fragmentation. You can learn more about it on the erts_alloc API (http://erlang.org/doc/man/erts_alloc.html http://erlang.org/doc/man/erts_alloc.html) and there is also a relatively recent article on instrumentation (https://blog.erlang.org/Memory-instrumentation-in-OTP-21/ https://blog.erlang.org/Memory-instrumentation-in-OTP-21/).
zinclozenge 5y ago
Hard to say, as you pointed out, Whatapp runs on Erlang, and despite their upstream patches, no BEAM based framework is competitive on the techempower benchmarks. But let's not forget, AWS SimpleDB was written in Erlang, as well as the first iteration of Facebook Messenger. I invested a decent amount of time in learning Elixir and Erlang and understanding BEAM. My personal takeaway, which seems to be mirrored by every big company, is that the ecosystem just isn't worth the investment.
- brightball 5y ago
  Techempower benchmarks don’t really tell the story, especially if you look into the code being used. Besides, Django, Rails, Laravel, etc are awful in those same benchmarks but adoption is fine. The selling points of Erlang/Elixir are far more than simply non-real world benchmarks.
- josevalim 5y ago
  > as well as the first iteration of Facebook Messenger I was told, from someone familiar with the project at the time, that the first iteration was done with ejabberd, which is written in Erlang, with the main purpose of bootstrapping it. Erlang was not really in consideration in the long term at the time. Not sure if it changes your take personally but hopefully adding some context helps. I am always glad to be corrected if wrong. :) I wrote about Techempower in the past. In a nutshell, the Erlang runtime does plenty of assumptions that you would absolutely use in production but which become overhead in benchmarks like these. Previous comment here: https://news.ycombinator.com/item?id=22894191 https://news.ycombinator.com/item?id=22894191 > My personal takeaway, which seems to be mirrored by every big company, is that the ecosystem just isn't worth the investment. Your experience was guided by a decent amount of time and I actually appreciate you invested in learning it before deciding it wasn't worth it. However, I doubt every big company had the same takeaway for two reasons: 1. I doubt every big company went through the same experience (i.e. have they learned about the ecosystem?) to mirror the takeaway 2. The every quantifier is provably false, since there is Facebook/Whatsapp, PepsiCo, Klarna, Ericsson, Nokia, Mozilla, Cisco, and others using the ecosystem in production and/or investing on it (plus at least another company from FAANG using it at multiple projects)
dudul 5y ago
Do you think Twitter using Scala had a big influence on the current state of its ecosystem?
- alberth 5y ago
  There was no major users of Scala prior to Twitter that I can recall. Twitter put Scala on-the-map. Plus, there is far less of an "ecosystem improvement" to be made with Scala since it's riding on the Java Runtime (which already had massive adoption and development prior to Twitter selecting it).
- dustingetz 5y ago
  twitter pumped tens of $M/yr of developer payroll into scala real-time infrastructure [citation needed]
- mbo 5y ago
  In the 3 years I spent writing Scala, the only major Twitter contribution I remember running into was Finagle (http://twitter.github.io/finagle/ http://twitter.github.io/finagle/). It might have been a big influence at the time, but these days it's relegated to the scrapheap of older frameworks.
- lmm 5y ago
  Twitter were the first to get the future/promise abstraction right and publish it widely (in Finagle). That inspired not just Scala's now-standard Futures but also the CompletableFuture in newer versions of Java and even e.g. Rust's Future, although none of them have the wonderful for/yield syntax (similar to Haskell do notation) that make it such a joy to work with in Scala. Sadly most of the tooling funding came from the other big Scala user, who favoured an invisible-yield-point style (which I'm pretty sure inspired the model that Kotlin now uses with suspend functions). One of the many tragedies of Scala (still the best general-purpose language on the whole, IMO) is that the community formed these somewhat siloed ecosystems that were each doing things their own way and had their own stack of libraries that didn't really play nice with each other. The language actually makes it very easy to write your own adapters (which may be why industrial users didn't create a lot of pressure to stop all the nonsense), but political fallings-out usually meant that it took years before there were "blessed"/official ways to use X with Y.
latchkey 5y ago
Early days of Twitter were a lot of scaling issues due to the choice and implementation of RoR and they considered a lot of things, which got a lot of public attention. 13 years ago... https://venturebeat.com/2008/05/01/twitter-to-jump-off-ruby-on-rails/ https://venturebeat.com/2008/05/01/twitter-to-jump-off-ruby-...
- sk55 5y ago
  Twitter probably couldn't scale because of bad engineering, not because of RoR. For example, Shopify embraces the Ruby ecosystem and it scales up very well.
  - fouc 5y ago
    I imagine shopify is more profitable for the amount of traffic it gets, so they can afford to throw more machines at the problem as well. Shopify does probably have better engineering though. They have an amazing process of 2 code reviewers per PR, and one code reviewer is from another team. They write tests for everything and deploy continuously to production.

dmitriid 5y ago

Facebook chat was originally written in Erlang and then migrated to internal C/C++ framework. Giants use and experiment with a variety of languages.

bcrosby95 5y ago
This is why I'm always skeptical when someone says "$bigco uses $language! It's on the map!" Yeah, well, somewhere, in some department, $bigco probably uses any random language you can think of.
- joelbluminator 5y ago
  Yes, this. They buy companies all the time. Also counting Erlang and Elixir together seems a bit odd, are they the same thing? Is a Clojure company actually a Java company?
  - josevalim 5y ago
    If someone makes the Java GC 3x faster, that will most likely positively impact all Java and Clojure programmers. Similarly, if someone wants to invest on Clojure performance, they most likely need to do so by investing on the JVM and core libraries, which will benefit the overall ecosystem. Plus one main reasons people use to pick JVM languages is the access to the whole Java ecosystem and related packages. Therefore, if you are considering the performance of the platform, improvements to the runtime and virtual machine, package ecosystem, etc, lumping them together makes total sense. On the other hand, Clojure and Java are quite different languages, with complete different paradigms, so there is probably a ticker layer specifically dedicated to Clojure. Erlang and Elixir are much closer to each other than Clojure and Java or even Scala and Java. The data structures are the same, the language constructs are mostly the same (different syntax but the same semantics), etc. You can call each other with no performance cost whatsoever. All of the companies investing on Erlang will directly impact Elixir and, if you want to make the Elixir compiler or runtime faster, odds are that this efforts goes to Erlang - it is one of the reasons I have ~100 PRs merged upstream, mostly focused on the compiler and/or performance. So, once again, if your goal is to assess the healthy of the ecosystem or how many companies are investing on the platform, then they must be considered together. The contributions done by Ericsson, Whatsapp, Klarna, Erlang Solutions, etc. benefit all languages running on the Erlang VM. In fact, if someone asks me how they can financially contribute to Elixir, I will 100% direct them to the Erlang Ecosystem Foundation (https://erlef.org/ https://erlef.org/), which focuses on domain areas around the ecosystem, rather than languages. Since you seem to use Ruby as a reference, I don't expect the dynamics to be much different. Most companies and contributors are likely working directly on the Ruby VM and the runtime, rather than the language itself. However, because Ruby is the only (main?) language running on it, we don't tend to separate them. But in practice the language is only a small surface into the complexity that is its underlying platform and working on said platforms requires a completely different set of skills and technologies. EDIT: this applies to the ecosystem too. For example, every Phoenix application is running a web server written in... Erlang. :) All Ecto database adapters are using the tcp modules provided by Erlang. The package manager that powers all languages running on the Erlang VM is written in Elixir. And so on. You don't need to know Erlang in order to write Elixir but, if for some reason you need to pick up, it is straightforward because of all of the above.
    - joelbluminator 5y ago
      Ruby runs on C, I guess if C gains in performance then Ruby (and Python, and JS(?) and PHP) all become faster as well. I know it's not a perfect analogy to Erlang/Elixir but there is a strong connection between C performance/memory management and Ruby. For instance the whole malloc/jemalloc memory issue which affects Ruby/Python/Node(?) apps. It could be that these dynamic languages misuse C or something and it's really not C's "fault", I'm not sure, it's above my pay grade :)
      - josevalim 5y ago
        I am sorry but that's a very inaccurate analogy on this topic. The Ruby VM, in particular MRI, is implemented in C. The Erlang VM is also implemented in C. A better C compiler can likely benefit them. But what runs your Ruby code is the Ruby VM (or even the JVM). What runs Java and Clojure code is the JVM. And what runs Elixir code is the Erlang VM. The relationship between Elixir and Erlang is much closer to, say, Rails and Ruby (i.e. Rails is built on Ruby's abstractions and semantics) than Ruby and C. It is not a good example but it makes much more sense than the analogy above.
        joelbluminator 5y ago
        OK, gotcha thanks for explaining. Just for my general knowledge is Erlang/Elixir apps also affected with memory bloat (I mean memory that grows and grows and never properly garbage collection) which seems to be improved for certain apps with jemalloc 3? I mean if it's built with C I would imagine the same problem may occur for Erlang/Elixir.
        josevalim 5y ago
        I am not familiar with the issues affecting the referenced languages to make an efficient comparison, sorry. But I can add some context if you are interested in looking further. First of all, let's talk about the memory model. The Erlang VM manages memory per process (where a process is a lightweight thread of execution inside the VM that runs concurrently and you can literally have millions of them). So for example, when you have a web request, you spawn a separate process for this web request and once the process terminates, all the memory allocated to it is reclaimed by the VM. This means you can serve a web request without triggering a garbage collector even once. And if you need to garbage collect, it is local per process, so no system wide pauses. However, what happens if you want to share a large blob of memory between processes? If you copied it between each process it would be expensive, so in this case the VM employs reference counting. One curious point is that the Erlang VM did not give allocated memory back to the OS until Erlang/OTP 22 (roughly two years ago). This was not much of a problem in the Erlang VM because you typically run only one instance of the VM per node, regardless if you have 4 or 32 cores. Given the issues you mentioned in Python/Ruby/Node, could it partially be because you would rather start 4 or 32 instances per node, so if one of them accumulates too much memory and doesn't give back, it starves the other ones? Or perhaps you would have background workers fighting each other? If you have links to said issues, please share them! Implementation wise, the Erlang VM uses its own memory allocation library to fight fragmentation. You can learn more about it on the erts_alloc API (http://erlang.org/doc/man/erts_alloc.html http://erlang.org/doc/man/erts_alloc.html) and there is also a relatively recent article on instrumentation (https://blog.erlang.org/Memory-instrumentation-in-OTP-21/ https://blog.erlang.org/Memory-instrumentation-in-OTP-21/).

zinclozenge 5y ago

Hard to say, as you pointed out, Whatapp runs on Erlang, and despite their upstream patches, no BEAM based framework is competitive on the techempower benchmarks. But let's not forget, AWS SimpleDB was written in Erlang, as well as the first iteration of Facebook Messenger. I invested a decent amount of time in learning Elixir and Erlang and understanding BEAM. My personal takeaway, which seems to be mirrored by every big company, is that the ecosystem just isn't worth the investment.

brightball 5y ago
Techempower benchmarks don’t really tell the story, especially if you look into the code being used. Besides, Django, Rails, Laravel, etc are awful in those same benchmarks but adoption is fine. The selling points of Erlang/Elixir are far more than simply non-real world benchmarks.
josevalim 5y ago
> as well as the first iteration of Facebook Messenger I was told, from someone familiar with the project at the time, that the first iteration was done with ejabberd, which is written in Erlang, with the main purpose of bootstrapping it. Erlang was not really in consideration in the long term at the time. Not sure if it changes your take personally but hopefully adding some context helps. I am always glad to be corrected if wrong. :) I wrote about Techempower in the past. In a nutshell, the Erlang runtime does plenty of assumptions that you would absolutely use in production but which become overhead in benchmarks like these. Previous comment here: https://news.ycombinator.com/item?id=22894191 https://news.ycombinator.com/item?id=22894191 > My personal takeaway, which seems to be mirrored by every big company, is that the ecosystem just isn't worth the investment. Your experience was guided by a decent amount of time and I actually appreciate you invested in learning it before deciding it wasn't worth it. However, I doubt every big company had the same takeaway for two reasons: 1. I doubt every big company went through the same experience (i.e. have they learned about the ecosystem?) to mirror the takeaway 2. The every quantifier is provably false, since there is Facebook/Whatsapp, PepsiCo, Klarna, Ericsson, Nokia, Mozilla, Cisco, and others using the ecosystem in production and/or investing on it (plus at least another company from FAANG using it at multiple projects)

dudul 5y ago

Do you think Twitter using Scala had a big influence on the current state of its ecosystem?

alberth 5y ago
There was no major users of Scala prior to Twitter that I can recall. Twitter put Scala on-the-map. Plus, there is far less of an "ecosystem improvement" to be made with Scala since it's riding on the Java Runtime (which already had massive adoption and development prior to Twitter selecting it).
dustingetz 5y ago
twitter pumped tens of $M/yr of developer payroll into scala real-time infrastructure [citation needed]
mbo 5y ago
In the 3 years I spent writing Scala, the only major Twitter contribution I remember running into was Finagle (http://twitter.github.io/finagle/ http://twitter.github.io/finagle/). It might have been a big influence at the time, but these days it's relegated to the scrapheap of older frameworks.
lmm 5y ago
Twitter were the first to get the future/promise abstraction right and publish it widely (in Finagle). That inspired not just Scala's now-standard Futures but also the CompletableFuture in newer versions of Java and even e.g. Rust's Future, although none of them have the wonderful for/yield syntax (similar to Haskell do notation) that make it such a joy to work with in Scala. Sadly most of the tooling funding came from the other big Scala user, who favoured an invisible-yield-point style (which I'm pretty sure inspired the model that Kotlin now uses with suspend functions). One of the many tragedies of Scala (still the best general-purpose language on the whole, IMO) is that the community formed these somewhat siloed ecosystems that were each doing things their own way and had their own stack of libraries that didn't really play nice with each other. The language actually makes it very easy to write your own adapters (which may be why industrial users didn't create a lot of pressure to stop all the nonsense), but political fallings-out usually meant that it took years before there were "blessed"/official ways to use X with Y.

latchkey 5y ago

Early days of Twitter were a lot of scaling issues due to the choice and implementation of RoR and they considered a lot of things, which got a lot of public attention. 13 years ago... https://venturebeat.com/2008/05/01/twitter-to-jump-off-ruby-on-rails/ https://venturebeat.com/2008/05/01/twitter-to-jump-off-ruby-...

sk55 5y ago
Twitter probably couldn't scale because of bad engineering, not because of RoR. For example, Shopify embraces the Ruby ecosystem and it scales up very well.
- fouc 5y ago
  I imagine shopify is more profitable for the amount of traffic it gets, so they can afford to throw more machines at the problem as well. Shopify does probably have better engineering though. They have an amazing process of 2 code reviewers per PR, and one code reviewer is from another team. They write tests for everything and deploy continuously to production.