8 ms·

I think that not using Erlang in this particular case was a mistake. Erlang is running some of the largest chats out there, including League of Legends and What

by pepesza 10y ago

I think that not using Erlang in this particular case was a mistake. Erlang is running some of the largest chats out there, including League of Legends and WhatsApp. They would have avoided all the hassle of GC pauses, since Erlang has per-process GC collection. And scaling single OS process to their number of connections was done for Erlang machines years ago.

gnuvince 10y ago
Isn't Facebook's chat also powered by Erlang?
- kevincox 10y ago
  It definitely was when it originally launched. After that I have no idea.
- nocarrier 10y ago
  The Erlang components of Facebook Chat were replaced with C++ several years ago. The two main reasons I remember are A) it was hard to maintain a group of engineers with acceptable competency in Erlang over time and B) the C++ code offered faster and more consistent performance albeit with somewhat less scalability in terms of sessions per host. We just added X% more servers to the channel pools and were happy to have the chat services in a language where more FB engineers could contribute. There's been a lot more changes in the architecture than just moving to C++ though, so it's hard to do a direct comparison between the products. This doesn't take anything away from WhatsApp though, who has built a strong product and infrastructure on top of Erlang.
weberc2 10y ago
There are likely other tradeoffs. This (GC pause times) is probably not the only criterium, nor even the most important. It's really hard to draw a conclusion based on such limited information.
jerf 10y ago
Possibly in the past, yes. But if they're now just paying 1ms in GC every so often, the advantage is now gone. Go is generally faster than Erlang (in Go-native vs. Erlang-native code) so the system is quite possibly net outperforming what Erlang can do now. 1ms is just noise when packet latency jitter is higher than that.
- d3ckard 10y ago
  That is not the point. The advantage of Erlang is not raw speed, but the sheer amount of language constructs helping you write distributed system without thinking too much about low-level stuff. If I had to do some parallel data crunching, I would probably use Go or something similar. To write an actual system, it's much easier just to stand on the shoulders of Erlang guys instead of developing everything by hand(i.e. whole supervision tree).
  - 010a 10y ago
    The advantage of language constructs helping you, but the disadvantage of finding devs to actually build with them.
    - davidw 10y ago
      > The advantage of language constructs helping you, but the disadvantage of finding devs to actually build with them. The flip side of that coin is you're liable to get someone reinventing the wheel - poorly - in whatever language doesn't have all those goodies.
      - d3ckard 10y ago
        Yep, Greenspun's tenth rule comes to mind.
        davidw 10y ago
        In the Erlang world, we call that Virding's rule: "Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." Edit: I'll add, though, that the Go people are pretty smart and seem like they're doing good things, so I wouldn't be too complacent in thinking Erlang is the only game in town. It still does get some things right that are hard to replicate in Go, though.
woodcut 10y ago
Finding an erlang programmer available on site within 1-2 months is the hardest part of deciding to go with erlang. With go you can take a C++/python programmer and have them writing production code pretty soon, i think this is what inhibits functional programming in general, the learning curve bundled with the amount of work around prevents people jumping onboard also willingness of some employers to hire someone without a ton of exp. with erlang makes it difficult for a senior programmer to switch.
- dozzie 10y ago
  Actually, the hardest part of deciding to go with Erlang is not finding a programmer within 1-2 months, but deciding that one will need to train the programmer themselves (which obviously takes time; it took me half a year dabbling with Erlang and OTP (totally on my own) to actually start writing idiomatic code).
- jcadam 10y ago
  Truthfully, I think the supply of programmers with functional programming skills far outstrips the demand. I use FP languages on personal projects (most recently Clojure) all the time, but the day job is still programming in Java at a shop that is all Java, all the time. Every time I suggest bringing a language like Scala or Clojure into the mix (where they would provide real benefits over Java), I always get the "And where will we find programmers to maintain the code you write?" line from management. The answer, of course, is that there are likely legions of programmers like me, who hack around with FP languages in their spare time but whose only 'professional' experience is in mainstream languages. I suspect the real reason is that most management is just too risk-averse to consider using technology that isn't mainstream.
  - gf263 10y ago
    Why don't you tell them that?
  - 102030485868 10y ago
    That's changing now, thankfully. Most people at the company I work for use Clojure every day.
Thaxll 10y ago
Go 1.6 GC is probably faster thant Erlang GC now.
- jerf 10y ago
  OK, having posted something defending Go in this thread, now let me exasperate everyone by going the other way. Because Erlang's GC is per Erlang-process whereas Go's GC is still OS-process-global, there really isn't a "faster than/slower than" comparison available, because their workloads are so dissimilar. When an Erlang GC runs, it may be running across a mere few hundred or thousand bytes, freezing only that one Erlang-process that was quite likely not running at the moment anyhow. Erlang also has the GC-time advantage that it doesn't have pointers, so there's no pointer-fixup penalty. (It may be a disadvantage at other times, but it's certainly an advantage at GC time.)
  - Thaxll 10y ago
    Do you have any benchmarks that running small GC collection ( per process ) vs one big Heap is faster?
    - jerf 10y ago
      Define "faster". I actually have experience that looping over all Erlang processes and running a GC on each of them is definitely human-clock-time orders of magnitude slower than a Go garbage collection across a similar set of data. But who cares? First of all, that was a bit of a desperation play on my part anyhow, run for diagnostic purposes in the REPL, not an operation you do all the time, and secondly, only one process at time was frozen then anyhow, so I didn't care that it took about 10 seconds. It didn't take my service down. Which was my point in the first place, that "faster" and "slower" don't really apply here, because what they're doing is so different from each other. There's too many different possible definitions of faster. And you have to be careful to use one that matters to your code, not just an artificial benchmark that shows your preferred choice in the better light. (For those who may be curious, the problem that led me to that play was some now long-fixed issues with large binaries.)
010a 10y ago
I would emphasize that, in many of those examples, Go wasn't a very viable choice when the apps were originally written. Twitch chose Go back at ~1.2 (2013), when Erlang might have made more sense. Today, for companies making a similar decision now, that argument is a bit different. Go 1.6/1.7 obviously has massive improvements in the areas the article outlines. But, in Erlang camp, we have Elixir making that more enticing. I would argue Twitch made the right choice. They will have a magnitude easier time finding devs to support a Go system over an Erlang system. And their product never suffered for it. And they are clearly a force behind making Go better, which has helped more people than just them.
- ken47 10y ago
  This propagates a myth that the choice of language is the bottleneck in a complex program. Twitch hires engineers who are good enough not to be constrained by the difficulties of learning a particular language. Odds are, Twitch wasn't trying to optimize over a long time horizon when they chose Go. They were once a scrappy startup, surely accumulating technical debt left and right to get product features out. Go was likely a locally optimal choice. Twitch chat also requires heavy string processing, and that's an arena where, if I had to guess, Go has an edge over Erlang.
  - chipperyman573 10y ago
    It's not really heavy string processing, it's all just replacing inner strings (not even Regex).
    - ken47 10y ago
      Sure. I think we're talking about different things when we say heavy. Perhaps less ambiguous phrasing would have been "frequent string manipulation."
rogerdpack 10y ago
According to the article they chose it because of "Its simplicity, safety, performance, and readability" perhaps it has more in some of those than Erlang does/did... ?
dayjah 10y ago
Hi there, I'm one of the original engineers who worked on our re-implementation of chat which ended up in Go. We've a culture of being willing to try new things at Twitch. When our twisted-python chat system no longer met our needs of being easy to iterate on we decided to rebuild it; it was a monolith and we decided to chunk it up to reflect needs of our users and the pace at which we could develop new features. Notably we wanted to no recycle TCP connections whenever a new feature was added (which was a short coming of the twisted-python solution - along with a bunch of global state that was becoming hard to reason about). As part of this re-work we had a pub-sub portion which was super simple and we decided to try this new exciting language with a lot of promise out on it - it worked amazingly well. Over the course of another year or so we ended up rebuilding all of the components in Go. When we first evaluated rebuilding chat we assessed a few options: - python - nodejs (we started with this, but random crashes and poor tooling at the time didn't work for us) - erlang (notably could we use ejabberd as the hub of the system) Ultimately we chose python because we knew python and we needed this to work right now. The move to go happened incrementally thereafter and was driven by: - increase in trust - great tooling None of this can be pitched as "Go vs X", it is purely a tools and expediency orientated set of decisions.
- STRML 10y ago
  > Notably we wanted to no recycle TCP connections whenever a new feature was added So with the Go server, you're able to redeploy without closing open connections? Do you just run multiple versions in parallel and load balance over to the new version once connections close, or something else?
  - Dobbs 10y ago
    There are actually two (or more) different services. One that sits and talks to the users via TCP and maintains the IRC connection state and then makes back end calls to the bit that makes decisions and publishes information. This allows us to almost never deploy changes to the first service, while frequently making changes to the second system. Of course when you do want to make changes to the first you have to reestablish all the TCP connections again, but if you engineer it correctly you can do it infrequently enough to be worthwhile. Disclaimer: I don't actually work on the chat team, this is based upon various conversations with people on the chat team and may be incorrect in some specifics or out of date.
smegel 10y ago
I'm glad they did. Sounds like they have helped push the development of Go along which is good for everyone.

by pepesza 10y ago

gnuvince 10y ago

Isn't Facebook's chat also powered by Erlang?

kevincox 10y ago
It definitely was when it originally launched. After that I have no idea.
nocarrier 10y ago
The Erlang components of Facebook Chat were replaced with C++ several years ago. The two main reasons I remember are A) it was hard to maintain a group of engineers with acceptable competency in Erlang over time and B) the C++ code offered faster and more consistent performance albeit with somewhat less scalability in terms of sessions per host. We just added X% more servers to the channel pools and were happy to have the chat services in a language where more FB engineers could contribute. There's been a lot more changes in the architecture than just moving to C++ though, so it's hard to do a direct comparison between the products. This doesn't take anything away from WhatsApp though, who has built a strong product and infrastructure on top of Erlang.

weberc2 10y ago

There are likely other tradeoffs. This (GC pause times) is probably not the only criterium, nor even the most important. It's really hard to draw a conclusion based on such limited information.

jerf 10y ago

Possibly in the past, yes. But if they're now just paying 1ms in GC every so often, the advantage is now gone. Go is generally faster than Erlang (in Go-native vs. Erlang-native code) so the system is quite possibly net outperforming what Erlang can do now. 1ms is just noise when packet latency jitter is higher than that.

d3ckard 10y ago
That is not the point. The advantage of Erlang is not raw speed, but the sheer amount of language constructs helping you write distributed system without thinking too much about low-level stuff. If I had to do some parallel data crunching, I would probably use Go or something similar. To write an actual system, it's much easier just to stand on the shoulders of Erlang guys instead of developing everything by hand(i.e. whole supervision tree).
- 010a 10y ago
  The advantage of language constructs helping you, but the disadvantage of finding devs to actually build with them.
  - davidw 10y ago
    > The advantage of language constructs helping you, but the disadvantage of finding devs to actually build with them. The flip side of that coin is you're liable to get someone reinventing the wheel - poorly - in whatever language doesn't have all those goodies.
    - d3ckard 10y ago
      Yep, Greenspun's tenth rule comes to mind.
      - davidw 10y ago
        In the Erlang world, we call that Virding's rule: "Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." Edit: I'll add, though, that the Go people are pretty smart and seem like they're doing good things, so I wouldn't be too complacent in thinking Erlang is the only game in town. It still does get some things right that are hard to replicate in Go, though.

woodcut 10y ago

Finding an erlang programmer available on site within 1-2 months is the hardest part of deciding to go with erlang. With go you can take a C++/python programmer and have them writing production code pretty soon, i think this is what inhibits functional programming in general, the learning curve bundled with the amount of work around prevents people jumping onboard also willingness of some employers to hire someone without a ton of exp. with erlang makes it difficult for a senior programmer to switch.

dozzie 10y ago
Actually, the hardest part of deciding to go with Erlang is not finding a programmer within 1-2 months, but deciding that one will need to train the programmer themselves (which obviously takes time; it took me half a year dabbling with Erlang and OTP (totally on my own) to actually start writing idiomatic code).
jcadam 10y ago
Truthfully, I think the supply of programmers with functional programming skills far outstrips the demand. I use FP languages on personal projects (most recently Clojure) all the time, but the day job is still programming in Java at a shop that is all Java, all the time. Every time I suggest bringing a language like Scala or Clojure into the mix (where they would provide real benefits over Java), I always get the "And where will we find programmers to maintain the code you write?" line from management. The answer, of course, is that there are likely legions of programmers like me, who hack around with FP languages in their spare time but whose only 'professional' experience is in mainstream languages. I suspect the real reason is that most management is just too risk-averse to consider using technology that isn't mainstream.
- gf263 10y ago
  Why don't you tell them that?
- 102030485868 10y ago
  That's changing now, thankfully. Most people at the company I work for use Clojure every day.

Thaxll 10y ago

Go 1.6 GC is probably faster thant Erlang GC now.

jerf 10y ago
OK, having posted something defending Go in this thread, now let me exasperate everyone by going the other way. Because Erlang's GC is per Erlang-process whereas Go's GC is still OS-process-global, there really isn't a "faster than/slower than" comparison available, because their workloads are so dissimilar. When an Erlang GC runs, it may be running across a mere few hundred or thousand bytes, freezing only that one Erlang-process that was quite likely not running at the moment anyhow. Erlang also has the GC-time advantage that it doesn't have pointers, so there's no pointer-fixup penalty. (It may be a disadvantage at other times, but it's certainly an advantage at GC time.)
- Thaxll 10y ago
  Do you have any benchmarks that running small GC collection ( per process ) vs one big Heap is faster?
  - jerf 10y ago
    Define "faster". I actually have experience that looping over all Erlang processes and running a GC on each of them is definitely human-clock-time orders of magnitude slower than a Go garbage collection across a similar set of data. But who cares? First of all, that was a bit of a desperation play on my part anyhow, run for diagnostic purposes in the REPL, not an operation you do all the time, and secondly, only one process at time was frozen then anyhow, so I didn't care that it took about 10 seconds. It didn't take my service down. Which was my point in the first place, that "faster" and "slower" don't really apply here, because what they're doing is so different from each other. There's too many different possible definitions of faster. And you have to be careful to use one that matters to your code, not just an artificial benchmark that shows your preferred choice in the better light. (For those who may be curious, the problem that led me to that play was some now long-fixed issues with large binaries.)

010a 10y ago

I would emphasize that, in many of those examples, Go wasn't a very viable choice when the apps were originally written. Twitch chose Go back at ~1.2 (2013), when Erlang might have made more sense. Today, for companies making a similar decision now, that argument is a bit different. Go 1.6/1.7 obviously has massive improvements in the areas the article outlines. But, in Erlang camp, we have Elixir making that more enticing. I would argue Twitch made the right choice. They will have a magnitude easier time finding devs to support a Go system over an Erlang system. And their product never suffered for it. And they are clearly a force behind making Go better, which has helped more people than just them.

ken47 10y ago
This propagates a myth that the choice of language is the bottleneck in a complex program. Twitch hires engineers who are good enough not to be constrained by the difficulties of learning a particular language. Odds are, Twitch wasn't trying to optimize over a long time horizon when they chose Go. They were once a scrappy startup, surely accumulating technical debt left and right to get product features out. Go was likely a locally optimal choice. Twitch chat also requires heavy string processing, and that's an arena where, if I had to guess, Go has an edge over Erlang.
- chipperyman573 10y ago
  It's not really heavy string processing, it's all just replacing inner strings (not even Regex).
  - ken47 10y ago
    Sure. I think we're talking about different things when we say heavy. Perhaps less ambiguous phrasing would have been "frequent string manipulation."

rogerdpack 10y ago

According to the article they chose it because of "Its simplicity, safety, performance, and readability" perhaps it has more in some of those than Erlang does/did... ?

dayjah 10y ago

Hi there, I'm one of the original engineers who worked on our re-implementation of chat which ended up in Go. We've a culture of being willing to try new things at Twitch. When our twisted-python chat system no longer met our needs of being easy to iterate on we decided to rebuild it; it was a monolith and we decided to chunk it up to reflect needs of our users and the pace at which we could develop new features. Notably we wanted to no recycle TCP connections whenever a new feature was added (which was a short coming of the twisted-python solution - along with a bunch of global state that was becoming hard to reason about). As part of this re-work we had a pub-sub portion which was super simple and we decided to try this new exciting language with a lot of promise out on it - it worked amazingly well. Over the course of another year or so we ended up rebuilding all of the components in Go. When we first evaluated rebuilding chat we assessed a few options: - python - nodejs (we started with this, but random crashes and poor tooling at the time didn't work for us) - erlang (notably could we use ejabberd as the hub of the system) Ultimately we chose python because we knew python and we needed this to work right now. The move to go happened incrementally thereafter and was driven by: - increase in trust - great tooling None of this can be pitched as "Go vs X", it is purely a tools and expediency orientated set of decisions.

STRML 10y ago
> Notably we wanted to no recycle TCP connections whenever a new feature was added So with the Go server, you're able to redeploy without closing open connections? Do you just run multiple versions in parallel and load balance over to the new version once connections close, or something else?
- Dobbs 10y ago
  There are actually two (or more) different services. One that sits and talks to the users via TCP and maintains the IRC connection state and then makes back end calls to the bit that makes decisions and publishes information. This allows us to almost never deploy changes to the first service, while frequently making changes to the second system. Of course when you do want to make changes to the first you have to reestablish all the TCP connections again, but if you engineer it correctly you can do it infrequently enough to be worthwhile. Disclaimer: I don't actually work on the chat team, this is based upon various conversations with people on the chat team and may be incorrect in some specifics or out of date.

smegel 10y ago

I'm glad they did. Sounds like they have helped push the development of Go along which is good for everyone.