5 ms·

Not trying to be mean, but I was honestly confused by a couple things: 1. The server for the company's core product is written in Erlang, but no one at the com

by jonahx 13y ago

Not trying to be mean, but I was honestly confused by a couple things: 1. The server for the company's core product is written in Erlang, but no one at the company is an Erlang expert? 2. The company decides to rewrite the core of its product in python, and gives the job to an intern? I feel like I must be missing something....

djhworld 13y ago
Usually in cases like this, the people who wrote the original implementation probably left the company.
josh2600 13y ago
This happened on Facebook's chat system too. For some reason, and I know no other language where this is true to quite this degree (except maybe lisp), but Erlang has a tendency for people to believe they understand how to build an app using it when in reality they can do nothing of the sort (at least in Erlang. It's common for folks to 'do it in Erlang' because its seen as different and somewhat elitist. People think of Erlang engineers as 'hardcore'; I shudder to attribute skill to a language rather than a person, but I digress...). In the case of FB, as told to me, a bunch of folks in the valley said "of course we know Erlang!" and built the app but when push came to shove the only hardcore Erlang guys were in Poland and that made working on chat difficult. I'm not sure how much of that is truth, but its a recurring theme in Erlang projects, one that I've personally heard time and again. For some reason, Erlang inspires confidence, perhaps even foolhardiness. Side note: I work at an Erlang shop and our lead engineers speak at Erlang Factory every year. It's a great language and quite powerful but it can be difficult to understand.
- davidw 13y ago
  Erlang has gotten some hype about how "amazingly scalable" it is. The reality is that, as with anything, it takes work to make it scale, and once things get Really Big, it gets difficult. My sneaking suspicion is that, in some cases, people pick it because they read that it's "web scale" or something like that without really knowing the details, and when things fall apart, they revert to something that they know better. BTW, reference for the FB chat thing being rewritten?
  - josh2600 13y ago
    JHeiliger was talking about it at an interview I went to discussing his years at Facebook. On the note of Erlang's complexity, it's not so much complex as it is different. I think all languages get hard at scale, Erlang just has some nice native concurrent processing capabilities that are making more sense with each passing day as we move towards more cores in processors. It's easier to Parallelize Erlang than say C++ for example.
- whopa 13y ago
  So I heard a bit differently about the FB Chat thing. Facebook likes to move its engineers around to different teams from time to time (if only temporarily) to spread institutional knowledge around and prevent bus factors from being too low. This also is a way to alleviate burnout. There simply weren't enough engineers at FB who knew Erlang to make this practical for chat, so they rewrote it in C++. The Erlang bits were reasonably written and working fine, but it didn't fit into this policy. Any non-mainstream languages wouldn't fit, this isn't something about Erlang in particular. Erlang is a great fit for a number of problem domains, but if you silo it into a small part of your organization, you're going to run into issues such as these. You have to be willing to train people in it. It isn't that hard to pick up if it's someone's full time job to do so, as long as they're smart. And you want to hire smart people anyway, right?
- evgen 13y ago
  FWIW, the Erlang bits (a rather small chunk of code considering its scale) took over a year to effectively replace and led to numerous chat SEVs as people learned the hard way that their c++ replacement was missing some little bit of "seldom used but very nice to have when things get dicey" component or behavior in OTP. Given how people move around projects the difficulty of maintaining an Erlang team for just chat is hard to justify, but the move to c++ for chat back-end service says more about FB internal structure than it does about Erlang.
  - josh2600 13y ago
    I think the truth is that whenever you migrate a code base, it's trivial to get 90% of the functionality quickly, but it turns out that things you thought of as unimportant, or edge use cases, are actually not rare events at all. I agree that maintaining an internal Erlang team for one piece of code isn't smart, but I tend to think that in light of the move to broader media services (chat, video and voice streaming) perhaps it might make sense to look at the language once again. Maybe this time they'll do it in Go.
ams6110 13y ago
I had the same reaction. This is really not a fair critique of any weaknesses in Erlang or any strengths in Python. In his description of how they implemented maintenance of some global state, this line caught my attention: Our code was not set up this way at all, and it was clearly crippled by being haphazardly implemented in a functional style. In addition to the statement that they had no Erlang experts currently on staff, it sounds like the people who wrote the system were not Erlang experts either. I wonder if they used OTP to build the system? The post doesn't say, but I'd bet they wrote a lot of their code with erlang's message passing primitives, if they were using messaging at all. This will get hairy and that's why most people should use OTP to build server applications in Erlang. Lesson learned: you build better systems using languages and frameworks you know vs. using something trendy that you don't understand.
trefn 13y ago
Hey Jonah, I can clear this up. The api server was one of the very first things written, back when Mixpanel was a fun side project for some college kids, well before YC. We picked erlang because it sounded cool. It was actually pretty stable, but the only piece of erlang in our stack. Replacing it was one of those important but nonurgent things that are hard to prioritize. We want interns to have meaningful work, though, and "important + nonurgent" is just about the ideal thing for an intern to work on. They get to work on something interesting, but there's plenty of time for review and refactoring.

by jonahx 13y ago

djhworld 13y ago

Usually in cases like this, the people who wrote the original implementation probably left the company.

josh2600 13y ago

This happened on Facebook's chat system too. For some reason, and I know no other language where this is true to quite this degree (except maybe lisp), but Erlang has a tendency for people to believe they understand how to build an app using it when in reality they can do nothing of the sort (at least in Erlang. It's common for folks to 'do it in Erlang' because its seen as different and somewhat elitist. People think of Erlang engineers as 'hardcore'; I shudder to attribute skill to a language rather than a person, but I digress...). In the case of FB, as told to me, a bunch of folks in the valley said "of course we know Erlang!" and built the app but when push came to shove the only hardcore Erlang guys were in Poland and that made working on chat difficult. I'm not sure how much of that is truth, but its a recurring theme in Erlang projects, one that I've personally heard time and again. For some reason, Erlang inspires confidence, perhaps even foolhardiness. Side note: I work at an Erlang shop and our lead engineers speak at Erlang Factory every year. It's a great language and quite powerful but it can be difficult to understand.

davidw 13y ago
Erlang has gotten some hype about how "amazingly scalable" it is. The reality is that, as with anything, it takes work to make it scale, and once things get Really Big, it gets difficult. My sneaking suspicion is that, in some cases, people pick it because they read that it's "web scale" or something like that without really knowing the details, and when things fall apart, they revert to something that they know better. BTW, reference for the FB chat thing being rewritten?
- josh2600 13y ago
  JHeiliger was talking about it at an interview I went to discussing his years at Facebook. On the note of Erlang's complexity, it's not so much complex as it is different. I think all languages get hard at scale, Erlang just has some nice native concurrent processing capabilities that are making more sense with each passing day as we move towards more cores in processors. It's easier to Parallelize Erlang than say C++ for example.
whopa 13y ago
So I heard a bit differently about the FB Chat thing. Facebook likes to move its engineers around to different teams from time to time (if only temporarily) to spread institutional knowledge around and prevent bus factors from being too low. This also is a way to alleviate burnout. There simply weren't enough engineers at FB who knew Erlang to make this practical for chat, so they rewrote it in C++. The Erlang bits were reasonably written and working fine, but it didn't fit into this policy. Any non-mainstream languages wouldn't fit, this isn't something about Erlang in particular. Erlang is a great fit for a number of problem domains, but if you silo it into a small part of your organization, you're going to run into issues such as these. You have to be willing to train people in it. It isn't that hard to pick up if it's someone's full time job to do so, as long as they're smart. And you want to hire smart people anyway, right?
evgen 13y ago
FWIW, the Erlang bits (a rather small chunk of code considering its scale) took over a year to effectively replace and led to numerous chat SEVs as people learned the hard way that their c++ replacement was missing some little bit of "seldom used but very nice to have when things get dicey" component or behavior in OTP. Given how people move around projects the difficulty of maintaining an Erlang team for just chat is hard to justify, but the move to c++ for chat back-end service says more about FB internal structure than it does about Erlang.
- josh2600 13y ago
  I think the truth is that whenever you migrate a code base, it's trivial to get 90% of the functionality quickly, but it turns out that things you thought of as unimportant, or edge use cases, are actually not rare events at all. I agree that maintaining an internal Erlang team for one piece of code isn't smart, but I tend to think that in light of the move to broader media services (chat, video and voice streaming) perhaps it might make sense to look at the language once again. Maybe this time they'll do it in Go.

ams6110 13y ago

I had the same reaction. This is really not a fair critique of any weaknesses in Erlang or any strengths in Python. In his description of how they implemented maintenance of some global state, this line caught my attention: Our code was not set up this way at all, and it was clearly crippled by being haphazardly implemented in a functional style. In addition to the statement that they had no Erlang experts currently on staff, it sounds like the people who wrote the system were not Erlang experts either. I wonder if they used OTP to build the system? The post doesn't say, but I'd bet they wrote a lot of their code with erlang's message passing primitives, if they were using messaging at all. This will get hairy and that's why most people should use OTP to build server applications in Erlang. Lesson learned: you build better systems using languages and frameworks you know vs. using something trendy that you don't understand.

trefn 13y ago

Hey Jonah, I can clear this up. The api server was one of the very first things written, back when Mixpanel was a fun side project for some college kids, well before YC. We picked erlang because it sounded cool. It was actually pretty stable, but the only piece of erlang in our stack. Replacing it was one of those important but nonurgent things that are hard to prioritize. We want interns to have meaningful work, though, and "important + nonurgent" is just about the ideal thing for an intern to work on. They get to work on something interesting, but there's plenty of time for review and refactoring.