9 ms·

I was an intern at Mixpanel at the time (but not the author of this article or involved in the discussed rewrite). For context, the company was then <10 people.

I was an intern at Mixpanel at the time (but not the author of this article or involved in the discussed rewrite). For context, the company was then <10 people. (I'm surprised to see anything from my first professional software experience on top of HN!) My recollection of events (of 8 years ago): The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebase. Eventually, it became painful to debug or add features to this endpoint because no one was particularly proficient in Erlang. Ankur (author of the post, intern) then rewrote it in Python. I wouldn't read this as a negative about Erlang or a positive article about Python. I'd instead read it as "use a language that makes sense for your organization and already existing codebase".

oconnor663 7y ago
I think Facebook chat had a very similar experience. The first version was written by someone who liked Erlang. Later on, when some performance issues came up (probably related to GC), no one knew Erlang, and all the high performance backend code in the company was in C++. So it made sense to rewrite the thing.
- trab 7y ago
  Then they acquired whatsapp that was written in Erlang.
  - ludwigvan 7y ago
    And I think I had read that Whatsapp used Erlang because Facebook Chat was using Erlang and they thought Facebook knew what they were doing and decided to use the same tech. What a twist
    - reducesuffering 7y ago
      Where did you read that? I believe Whatsapp used erlang because they leveraged ejabberd [0] for XMPP. Wouldn't surprise me if this is why FB originally used Erlang for the chat servers too. [0] https://en.wikipedia.org/wiki/Ejabberd https://en.wikipedia.org/wiki/Ejabberd
      - darkr 7y ago
        Yep. Facebook chat was originally based on ejabberd. Source: https://web.archive.org/web/20121224094022/http://cufp.galois.com/2009/slides/PiroLetuchy.pdf https://web.archive.org/web/20121224094022/http://cufp.galoi...
        fineoldcannibal 7y ago
        I believe it was because of the Beluga acquisition which was in Erlang
_RPM 7y ago
Is it common for an intern to get such an impactful project?
- exelius 7y ago
  Yeah; it’s actually pretty easy work. Interns have a lot of uncommitted time, and they’re going to leave soon so you don’t want to integrate them into the team too much. It’s also way easier to build to a well-defined spec (which an in-production endpoint would likely have). So you let them rewrite something that already works. If what they build is good, you‘ve just knocked out a good chunk of technical debt and found someone you probably want to hire. If not, they don’t get an offer. Either way you probably didn’t pay them much to begin with so there’s a lot of upside and not much downside.
  - ajross 7y ago
    The question isn't whether or not the work is easy for the intern, it's whether or not it's appropriate. And this cuts both ways: a company that is doing serious core refactoring/redevelopment on the back of an underpaid "intern" is probably exploiting that labor in an unfair way. If it's in fact an unpaid internship, then they're straight up in violation of the Fair Labor Standards Act. And, of course, a company that puts this kind of work on the back of a temporary worker is not treating its own codebase with the respect it deserves. This is rolling the dice with your expertise store -- you have to hope they did it well, because they aren't going to be around to answer questions if they didn't.
    - kbutler 7y ago
      If they didn't build it well, you throw it away, and you're only out what you paid the temporary worker and your code review/acceptance test. The point is this was a well-specified, independent component with an existing functional implementation. This makes an ideal intern/temp/first project, because of those factors - it doesn't require deep knowledge of the organization or other services in the environment or business requirements. You build this, and if we like it, we use it and may give you a job offer. Unless given other information, you can assume the company complied with the law with respect to paying the intern and the intern accepted the internship offer, and it sounds like he got a great experience out of it.
deleted 7y ago
[deleted]
bonif 7y ago
Thanks for this comment, gives context and explains "why we switched from X to Y", answer: I did not know X, or it didn't make sense at the time. It does not mean X is bad, or Y > X (as usually suggested in this kind of articles)
erlangNewb 7y ago
Seems like an oversight that nobody thought to get proficient with erlang. As someone who has started learning it, this seems like the obvious choice.
- diminoten 7y ago
  From the company's perspective, you'd need at least 2 people doing that, preferably everybody. Bus factor of 1 isn't great, and when the rest of the team already knows Python, the benefits of switching to a language everyone knows is huge. You're also presupposing Python is inferior to Erlang, which I'm not really going to argue about here, but I think we can all agree that Python isn't so bad that the decision to continue to use Python substantially and negatively impacted the org's ability to succeed.
- macspoofing 7y ago
  >Seems like an oversight that nobody thought to get proficient with erlang. If it's a component that nobody owns, and nobody cares about, then I'm not very surprised. >As someone who has started learning it, this seems like the obvious choice. The other obvious choice is to rewrite it.
rlander 7y ago
So, in a Python-only org, someone with no Erlang experience decided to ship his first ever server to production, thinking Erlang would be a magic bullet to “scalability issues”. Obviously, there were maintainability issues and the org, blaming the language, decided to switch back to python. To me, the fact that a novice was able to deploy production code that went on unmaintained for months, running a crucial part of the operation, speaks positively about Erlang.
- robocat 7y ago
  "the org, blaming the language". I didn't see anywhere in the article blaming the language - can you find a quote to support your assertion? From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."
bradleyjg 7y ago
In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit. If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.
- michaelterryio 7y ago
  Erlang excels at being used in mixed-language shops. Erlang is an incredible control plane language.
- nojvek 7y ago
  Language diversity is a pragmatic balance. Sometimes a language offers superior abstraction or some guarantees that eliminate a whole host of bugs and manual effort. It’s always worth trying a hypothesis in a small spike and reliably proving whether it will scale to the rest of the org and the impact will be worth the effort. But you’re right, there should be one official supported way of doing some kind of thing at a company.
  - lightcatcher 7y ago
    I think of it as a trade-off between the tool perfectly designed for the job and the tool that can be used for the job of which I'm already an expert user. As an example, my current team's codebase is Python and C++. When we need to do some basic Linux scripting (check if this file exists, if not send an error email), the two prime candidates are Bash and Python. Bash might be exactly designed for this type of thing, but a lot of the team would need to Google stuff like "bash logical and of two booleans" for Bash where they already know the Python syntax. My current rule of thumb here is "<10 lines of code: use Bash. else use Python". More generally, I often try to err on the side of using the tool I know than the unknown tool that might be perfect for the problem.
- 7y ago
mst 7y ago
Your recollection is valuable, and thank you - but contra a large chunk of the rest of this thread, the article was very clear to me that it was "our erlang was crap erlang and we didn't know how to make it not crap so we switched" which is totally reasonable and didn't seem like a negative towards erlang at all. I've supported rewrites from a language I liked more to a language I liked less for similar reasons, and what Ankur did seems like an entirely sane decision. But even if I didn't need the clarification, the additional data was awesome, so my thank you remain sincere :)

oconnor663 7y ago

I think Facebook chat had a very similar experience. The first version was written by someone who liked Erlang. Later on, when some performance issues came up (probably related to GC), no one knew Erlang, and all the high performance backend code in the company was in C++. So it made sense to rewrite the thing.

trab 7y ago
Then they acquired whatsapp that was written in Erlang.
- ludwigvan 7y ago
  And I think I had read that Whatsapp used Erlang because Facebook Chat was using Erlang and they thought Facebook knew what they were doing and decided to use the same tech. What a twist
  - reducesuffering 7y ago
    Where did you read that? I believe Whatsapp used erlang because they leveraged ejabberd [0] for XMPP. Wouldn't surprise me if this is why FB originally used Erlang for the chat servers too. [0] https://en.wikipedia.org/wiki/Ejabberd https://en.wikipedia.org/wiki/Ejabberd
    - darkr 7y ago
      Yep. Facebook chat was originally based on ejabberd. Source: https://web.archive.org/web/20121224094022/http://cufp.galois.com/2009/slides/PiroLetuchy.pdf https://web.archive.org/web/20121224094022/http://cufp.galoi...
      - fineoldcannibal 7y ago
        I believe it was because of the Beluga acquisition which was in Erlang

_RPM 7y ago

Is it common for an intern to get such an impactful project?

exelius 7y ago
Yeah; it’s actually pretty easy work. Interns have a lot of uncommitted time, and they’re going to leave soon so you don’t want to integrate them into the team too much. It’s also way easier to build to a well-defined spec (which an in-production endpoint would likely have). So you let them rewrite something that already works. If what they build is good, you‘ve just knocked out a good chunk of technical debt and found someone you probably want to hire. If not, they don’t get an offer. Either way you probably didn’t pay them much to begin with so there’s a lot of upside and not much downside.
- ajross 7y ago
  The question isn't whether or not the work is easy for the intern, it's whether or not it's appropriate. And this cuts both ways: a company that is doing serious core refactoring/redevelopment on the back of an underpaid "intern" is probably exploiting that labor in an unfair way. If it's in fact an unpaid internship, then they're straight up in violation of the Fair Labor Standards Act. And, of course, a company that puts this kind of work on the back of a temporary worker is not treating its own codebase with the respect it deserves. This is rolling the dice with your expertise store -- you have to hope they did it well, because they aren't going to be around to answer questions if they didn't.
  - kbutler 7y ago
    If they didn't build it well, you throw it away, and you're only out what you paid the temporary worker and your code review/acceptance test. The point is this was a well-specified, independent component with an existing functional implementation. This makes an ideal intern/temp/first project, because of those factors - it doesn't require deep knowledge of the organization or other services in the environment or business requirements. You build this, and if we like it, we use it and may give you a job offer. Unless given other information, you can assume the company complied with the law with respect to paying the intern and the intern accepted the internship offer, and it sounds like he got a great experience out of it.

deleted 7y ago

[deleted]

bonif 7y ago

Thanks for this comment, gives context and explains "why we switched from X to Y", answer: I did not know X, or it didn't make sense at the time. It does not mean X is bad, or Y > X (as usually suggested in this kind of articles)

erlangNewb 7y ago

Seems like an oversight that nobody thought to get proficient with erlang. As someone who has started learning it, this seems like the obvious choice.

diminoten 7y ago
From the company's perspective, you'd need at least 2 people doing that, preferably everybody. Bus factor of 1 isn't great, and when the rest of the team already knows Python, the benefits of switching to a language everyone knows is huge. You're also presupposing Python is inferior to Erlang, which I'm not really going to argue about here, but I think we can all agree that Python isn't so bad that the decision to continue to use Python substantially and negatively impacted the org's ability to succeed.
macspoofing 7y ago
>Seems like an oversight that nobody thought to get proficient with erlang. If it's a component that nobody owns, and nobody cares about, then I'm not very surprised. >As someone who has started learning it, this seems like the obvious choice. The other obvious choice is to rewrite it.

rlander 7y ago

So, in a Python-only org, someone with no Erlang experience decided to ship his first ever server to production, thinking Erlang would be a magic bullet to “scalability issues”. Obviously, there were maintainability issues and the org, blaming the language, decided to switch back to python. To me, the fact that a novice was able to deploy production code that went on unmaintained for months, running a crucial part of the operation, speaks positively about Erlang.

robocat 7y ago
"the org, blaming the language". I didn't see anywhere in the article blaming the language - can you find a quote to support your assertion? From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."

bradleyjg 7y ago

In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit. If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.

michaelterryio 7y ago
Erlang excels at being used in mixed-language shops. Erlang is an incredible control plane language.
nojvek 7y ago
Language diversity is a pragmatic balance. Sometimes a language offers superior abstraction or some guarantees that eliminate a whole host of bugs and manual effort. It’s always worth trying a hypothesis in a small spike and reliably proving whether it will scale to the rest of the org and the impact will be worth the effort. But you’re right, there should be one official supported way of doing some kind of thing at a company.
- lightcatcher 7y ago
  I think of it as a trade-off between the tool perfectly designed for the job and the tool that can be used for the job of which I'm already an expert user. As an example, my current team's codebase is Python and C++. When we need to do some basic Linux scripting (check if this file exists, if not send an error email), the two prime candidates are Bash and Python. Bash might be exactly designed for this type of thing, but a lot of the team would need to Google stuff like "bash logical and of two booleans" for Bash where they already know the Python syntax. My current rule of thumb here is "<10 lines of code: use Bash. else use Python". More generally, I often try to err on the side of using the tool I know than the unknown tool that might be perfect for the problem.
7y ago

mst 7y ago

Your recollection is valuable, and thank you - but contra a large chunk of the rest of this thread, the article was very clear to me that it was "our erlang was crap erlang and we didn't know how to make it not crap so we switched" which is totally reasonable and didn't seem like a negative towards erlang at all. I've supported rewrites from a language I liked more to a language I liked less for similar reasons, and what Ankur did seems like an entirely sane decision. But even if I didn't need the clarification, the additional data was awesome, so my thank you remain sincere :)