8 ms·

I love how Facebook and Instagram never went the route of "full rewrite" for their apps as they scaled. I my experience "Language X is slow and we could save Y

by msoad 5y ago

I love how Facebook and Instagram never went the route of "full rewrite" for their apps as they scaled. I my experience "Language X is slow and we could save Y switching to Z" is always a false promise. You can pick parts of the system that are costing a lot and port them to other language/frameworks to capture the bulk of savings while keeping your developers happy working on familiar code. Or if you'e big enough like Facebook, you can go see why X is slow and if it is possible to improve at the compiler level. Never disrupt the developers flow (Twitter did this with their Scala craze back in the day)

re 5y ago
> keeping your developers happy working on familiar code That turns pretty quickly into "developers being unhappy that they have to maintain legacy code written for the old language/platform," though, especially with any sort of employee turnover.
- peterhunt 5y ago
  It doesn't have to be this way. If you're going to go all-in on the monolith approach like FB and Instagram did for their product code (at least when I was there), you need to have a central team whose entire job is scaling the codebase. This means actively refactoring other teams' legacy code, making sure that tests are fast, etc.
  - skrtskrt 5y ago
    Yeah I have seen this successfully done at a smallish (200 person) company. There was a "platform" team which was essentially an "un-suck the monolith team". Refactor slowly over time, turn a ball of mud into cleanly defined domain boundaries, parallelize the tests, etc. etc.
kevingadd 5y ago
I think while performance is not a proper justification, if you get other stuff in the bargain it can be really worth it. Rust, for example - the justification to switch is usually not just speed but safety and the safety enabling more optimizations, parallelism etc. If all you care about is speed you stay in C/C++. Sometimes the choice is also made when you already have to pay a transitional cost - at a previous employer we had to rewrite a bunch of code to move from an old version of PHP to a newer one, and a strong argument was made that at that point we should just adopt a better language since we already had to pay that cost. I think eventually a lot of stuff transitioned to Haskell as part of that process, even if other stuff stayed in PHP.
halfmatthalfcat 5y ago
Did Twitter actually contribute to Scala core though? They definitely created and contributed to various Scala libraries but they didn't fork/augment the Scala compiler like Facebook did with HipHop.
- spullara 5y ago
  They did contribute to the JVM but directly, not by forking it.
  - pjmlp 5y ago
    And are nowadays one of the reference users of GraalVM in production.
    - spullara 5y ago
      Oh right! I forgot they moved to that.
neya 5y ago
I disagree, this logic doesn't work always unless you have investor money lying around. For a bootstrapped business, it could mean living or dying. As an example, a client of mine was paying close to $5000 per month in server costs simply because of the scale of traffic they had on their site. By re-writing Wordpress in Elixir, I was able to bring their costs down to $1000 odd per month. It's actually cheaper than what any of their competitors are paying for, in servers as well. This $4000 in savings actually allowed them to hire an additional full time developer to maintain their site. So, your logic only works for certain use cases, not all.
- leesalminen 5y ago
  How much did you charge to rewrite WordPress in elixir though?
  - code-is-code 5y ago
    And how big was the risk to have the invested time wasted? How long did you block new feature developement?
    - neya 5y ago
      Safe to say they would've gone out of business if they kept burning that kind of money for the remainder of the year. This was around covid era, they didn't have a solid revenue stream from the traffic they got.
  - neya 5y ago
    I'm obviously not going to reveal actual numbers, but it's safe to they spent less than what they would have spent being on Wordpress for the remainder of the year. Time to completion is 3 months. No new "feature" development was blocked. Remember - this is a content website (Wordpress is mostly used for content AFAIK and less for building features). The moment it was done, we just switched nameservers. It had no impact on client's workflow or day to day life.
    - 5y ago
spullara 5y ago
If Twitter's port to the JVM didn't result in 10x fewer servers with 10x lower latency vs an identical Ruby implementation we wouldn't have done it. Sadly not much progress has been made on the RubyVM though Twitter did try with improvements to GC and other changes. We even tried JRuby as part of the evaluation.
- tekknolagi 5y ago
  Shopify is working on this! They have YJIT, led by maximecb
- nerdponx 5y ago
  Is TruffleRuby promising at all?
  - jashmatthews 5y ago
    Enormously. The latest Railsbench results on TruffleRuby are ~3x CRuby.
    - spullara 5y ago
      Nice. 10 years later but glad that Ruby programmers might have an option that lets them program the way they want without having to choose a different language / platform for performance.
      - ksec 5y ago
        It is still no where near the level that Twitter would have stayed with Ruby though. Although I believe TruffleRuby being 3x faster than current CRuby is only the beginning. JVM got tons of improvement in the past 10 years as well it is uncanny. Although Dr Chris Seaton mentioned it is not his goal to have Shopify Rails running on top of TurffleRuby. Not sure if he meant it was short term or long term. I think Dr Stefan Marr will be joining him soon, and Dr Maxime Chevalier-Boisvert is leading a team working on YJIT in Shopify. ( Did Shopify just hire all known Dr. working on Ruby performance )
cageface 5y ago
"Language X is slow and we could save Y switching to Z" is always a false promise. The thing that keeps software interesting for me is that there are almost no absolute rules. So I'll agree that complete rewrites are usually a mistake and performance problems are often not in the language. But I can't agree that rewrites are always a mistake. esbuild is a recent good example of how much difference switching to a faster language can make.
mgraczyk 5y ago
Idk, I worked at IG for a year and wrote a lot of backend code in the Python webserver stack. I believe they would benefit massively by aggressively migrating services to the FB Hack stack. A ton of work goes into making sure the Python code isn't slow, and there are complicated C++ services built as workarounds for the slowness of python.
- vpql 5y ago
  Heresy! This is a Python thread, please summon the required positivity!
nemothekid 5y ago
I'm not sure many companies have the luxury of doing what FB did. I can't imagine a world where choosing to rewrite your app, or choosing to rewrite the entire language, where rewriting the entire language would be the cheaper solution. Twitter did this with their Scala craze, but Twitter struggled to scale it's Ruby app. Twitter didn't have nearly as deep pockets to write a performant alternative Ruby VM.
- deleted 5y ago
  [deleted]
- jfoutz 5y ago
  Joel Spolsky did it, city something?
  - chucksmash 5y ago
    I believe the product was called "CityDesk" and the in-house language that compiled to several different target languages was called Wasabi. Edit: I was wrong, Fog Creek did have a "CityDesk" product at one point, but Wasabi was for on-prem FogBugz[1]. [1]: https://www.joelonsoftware.com/2006/09/01/wasabi/ https://www.joelonsoftware.com/2006/09/01/wasabi/
flakiness 5y ago
Yeah, it kind of reminded me the "python strict mode" they talked about before [1]. No designed to be cool but to be useful. [1] https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834 https://instagram-engineering.com/python-at-scale-strict-mod...
- czardoz 5y ago
  Yes, Static Python especially relies heavily on strict modules, since they enable us to perform module-local analysis, which enables some cool optimizations.
- bjoli 5y ago
  This is something that people have known for a long time: a lot of money has been spent on making compilers make fast code. For many intents and purposes we know what kind of reasoning we want to be able to do to produce fast code. Strict modules is probably one of the sinples yet efficient optimizations you can do: being able to reason about where and what something is without having to do an expensive lookup every time it is used is brain dead simple to understand. Yet many of the languages that evolved as "scripting languages" threw much of that knowledge out. In the middle of the 90s we knew how to make fast dynamic languages. Maybe not as crazy dynamic as something like ruby, but not far off. Or at least, they had the sense to codify it as a separate thing (CLOS in common lisp comes to mind). Something like SBCL is more dynamic than you need(TM), yet produces code that is often at least an order of magnitude faster than python.
joe_the_user 5y ago
One factor for Twitter was that Ruby was simply orders of magnitude worse for Twitter usage than I think Php or Python was for Facebook or instagram. Or at least one never heard of these providing horror stories where Twitter definitely had horror stories. In one of the few times I've been "close to the source", I heard Twitter's head of operations complaining the electricity had become their primary expense from starting X many Ruby instances per second. It was one instance per connection or something close, because Ruby was unstable and because Ruby never gave memory back to the system.
drunkenmagician 5y ago
Not sure I can agree with that. Sinking resources in to re-engineering a language infrastructure, mostly because the language / infra is not actually a good choice for the target problem is an approach only very large engineering teams can tackle. As has been already been started (further down the thread). Breaking down your code / monolith into manageable components and re targeting for performance in the key spots is almost always the right approach. Unless of course you have the engineering resources and hubris of someone like FB
deleted 5y ago
[deleted]
guenthert 5y ago
Rewriting in full for the sake of performance gain is indeed a questionable proposal. Rewriting hot spots being the more sensible approach. A better reason for rewriting however is to migrate to a more maintainable language with focus on readability from one which focus on expressiveness (and hence favored for rapid development) as the project matures.

by msoad 5y ago

re 5y ago

> keeping your developers happy working on familiar code That turns pretty quickly into "developers being unhappy that they have to maintain legacy code written for the old language/platform," though, especially with any sort of employee turnover.

peterhunt 5y ago
It doesn't have to be this way. If you're going to go all-in on the monolith approach like FB and Instagram did for their product code (at least when I was there), you need to have a central team whose entire job is scaling the codebase. This means actively refactoring other teams' legacy code, making sure that tests are fast, etc.
- skrtskrt 5y ago
  Yeah I have seen this successfully done at a smallish (200 person) company. There was a "platform" team which was essentially an "un-suck the monolith team". Refactor slowly over time, turn a ball of mud into cleanly defined domain boundaries, parallelize the tests, etc. etc.

kevingadd 5y ago

I think while performance is not a proper justification, if you get other stuff in the bargain it can be really worth it. Rust, for example - the justification to switch is usually not just speed but safety and the safety enabling more optimizations, parallelism etc. If all you care about is speed you stay in C/C++. Sometimes the choice is also made when you already have to pay a transitional cost - at a previous employer we had to rewrite a bunch of code to move from an old version of PHP to a newer one, and a strong argument was made that at that point we should just adopt a better language since we already had to pay that cost. I think eventually a lot of stuff transitioned to Haskell as part of that process, even if other stuff stayed in PHP.

halfmatthalfcat 5y ago

Did Twitter actually contribute to Scala core though? They definitely created and contributed to various Scala libraries but they didn't fork/augment the Scala compiler like Facebook did with HipHop.

spullara 5y ago
They did contribute to the JVM but directly, not by forking it.
- pjmlp 5y ago
  And are nowadays one of the reference users of GraalVM in production.
  - spullara 5y ago
    Oh right! I forgot they moved to that.

neya 5y ago

I disagree, this logic doesn't work always unless you have investor money lying around. For a bootstrapped business, it could mean living or dying. As an example, a client of mine was paying close to $5000 per month in server costs simply because of the scale of traffic they had on their site. By re-writing Wordpress in Elixir, I was able to bring their costs down to $1000 odd per month. It's actually cheaper than what any of their competitors are paying for, in servers as well. This $4000 in savings actually allowed them to hire an additional full time developer to maintain their site. So, your logic only works for certain use cases, not all.

leesalminen 5y ago
How much did you charge to rewrite WordPress in elixir though?
- code-is-code 5y ago
  And how big was the risk to have the invested time wasted? How long did you block new feature developement?
  - neya 5y ago
    Safe to say they would've gone out of business if they kept burning that kind of money for the remainder of the year. This was around covid era, they didn't have a solid revenue stream from the traffic they got.
- neya 5y ago
  I'm obviously not going to reveal actual numbers, but it's safe to they spent less than what they would have spent being on Wordpress for the remainder of the year. Time to completion is 3 months. No new "feature" development was blocked. Remember - this is a content website (Wordpress is mostly used for content AFAIK and less for building features). The moment it was done, we just switched nameservers. It had no impact on client's workflow or day to day life.
  - 5y ago

spullara 5y ago

If Twitter's port to the JVM didn't result in 10x fewer servers with 10x lower latency vs an identical Ruby implementation we wouldn't have done it. Sadly not much progress has been made on the RubyVM though Twitter did try with improvements to GC and other changes. We even tried JRuby as part of the evaluation.

tekknolagi 5y ago
Shopify is working on this! They have YJIT, led by maximecb
nerdponx 5y ago
Is TruffleRuby promising at all?
- jashmatthews 5y ago
  Enormously. The latest Railsbench results on TruffleRuby are ~3x CRuby.
  - spullara 5y ago
    Nice. 10 years later but glad that Ruby programmers might have an option that lets them program the way they want without having to choose a different language / platform for performance.
    - ksec 5y ago
      It is still no where near the level that Twitter would have stayed with Ruby though. Although I believe TruffleRuby being 3x faster than current CRuby is only the beginning. JVM got tons of improvement in the past 10 years as well it is uncanny. Although Dr Chris Seaton mentioned it is not his goal to have Shopify Rails running on top of TurffleRuby. Not sure if he meant it was short term or long term. I think Dr Stefan Marr will be joining him soon, and Dr Maxime Chevalier-Boisvert is leading a team working on YJIT in Shopify. ( Did Shopify just hire all known Dr. working on Ruby performance )

cageface 5y ago

"Language X is slow and we could save Y switching to Z" is always a false promise. The thing that keeps software interesting for me is that there are almost no absolute rules. So I'll agree that complete rewrites are usually a mistake and performance problems are often not in the language. But I can't agree that rewrites are always a mistake. esbuild is a recent good example of how much difference switching to a faster language can make.

mgraczyk 5y ago

Idk, I worked at IG for a year and wrote a lot of backend code in the Python webserver stack. I believe they would benefit massively by aggressively migrating services to the FB Hack stack. A ton of work goes into making sure the Python code isn't slow, and there are complicated C++ services built as workarounds for the slowness of python.

vpql 5y ago
Heresy! This is a Python thread, please summon the required positivity!

nemothekid 5y ago

I'm not sure many companies have the luxury of doing what FB did. I can't imagine a world where choosing to rewrite your app, or choosing to rewrite the entire language, where rewriting the entire language would be the cheaper solution. Twitter did this with their Scala craze, but Twitter struggled to scale it's Ruby app. Twitter didn't have nearly as deep pockets to write a performant alternative Ruby VM.

deleted 5y ago
[deleted]
jfoutz 5y ago
Joel Spolsky did it, city something?
- chucksmash 5y ago
  I believe the product was called "CityDesk" and the in-house language that compiled to several different target languages was called Wasabi. Edit: I was wrong, Fog Creek did have a "CityDesk" product at one point, but Wasabi was for on-prem FogBugz[1]. [1]: https://www.joelonsoftware.com/2006/09/01/wasabi/ https://www.joelonsoftware.com/2006/09/01/wasabi/

flakiness 5y ago

Yeah, it kind of reminded me the "python strict mode" they talked about before [1]. No designed to be cool but to be useful. [1] https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834 https://instagram-engineering.com/python-at-scale-strict-mod...

czardoz 5y ago
Yes, Static Python especially relies heavily on strict modules, since they enable us to perform module-local analysis, which enables some cool optimizations.
bjoli 5y ago
This is something that people have known for a long time: a lot of money has been spent on making compilers make fast code. For many intents and purposes we know what kind of reasoning we want to be able to do to produce fast code. Strict modules is probably one of the sinples yet efficient optimizations you can do: being able to reason about where and what something is without having to do an expensive lookup every time it is used is brain dead simple to understand. Yet many of the languages that evolved as "scripting languages" threw much of that knowledge out. In the middle of the 90s we knew how to make fast dynamic languages. Maybe not as crazy dynamic as something like ruby, but not far off. Or at least, they had the sense to codify it as a separate thing (CLOS in common lisp comes to mind). Something like SBCL is more dynamic than you need(TM), yet produces code that is often at least an order of magnitude faster than python.

joe_the_user 5y ago

One factor for Twitter was that Ruby was simply orders of magnitude worse for Twitter usage than I think Php or Python was for Facebook or instagram. Or at least one never heard of these providing horror stories where Twitter definitely had horror stories. In one of the few times I've been "close to the source", I heard Twitter's head of operations complaining the electricity had become their primary expense from starting X many Ruby instances per second. It was one instance per connection or something close, because Ruby was unstable and because Ruby never gave memory back to the system.

drunkenmagician 5y ago

Not sure I can agree with that. Sinking resources in to re-engineering a language infrastructure, mostly because the language / infra is not actually a good choice for the target problem is an approach only very large engineering teams can tackle. As has been already been started (further down the thread). Breaking down your code / monolith into manageable components and re targeting for performance in the key spots is almost always the right approach. Unless of course you have the engineering resources and hubris of someone like FB

deleted 5y ago

[deleted]

guenthert 5y ago

Rewriting in full for the sake of performance gain is indeed a questionable proposal. Rewriting hot spots being the more sensible approach. A better reason for rewriting however is to migrate to a more maintainable language with focus on readability from one which focus on expressiveness (and hence favored for rapid development) as the project matures.