10 ms·

And people underestimate how well some solid, dumb solutions can scale. Boring spring boot with a decent data model and a bit effort to stay stateless scales to

by tetha 3y ago

And people underestimate how well some solid, dumb solutions can scale. Boring spring boot with a decent data model and a bit effort to stay stateless scales to the moon. Or, we're having a grand "data export" system customers use to collect data from us for their own DWHs. It has survived 2 attempts at replacement so far. At it's core, it's psql + rsync, or was recently migrated to psql + s3 when we decomissioned our FTP servers. And it's easy to extend and customers are happy because it integrates well.

tyingq 3y ago
Yep. Facebook got pretty far down the road with PHP, MySQL, memcached, etc.
- fauigerzigerk 3y ago
  All they had to do was write a PHP compiler and a new storage engine for MySQL.
  - tyingq 3y ago
    HipHop (later HHVM) was around 2010, so they scaled from 2004-2010 before that became needed. MyRocks was 2015. Wikipedia says FB was around 300 million users in 2009, then 400 million users in 2010.
    - fauigerzigerk 3y ago
      Yes, good point. But you have to wonder what kind of engineering effort went into scaling PHP and MySQL up to the point where they decided to build a compiler and a storage engine.
      - rafark 3y ago
        When you have half a billion users that both read and write all day, you have to optimize, no matter the tech.
        fauigerzigerk 3y ago
        That is undeniably true, but I do think the starting point still matters.
      - disgruntledphd2 3y ago
        It was much, much, much cheaper than a rewrite would have been, that's why they did it. Edit: also, in 2004 when they got started, what else could they have used?
  - ericbarrett 3y ago
    The trick was there was enough growth that the savings from the compiler were massive. (I worked there at the time.) The inefficiency of the PHP interpreter was a great problem to have, because it came from the success it enabled.
    - fauigerzigerk 3y ago
      So I think the interesting question is whether the rest of us can learn anything from what happened there. I believe Mark Zuckerberg simply used the technology he knew and took it from there. That's fine. I probably would have done the same thing. But many people are making an ideology out of this, arguing that not giving a shit about performance is always the right choice initially because that's how you grow fast enough to be able to retool later. I think this is based assumptions that are no longer true. In the early 2000s, mainstream programming languages and runtimes were either fast and low productivity or slow and high productivity (Exceptions such as Pascal/Delphi did exist but they were not mainstream). And the cost of scaling up was prohibitive compared to scaling out. Today, you can choose any fast high productivity language/runtime and go very far mostly scaling up.
      - rafark 3y ago
        Learn what? That you should use the language that you’re more comfortable with and then scale? Or that languages have become more efficient? Php 8, for example, is many times faster than the php 4 and 5 that Facebook was using.
        fauigerzigerk 3y ago
        I think using what you already know remains a choice that is very hard to criticise. But we didn't have to learn that, did we? Beyond that I think there is more to unlearn than to learn from the history of the Y2K batch of startups. The economics of essentially everything related to writing, running and distributing software have changed completely.
        noir_lord 3y ago
  - cagenut 3y ago
    your comment implies your understanding of the timeline is backwards. they had to do those things after they had gotten hundreds of millions of users.
    - fauigerzigerk 3y ago
      Depends on what you consider "far down the road" and what they had to do before writing a compiler and a storage engine. How long did it take until Facebook engineers realised that their technology stack was not the best tool for the job? It definitely wasn't the day when they decided to build a compiler and a storage engine.
      - tyingq 3y ago
        I'm not sure there was really a best tool for the job in 2003-2004 that would have been high-level enough to be productive, and scalable enough to stay mostly as-is. Java, maybe.
        fauigerzigerk 3y ago
        I agree, and I'm not criticising the choices that Mark Zuckerberg made at the time. But we are no longer facing the same situation he did. We do now have high productivity, high performance language runtimes. And scaling up has become much cheaper (relative to the number of users you can serve). That's why I think it can't hurt to remind people of the great lengths to which Facebook had to go in order to deal with the limitations of their chosen platform.
        balder1991 3y ago
        Yeah, I kinda don’t agree with the dichotomy of “you either optimize or you build features”. They’re not exclusive. If you understand the tools and their trade offs you should be able to use the right tools for the job which won’t hinder you in the future. Of course if all you know is JavaScript, then that requires going a bit outside your comfort zone.
phkahler 3y ago
>> And people underestimate how well some solid, dumb solutions can scale. I think this started with the old Apache web server. When the www got started, that server did the job so everyone used it. The problem was it didn't scale, so all kinds of cool solutions (load balancers and such) were developed and everyone building something bigger than a personal blog used that stuff. For most the root problem was that Apache had terrible performance. Nginx has solved that now, and we also have faster hardware and networks, so anything less than HN can probably be hosted on an rpi on your home network. OK, I'm exaggerating, but only a little. Bottom line is that scaling is still treated like a big fundamental problem for everyone, but it doesn't need to be.
ilyt 3y ago
> And people underestimate how well some solid, dumb solutions can scale. I'd say they underestimate how long "just throwing money" (servers) at the problem can work. If you earn decent money now, scaling number of app servers 10x times to serve 10x times the traffic will still earn decent money. Doesn't matter that "PHP is slow", deal with it when your infrastructure cost warrant hiring more/better developers to fix it. Especially now. Pair of fat servers on some NVMes and 1TB of RAM gonna cost you less than few dev-months and that can serve plenty of users in most use cases even before any extra caching will be needed.
- waboremo 3y ago
  Even then, you don't have to throw money at the problem right away. If you feel you can save time and money by using Rust instead of PHP (just using the two languages as examples, not a specific indication of Rust or PHP's resource drain), go ahead. Making that decision early on costs nothing. It's only after a project is off the ground that caring about these decisions winds up wasting everyone's time, that's when you wind up slowing momentum tremendously due to dangling a potential new toy in front of your team.
Cthulhu_ 3y ago
> And people underestimate how well some solid, dumb solutions can scale. And overestimate how expensive it is to just add another server, or underestimate how expensive a rebuild is. But then, part of that is also that IT departments want the budget; if they don't spend their annual budgets on innovation, their budget gets cut. Or in my neck of the woods, the EU subsidies will stop if they don't have anything they can file as "innovation".

by tetha 3y ago

tyingq 3y ago

Yep. Facebook got pretty far down the road with PHP, MySQL, memcached, etc.

fauigerzigerk 3y ago
All they had to do was write a PHP compiler and a new storage engine for MySQL.
- tyingq 3y ago
  HipHop (later HHVM) was around 2010, so they scaled from 2004-2010 before that became needed. MyRocks was 2015. Wikipedia says FB was around 300 million users in 2009, then 400 million users in 2010.
  - fauigerzigerk 3y ago
    Yes, good point. But you have to wonder what kind of engineering effort went into scaling PHP and MySQL up to the point where they decided to build a compiler and a storage engine.
    - rafark 3y ago
      When you have half a billion users that both read and write all day, you have to optimize, no matter the tech.
      - fauigerzigerk 3y ago
        That is undeniably true, but I do think the starting point still matters.
    - disgruntledphd2 3y ago
      It was much, much, much cheaper than a rewrite would have been, that's why they did it. Edit: also, in 2004 when they got started, what else could they have used?
- ericbarrett 3y ago
  The trick was there was enough growth that the savings from the compiler were massive. (I worked there at the time.) The inefficiency of the PHP interpreter was a great problem to have, because it came from the success it enabled.
  - fauigerzigerk 3y ago
    So I think the interesting question is whether the rest of us can learn anything from what happened there. I believe Mark Zuckerberg simply used the technology he knew and took it from there. That's fine. I probably would have done the same thing. But many people are making an ideology out of this, arguing that not giving a shit about performance is always the right choice initially because that's how you grow fast enough to be able to retool later. I think this is based assumptions that are no longer true. In the early 2000s, mainstream programming languages and runtimes were either fast and low productivity or slow and high productivity (Exceptions such as Pascal/Delphi did exist but they were not mainstream). And the cost of scaling up was prohibitive compared to scaling out. Today, you can choose any fast high productivity language/runtime and go very far mostly scaling up.
    - rafark 3y ago
      Learn what? That you should use the language that you’re more comfortable with and then scale? Or that languages have become more efficient? Php 8, for example, is many times faster than the php 4 and 5 that Facebook was using.
      - fauigerzigerk 3y ago
        I think using what you already know remains a choice that is very hard to criticise. But we didn't have to learn that, did we? Beyond that I think there is more to unlearn than to learn from the history of the Y2K batch of startups. The economics of essentially everything related to writing, running and distributing software have changed completely.
      - noir_lord 3y ago
- cagenut 3y ago
  your comment implies your understanding of the timeline is backwards. they had to do those things after they had gotten hundreds of millions of users.
  - fauigerzigerk 3y ago
    Depends on what you consider "far down the road" and what they had to do before writing a compiler and a storage engine. How long did it take until Facebook engineers realised that their technology stack was not the best tool for the job? It definitely wasn't the day when they decided to build a compiler and a storage engine.
    - tyingq 3y ago
      I'm not sure there was really a best tool for the job in 2003-2004 that would have been high-level enough to be productive, and scalable enough to stay mostly as-is. Java, maybe.
      - fauigerzigerk 3y ago
        I agree, and I'm not criticising the choices that Mark Zuckerberg made at the time. But we are no longer facing the same situation he did. We do now have high productivity, high performance language runtimes. And scaling up has become much cheaper (relative to the number of users you can serve). That's why I think it can't hurt to remind people of the great lengths to which Facebook had to go in order to deal with the limitations of their chosen platform.
        balder1991 3y ago
        Yeah, I kinda don’t agree with the dichotomy of “you either optimize or you build features”. They’re not exclusive. If you understand the tools and their trade offs you should be able to use the right tools for the job which won’t hinder you in the future. Of course if all you know is JavaScript, then that requires going a bit outside your comfort zone.

phkahler 3y ago

>> And people underestimate how well some solid, dumb solutions can scale. I think this started with the old Apache web server. When the www got started, that server did the job so everyone used it. The problem was it didn't scale, so all kinds of cool solutions (load balancers and such) were developed and everyone building something bigger than a personal blog used that stuff. For most the root problem was that Apache had terrible performance. Nginx has solved that now, and we also have faster hardware and networks, so anything less than HN can probably be hosted on an rpi on your home network. OK, I'm exaggerating, but only a little. Bottom line is that scaling is still treated like a big fundamental problem for everyone, but it doesn't need to be.

ilyt 3y ago

> And people underestimate how well some solid, dumb solutions can scale. I'd say they underestimate how long "just throwing money" (servers) at the problem can work. If you earn decent money now, scaling number of app servers 10x times to serve 10x times the traffic will still earn decent money. Doesn't matter that "PHP is slow", deal with it when your infrastructure cost warrant hiring more/better developers to fix it. Especially now. Pair of fat servers on some NVMes and 1TB of RAM gonna cost you less than few dev-months and that can serve plenty of users in most use cases even before any extra caching will be needed.

waboremo 3y ago
Even then, you don't have to throw money at the problem right away. If you feel you can save time and money by using Rust instead of PHP (just using the two languages as examples, not a specific indication of Rust or PHP's resource drain), go ahead. Making that decision early on costs nothing. It's only after a project is off the ground that caring about these decisions winds up wasting everyone's time, that's when you wind up slowing momentum tremendously due to dangling a potential new toy in front of your team.

Cthulhu_ 3y ago

> And people underestimate how well some solid, dumb solutions can scale. And overestimate how expensive it is to just add another server, or underestimate how expensive a rebuild is. But then, part of that is also that IT departments want the budget; if they don't spend their annual budgets on innovation, their budget gets cut. Or in my neck of the woods, the EU subsidies will stop if they don't have anything they can file as "innovation".