5 ms·

> ... unless it becomes a crazy persons labour of love Or there are valid business reasons to keep chipping away, in particular on performance-per-dollar probl

by finnh 4y ago

> ... unless it becomes a crazy persons labour of love Or there are valid business reasons to keep chipping away, in particular on performance-per-dollar problems at scale. If you're spending $1M/month on hardware to run something, each 10% win is worth enough to employ a couple senior engineers.

deleted 4y ago
[deleted]
abriosi 4y ago
Many senior engineers who operate systems which spend 1M/month make 500k yearly 1M * 12 * 10% = 1.2M 500k * 12 * 2 * taxes >= 1.2M How many 10% wins do they have to achieve in what timeframe to have a good ROI on micro-optimization, considering that these seniors could be producing stuff that allows for multiple times their salary in revenue?
- charleslmunger 4y ago
  Improvements are one-time investments of engineering time, but ongoing savings in infra costs. Consider also that these savings also permit scaling to more customers or more work. Suppose you have two engineers that total to $1M of personnel cost. If they deliver a cost savings of 1% per month (compounding) the savings-per-month will have surpassed their pay per month; you will see your infra costs at 85% of what they were after 16 months, and on month 17 you will have made a net profit on those engineers, where the savings they got you has totalled more than what you've paid them. If your business is growing, you could project your infra costs to grow beyond $1M per month, and the numbers become even more favorable.
  - abriosi 4y ago
    Unless a company is running an engineering team composed of monkeys, it will be hard to achieve 1% optimization per month, even without compounding. There are a lot of optimizations which look good on paper but in practice bring other unforeseen cons Anyway, a profitable software company will be looking at getting a big multiple of 10x the salary of their engineers
    - saagarjha 4y ago
      I work in perform engineering, although in a client rather than backend role. We usually do better than 1% a month. In fact we do this even as >1% of regressions make it into the build from the software development process.
    - charleslmunger 4y ago
      It depends what you're optimizing. Mature c++ codebase that is routinely profiled? Large legacy codebase with few tests? Mixed workloads? All hard. But there's also a lot of easy cases out there - even good engineers might code like monkeys if they're under time pressure and trying to grow the business. I've done a lot of performance work, and while of course no one individual could achieve a 1% savings per month on the entirety of FAANG compute loads, there is lots of code out there that nobody thought would run at scale or be on the critical path of something important, but now it is and it has gotten expensive.
- CSMastermind 4y ago
  As a rule of thumb, I devote about 0.5% of my engineering org to cost optimization. At about 200 people it makes sense to invest a single engineer. By the team you're approaching 800 it makes sense to have a team dedicates to it. Though cost opt generally involves way more than optimizing code. It's normally about auditing things, creating budget tooling, right sizing, broader architecture redesigns, etc.
  - klodolph 4y ago
    I don't see how a rule of thumb could make sense, since the cost of operations can vary so much from company to company.
- bilkow 4y ago
  You shouldn't multiply by 12 if the 500k is yearly. Also what's the 2? 2 engineers? The correct math would be: 1M * 12 * 10% = 1.2M 500k * 2 * taxes >= 1M
  - abriosi 4y ago
    Someone said "a couple engineers" hence 2. The 12 was a bad copy paste, thanks for pointing that out
Cthulhu_ 4y ago
I mean in this case, I'm confident that the codebase for Mario64 was used as a basis for subsequent N64 titles; when pioneering something like this, you want to spend some extra time during or after development to tweak these codebases.
- losingom 4y ago
  It was definitely used for at least Ocarina of Time and Majora's Mask, albeit tweaked. I don't doubt that they used it in other titles too.

by finnh 4y ago

deleted 4y ago

[deleted]

abriosi 4y ago

Many senior engineers who operate systems which spend 1M/month make 500k yearly 1M * 12 * 10% = 1.2M 500k * 12 * 2 * taxes >= 1.2M How many 10% wins do they have to achieve in what timeframe to have a good ROI on micro-optimization, considering that these seniors could be producing stuff that allows for multiple times their salary in revenue?

charleslmunger 4y ago
Improvements are one-time investments of engineering time, but ongoing savings in infra costs. Consider also that these savings also permit scaling to more customers or more work. Suppose you have two engineers that total to $1M of personnel cost. If they deliver a cost savings of 1% per month (compounding) the savings-per-month will have surpassed their pay per month; you will see your infra costs at 85% of what they were after 16 months, and on month 17 you will have made a net profit on those engineers, where the savings they got you has totalled more than what you've paid them. If your business is growing, you could project your infra costs to grow beyond $1M per month, and the numbers become even more favorable.
- abriosi 4y ago
  Unless a company is running an engineering team composed of monkeys, it will be hard to achieve 1% optimization per month, even without compounding. There are a lot of optimizations which look good on paper but in practice bring other unforeseen cons Anyway, a profitable software company will be looking at getting a big multiple of 10x the salary of their engineers
  - saagarjha 4y ago
    I work in perform engineering, although in a client rather than backend role. We usually do better than 1% a month. In fact we do this even as >1% of regressions make it into the build from the software development process.
  - charleslmunger 4y ago
    It depends what you're optimizing. Mature c++ codebase that is routinely profiled? Large legacy codebase with few tests? Mixed workloads? All hard. But there's also a lot of easy cases out there - even good engineers might code like monkeys if they're under time pressure and trying to grow the business. I've done a lot of performance work, and while of course no one individual could achieve a 1% savings per month on the entirety of FAANG compute loads, there is lots of code out there that nobody thought would run at scale or be on the critical path of something important, but now it is and it has gotten expensive.
CSMastermind 4y ago
As a rule of thumb, I devote about 0.5% of my engineering org to cost optimization. At about 200 people it makes sense to invest a single engineer. By the team you're approaching 800 it makes sense to have a team dedicates to it. Though cost opt generally involves way more than optimizing code. It's normally about auditing things, creating budget tooling, right sizing, broader architecture redesigns, etc.
- klodolph 4y ago
  I don't see how a rule of thumb could make sense, since the cost of operations can vary so much from company to company.
bilkow 4y ago
You shouldn't multiply by 12 if the 500k is yearly. Also what's the 2? 2 engineers? The correct math would be: 1M * 12 * 10% = 1.2M 500k * 2 * taxes >= 1M
- abriosi 4y ago
  Someone said "a couple engineers" hence 2. The 12 was a bad copy paste, thanks for pointing that out

Cthulhu_ 4y ago

I mean in this case, I'm confident that the codebase for Mario64 was used as a basis for subsequent N64 titles; when pioneering something like this, you want to spend some extra time during or after development to tweak these codebases.

losingom 4y ago
It was definitely used for at least Ocarina of Time and Majora's Mask, albeit tweaked. I don't doubt that they used it in other titles too.