5 ms·

Classic story of a startup taking a "good enough" shortcut and then coming back later to optimize. --- I have a similar story: Where I work, we had a cluster

Classic story of a startup taking a "good enough" shortcut and then coming back later to optimize. --- I have a similar story: Where I work, we had a cluster of VMs that were always high CPU and a bit of a problem. We had a lot of fire drills where we'd have to bump up the size of the cluster, abort in-progress operations, or some combination of both. Because this cluster of VMs was doing batch processing that the founder believed should be CPU intense, everyone just assumed that increasing load came with increasing customer size; and that this was just an annoyance that we could get to after we made one more feature. But, at one point the bean counters pointed out that we spent disproportionately more on cloud than a normal business did. After one round of combining different VM clusters (that really didn't need to be separate servers), I decided that I could take some time to hook up this very CPU intense cluster up to a profiler. I thought I was going to be in for a 1-2 week project and would follow a few worms. Instead, the CPU load was because we were constantly loading an entire table, that we never deleted from, into the application's process. The table had transient data that should only last a few hours at most. I quickly deleted almost a decade's worth of obsolete data from the table. After about 15 minutes, CPU usage for this cluster dropped to almost nothing. The next day we made the VM cluster a fraction of its size, and in the next release, we got rid of the cluster and merged the functionality into another cluster. I also made a pull request that introduced a simple filter to the query to only load 3 days of data; and then introduced a background operation to clean out the table periodically.

alsetmusic 2y ago
As much as you can say (perhaps not hard numbers, but as a percentage), what was the savings to the bottom line / cloud costs?
- gwbas1c 2y ago
  Probably ~5% of cloud costs. Combined with the prior round of optimizations, it was substantial. I was really disappointed when my wife couldn't get the night off from work when the company took everyone out to a fancy steak house.
  - chgs 2y ago
    So you saved the company $10k a month and got a $200 meal in gratitude? Awesome.
    - Cyphase 2y ago
      What should they have gotten?
      - tempest_ 2y ago
        They are in theory owed nothing more than their salary but it can be very good for moral to reward that type of thing (assuming they are not introducing a perverse incentive)
    - bagels 2y ago
      They're presumably already being paid a salary to do this work.
    - ponty_rick 2y ago
      They're more pissed about the 1.2M they spent than about the 10k a month they saved.
    - _hyn3 2y ago
      > So you saved the company $10k a month and got a $200 meal in gratitude? Awesome. You seem to be assuming that a $200 meal was the only compensation the person received, and they weren't just getting a nice meal as a little something extra on top of getting paid for doing their job competently and efficiently. But that's the kind of deal I make when I take a job: I do the work (pretty well most of the time), and I get paid. If I stop doing the work, I stop getting paid. If they stop paying, I stop doing the work. (And bonus, literally, if I get a perk once in a while like a free steak dinner that I wasn't expecting) It doesn't have to be more complicated than that.
      - meiraleal 2y ago
        Yeah? Well, proper rewards make those savings and optimizations more common. Otherwise most people will do the work needed just to have work tomorrow.
antisthenes 2y ago
99% of the time, it's either a quadratic (or exponential) algorithm or a really bad DB query.
- dd82 2y ago
  can also be a linear algorithm that does N+1 queries. ORMs can be very good at hiding this implementation detail
  - Quekid5 2y ago
    > a linear algorithm that does N+1 queries. That's what quadratic means.
    - sgerenser 2y ago
      Typically implemented accidentally: https://www.tumblr.com/accidentallyquadratic https://www.tumblr.com/accidentallyquadratic
    - jon_richards 2y ago
      Not really what they're talking about https://stackoverflow.com/questions/97197/what-is-the-n1-selects-problem-in-orm-object-relational-mapping https://stackoverflow.com/questions/97197/what-is-the-n1-sel...
    - nwellnhof 2y ago
      No, N+1 is still linear.
- DylanSp 2y ago
  It wasn't a financial cost, but the biggest single performance improvement I've seen firsthand came from optimizing a SQL query. One of our Professional Services people had written a query that did repeated self-joins on a fairly large table, which took ~15 minutes to run. A DBA-turned-dev on our team rewrote it using MSSQL's PIVOT operator, and the query started executing in less than a second.
magundu 2y ago
Great. It will be great if anyone write a checklist(playbook) to be checked for CPU, Memory, Disk, IO and network issues.
sly010 2y ago
Hah. At a previous place I found that our cloud cost consisted of 90% storage costs. The data? Tens of thousands of incomplete backups of the in-office file server. 3 years of the NAS continuously trying to back itself up to S3 and failing every time.
declan_roberts 2y ago
I love these stories. I have a few as well. In the end I know we're all just doing our job, but I've been tempted at times to say to my manager: "I will save the company $10k/month tomorrow if you give me a cut of the pie."
- euroderf 2y ago
  This should be the norm, actually. It drives ya nuts to read a story where some guy on the shop floor saves his employer ten million dollars and to reward him they give him a 20% off coupon for Home Depot.
  - freedomben 2y ago
    This sounds nice in theory, but would incentivize people to introduce unnecessarily expensive things initially and optimize them later to claim some of the savings. We would like to think that nobody would do something like that, but the sad reality is that there are plenty, especially as the potential reward goes up high enough.
  - zuhsetaqi 2y ago
    > It drives ya nuts to read a story where some guy on the shop floor saves his employer ten million dollars and to reward him they give him a 20% off coupon for Home Depot. It’s your job as an employee, it’s why you get paid in the first place
    - mgkimsal 2y ago
      Is it? Or is an employee's job to just do the work they're asked to do? "It depends" is, of course, the common answer, but in most places I've worked, "please help find operational optimizations that can have a positive impact for the team, department or organization" has certainly never been an explicit ask. Ask team mates to change something to help on a project may fall under the same category, but usually the effect isn't felt beyond a project. My default mode when coming in to jobs is to try to get a 'full company' view, because I want to know how things work and how they might be made better. That approach is usually not met with much enthusiasm, and usually more with "that's not your job, you don't need to know that", etc. I took a daily import routine that was taking 25+ hours (meaning we couldn't show 'daily info' because it was out of date before finishing import) and got it down to 30 minutes. This was after having to fight/argue to see the data, and being told for a couple weeks "it can't be sped up, we'll have to buy faster hardware" ($8-$10k min, but they weren't looking at $15-20k IIRC). I spent a few hours over a weekend and got it down to 30 minutes, and saved the company minimum $8k. But I had to fight/argue to even do that ("that's not your job", "Charles is taking care of that", "the client will just have to deal with more delays while we upgrade", etc).
misstercool 2y ago
It is a good problem to have for a startup, most startups are struggling finding customers to use their thing. Better to go with "good enough" shortcut and prioritize on growth. Recall is a YC company, I am sure they took advantage of huge amount AWS of credits in the first few years.

alsetmusic 2y ago

As much as you can say (perhaps not hard numbers, but as a percentage), what was the savings to the bottom line / cloud costs?

gwbas1c 2y ago
Probably ~5% of cloud costs. Combined with the prior round of optimizations, it was substantial. I was really disappointed when my wife couldn't get the night off from work when the company took everyone out to a fancy steak house.
- chgs 2y ago
  So you saved the company $10k a month and got a $200 meal in gratitude? Awesome.
  - Cyphase 2y ago
    What should they have gotten?
    - tempest_ 2y ago
      They are in theory owed nothing more than their salary but it can be very good for moral to reward that type of thing (assuming they are not introducing a perverse incentive)
  - bagels 2y ago
    They're presumably already being paid a salary to do this work.
  - ponty_rick 2y ago
    They're more pissed about the 1.2M they spent than about the 10k a month they saved.
  - _hyn3 2y ago
    > So you saved the company $10k a month and got a $200 meal in gratitude? Awesome. You seem to be assuming that a $200 meal was the only compensation the person received, and they weren't just getting a nice meal as a little something extra on top of getting paid for doing their job competently and efficiently. But that's the kind of deal I make when I take a job: I do the work (pretty well most of the time), and I get paid. If I stop doing the work, I stop getting paid. If they stop paying, I stop doing the work. (And bonus, literally, if I get a perk once in a while like a free steak dinner that I wasn't expecting) It doesn't have to be more complicated than that.
    - meiraleal 2y ago
      Yeah? Well, proper rewards make those savings and optimizations more common. Otherwise most people will do the work needed just to have work tomorrow.

antisthenes 2y ago

99% of the time, it's either a quadratic (or exponential) algorithm or a really bad DB query.

dd82 2y ago
can also be a linear algorithm that does N+1 queries. ORMs can be very good at hiding this implementation detail
- Quekid5 2y ago
  > a linear algorithm that does N+1 queries. That's what quadratic means.
  - sgerenser 2y ago
    Typically implemented accidentally: https://www.tumblr.com/accidentallyquadratic https://www.tumblr.com/accidentallyquadratic
  - jon_richards 2y ago
    Not really what they're talking about https://stackoverflow.com/questions/97197/what-is-the-n1-selects-problem-in-orm-object-relational-mapping https://stackoverflow.com/questions/97197/what-is-the-n1-sel...
  - nwellnhof 2y ago
    No, N+1 is still linear.
DylanSp 2y ago
It wasn't a financial cost, but the biggest single performance improvement I've seen firsthand came from optimizing a SQL query. One of our Professional Services people had written a query that did repeated self-joins on a fairly large table, which took ~15 minutes to run. A DBA-turned-dev on our team rewrote it using MSSQL's PIVOT operator, and the query started executing in less than a second.

magundu 2y ago

Great. It will be great if anyone write a checklist(playbook) to be checked for CPU, Memory, Disk, IO and network issues.

sly010 2y ago

Hah. At a previous place I found that our cloud cost consisted of 90% storage costs. The data? Tens of thousands of incomplete backups of the in-office file server. 3 years of the NAS continuously trying to back itself up to S3 and failing every time.

declan_roberts 2y ago

I love these stories. I have a few as well. In the end I know we're all just doing our job, but I've been tempted at times to say to my manager: "I will save the company $10k/month tomorrow if you give me a cut of the pie."

euroderf 2y ago
This should be the norm, actually. It drives ya nuts to read a story where some guy on the shop floor saves his employer ten million dollars and to reward him they give him a 20% off coupon for Home Depot.
- freedomben 2y ago
  This sounds nice in theory, but would incentivize people to introduce unnecessarily expensive things initially and optimize them later to claim some of the savings. We would like to think that nobody would do something like that, but the sad reality is that there are plenty, especially as the potential reward goes up high enough.
- zuhsetaqi 2y ago
  > It drives ya nuts to read a story where some guy on the shop floor saves his employer ten million dollars and to reward him they give him a 20% off coupon for Home Depot. It’s your job as an employee, it’s why you get paid in the first place
  - mgkimsal 2y ago
    Is it? Or is an employee's job to just do the work they're asked to do? "It depends" is, of course, the common answer, but in most places I've worked, "please help find operational optimizations that can have a positive impact for the team, department or organization" has certainly never been an explicit ask. Ask team mates to change something to help on a project may fall under the same category, but usually the effect isn't felt beyond a project. My default mode when coming in to jobs is to try to get a 'full company' view, because I want to know how things work and how they might be made better. That approach is usually not met with much enthusiasm, and usually more with "that's not your job, you don't need to know that", etc. I took a daily import routine that was taking 25+ hours (meaning we couldn't show 'daily info' because it was out of date before finishing import) and got it down to 30 minutes. This was after having to fight/argue to see the data, and being told for a couple weeks "it can't be sped up, we'll have to buy faster hardware" ($8-$10k min, but they weren't looking at $15-20k IIRC). I spent a few hours over a weekend and got it down to 30 minutes, and saved the company minimum $8k. But I had to fight/argue to even do that ("that's not your job", "Charles is taking care of that", "the client will just have to deal with more delays while we upgrade", etc).

misstercool 2y ago

It is a good problem to have for a startup, most startups are struggling finding customers to use their thing. Better to go with "good enough" shortcut and prioritize on growth. Recall is a YC company, I am sure they took advantage of huge amount AWS of credits in the first few years.