5 ms·

> The gains for doing this for individual large services are significant (in the case of service-1, it's [mid 7 figures per year] for the service and [low 8 fig

> The gains for doing this for individual large services are significant (in the case of service-1, it's [mid 7 figures per year] for the service and [low 8 figures per year] including services that are clones of it, but tuning every service by hand isn't scalable. This point seems wrong to me, bound too much by requiring solutions to be done by a small team of engineers who already have a mandate to work on the problem. With numbers like that Twitter could, profitably, hire dozens of engineers that do literally nothing else. Just tweak thread pool sizes all day, every day, for service after service. Even though it's a boring, manual, "noncomplex" thing, this type of work is clearly valuable and should have happened years ago. Most likely Twitter's job ladder, promotion process, and hiring pipeline is highly incentivizing people to avoid such work even when it has clear impact. They are very much not alone in that regard.

xyzzy_plugh 5y ago
I solved this same problem for a company also in 2019 (as the CPU quota bug hadn't been fixed yet) and it resulted in something like 8 figures of yearly cost savings. You are correct in that most companies are not equipped to staff issues like this. Most places just accept their bills as a cost of doing business, not something that can be optimized.
- deleted 5y ago
  [deleted]
- eloff 5y ago
  This. I've never worked anywhere that had dedicated ongoing effort to cost reduction in compute services. It's always a once every couple of years thing to look at the cloud spending and spend a little effort dealing with the low hanging fruit.
- _jal 5y ago
  A side effect of deciding systems engineers can be replaced by devops. In reality, you want both. A good systems person can save you a ton of money.
  - ithkuil 5y ago
    A nice approach is to staff a DevOps team with people from diverse backgrounds; some more towards the system side of the spectrum and some more towards the dev side of the spectrum. As long as everybody knows a little bit of the other side. This helps ok avoiding a culture where devs "throw some code over the fence" and sysops people just moan that devs are careless and/or that they should do things differently, but without a clear way of showing exactly how differently things should be made (and also without a clear understanding of what devs ended up choosing the way they choose)
    - bradknowles 5y ago
      Who does DevOps teams anymore? Don’t you know that all devs should also be ops, and you can cut your staff in half? Seriously, some employers I’ve know seem to take exactly that approach. Just fire all the ops guys, and tell the devs that they’re now going to be doing DevOps.
      - ithkuil 5y ago
        Oops I clearly didn't mean a dedicated DevOps team; I mean a team that develops and runs a product, what DevOps originally meant (and not the glorified developer tools team that often people have nowadays with that name)
        waynesonfire 5y ago
        Devops developed and ran product? Never heard of that. What disconnected manifesto did you read?
        ithkuil 5y ago
        Is the word product rather than service that bothers you or the fact that DevOps develop and operate?
- karmakaze 5y ago
  I can see how there are diminishing returns when optimizing but I would never say that server bills are not a metric to be aware of and address. I've always had some idea of what's practically achievable in terms of efficiency within a given architecture and aim for something that gets a good amount of the way there without undue effort. I also enjoy thinking of longer term improvements for efficiency whether that could improve latency or the bottom line and at the same time know that's secondary to providing additional value and gaining customers during a growth period.
vegetablepotpie 5y ago
They could hire contractors or consultants to do the job, no? That class of worker would not be concerned about promotion opportunities. For some reason they haven’t done that either.
ghusbands 5y ago
Certainly, at one place I worked, the higher-ups were very clear that any work on cost reduction was wasted and devs and ops should always work on increasing net income, not decreasing costs. It was consistently claimed that cost reduction can only get you small percentage decreases, whereas increases in income are larger and compound better.
- tomrod 5y ago
  The higher ups need some basic economics education, it appears. Certainly you shouldn't invest everything in long term returns, but you should be open to it. Instead, when something has a payoff of 3 years, executives get antsy in orgs that have a 2-year cycle on exec positions.
- avianlyric 5y ago
  The big difference between cost reduction and income increase, is that one has a hard limit on possible upside, whereas the other does not. You can reduce your costs by more than your total costs, but it’s quite possible to increase your income by many multiples of your existing income. Result is that maximising income is generally better than reducing cost. Of course, as with all generalisation, there are situations where this approach doesn’t hold true. But as a high level first order strategy, it’s a good one to adopt.
  - wolf550e 5y ago
    You meant to write "You cannot reduce your costs"
  - ClumsyPilot 5y ago
    "one has a hard limit on possible upside, whereas the other does not." Thats plain wrong, the gloval market for cars, bicycles and what have you has a limited size. Every large company that's a market leader understands that.
    - beiller 5y ago
      Would capturing 100% of the market granting a monopoly essentially grant unlimited upside? Cause you can just jack the price to absurdity? Also wouldn't reducing cost to zero have an infinite upside as well? Basically zero cost you can produce infinite output. Gettin real pedantic here heh.
mnutt 5y ago
There was a startup I talked to at one point that had a service where they’d run an agent on your instances that would collect performance data and live tune kernel parameters, and they had some AI to find the best parameters for your workload. No idea how well it worked, but it seems like a potentially good application of AI.
- servytor 5y ago
  Do you remember the name for it? Sounds really useful.
  - syngrog66 5y ago
    Log4Shell.com
  - wmf 5y ago
    Was it Granulate?
    - mnutt 5y ago
      Yes, that was it.
spockz 5y ago
Afaik, Twitter already has significant and mature infrastructure in place to run a plethora of different instances on shadowed traffic and compare settings. It is used at least by the people working on the optimised jre.
joshuamorton 5y ago
> With numbers like that Twitter could, profitably, hire dozens of engineers that do literally nothing else. Just tweak thread pool sizes all day, every day, for service after service. Even though it's a boring, manual, "noncomplex" thing, this type of work is clearly valuable and should have happened years ago. The issue is that once you hire a dozen engineers to do this (say for 5M a year in total), and they do it for a year, they save mid 8 figures (keep in mind this was the largest service, so the savings across other services will be smaller). Then can they keep saving mid 8 figures every year? I'll paraphrase something I previously wrote privately, but imagine you have some team that's able to save 10% of your fleetwide resources this year. They densify and optimize and improve defaults. So you can now grow your services by 10% without any additional cost increase, and you do! The next year, they've already saved the easiest 10%. Can they save 10% again? Can they keep it up every year? How long until they're saving 3% a year, or 1% a year? And that's if you keep the team the same size, where its clearly loosing money! If you could afford a dozen people to save 10%, you can only really afford 1-2 to save 1%, but then you're likely to get an even smaller return. Unless you expect to be able to maintain the same relative value of optimizations every year, 3 or 5 years out, its not worth it to hire an FTE to work on them. I should note that I've experienced this myself: I was working in an area where resource optimization could lead to "significant" savings (not 8 figures, but 6 or maybe 7). My first 6 months working in this area, I found all sorts of low hanging fruit and saved a fair amount. The second six months, 5-10x less ROI. I gave up even trying in the third six moths, if I come across a thing, I'll fix it, but its no longer worthwhile to look.
- ZephyrBlu 5y ago
  If you're looking in the same area/domain, what you're saying is almost certainly true. If you're looking across the business as a whole, it seems likely that there is a lot of this kind of work lying around because there is not much incentive for people to tackle it as described in this comment: https://news.ycombinator.com/item?id=29691847 https://news.ycombinator.com/item?id=29691847.
  - joshuamorton 5y ago
    I'm replying to that comment. The issue is still the same: what do you do in the second year (or the third), after you've fixed all the low hanging fruit. If it takes a constant amount of time to investigate a particular service, it's not even worth examining the long tail, because the investigation is costlier than the savings. So once you've saved the high 8 figures by fixing the low hanging fruit in the biggest 20 services or whatever, what next? The broader point is that while it is often very worthwhile to have individual employees work on optimizations, it is much less often worthwhile to task teams entirely with optimizations, especially in the way you describe. Having some group in charge of generalizable fleetwide optimizations is possibly useful. Doing some kind of resource quota thing where you get fewer resources than you're forecast to need (to force you to do some kind of local optimization) may make sense, but having a strike team whose job it is to tune JVM parameters isn't useful. A central team writing a document on best practices, setting better defaults, or using/building some kind of grid-search optimizer are all probably better investments.

xyzzy_plugh 5y ago

I solved this same problem for a company also in 2019 (as the CPU quota bug hadn't been fixed yet) and it resulted in something like 8 figures of yearly cost savings. You are correct in that most companies are not equipped to staff issues like this. Most places just accept their bills as a cost of doing business, not something that can be optimized.

deleted 5y ago
[deleted]
eloff 5y ago
This. I've never worked anywhere that had dedicated ongoing effort to cost reduction in compute services. It's always a once every couple of years thing to look at the cloud spending and spend a little effort dealing with the low hanging fruit.
_jal 5y ago
A side effect of deciding systems engineers can be replaced by devops. In reality, you want both. A good systems person can save you a ton of money.
- ithkuil 5y ago
  A nice approach is to staff a DevOps team with people from diverse backgrounds; some more towards the system side of the spectrum and some more towards the dev side of the spectrum. As long as everybody knows a little bit of the other side. This helps ok avoiding a culture where devs "throw some code over the fence" and sysops people just moan that devs are careless and/or that they should do things differently, but without a clear way of showing exactly how differently things should be made (and also without a clear understanding of what devs ended up choosing the way they choose)
  - bradknowles 5y ago
    Who does DevOps teams anymore? Don’t you know that all devs should also be ops, and you can cut your staff in half? Seriously, some employers I’ve know seem to take exactly that approach. Just fire all the ops guys, and tell the devs that they’re now going to be doing DevOps.
    - ithkuil 5y ago
      Oops I clearly didn't mean a dedicated DevOps team; I mean a team that develops and runs a product, what DevOps originally meant (and not the glorified developer tools team that often people have nowadays with that name)
      - waynesonfire 5y ago
        Devops developed and ran product? Never heard of that. What disconnected manifesto did you read?
        ithkuil 5y ago
        Is the word product rather than service that bothers you or the fact that DevOps develop and operate?
karmakaze 5y ago
I can see how there are diminishing returns when optimizing but I would never say that server bills are not a metric to be aware of and address. I've always had some idea of what's practically achievable in terms of efficiency within a given architecture and aim for something that gets a good amount of the way there without undue effort. I also enjoy thinking of longer term improvements for efficiency whether that could improve latency or the bottom line and at the same time know that's secondary to providing additional value and gaining customers during a growth period.

vegetablepotpie 5y ago

They could hire contractors or consultants to do the job, no? That class of worker would not be concerned about promotion opportunities. For some reason they haven’t done that either.

ghusbands 5y ago

Certainly, at one place I worked, the higher-ups were very clear that any work on cost reduction was wasted and devs and ops should always work on increasing net income, not decreasing costs. It was consistently claimed that cost reduction can only get you small percentage decreases, whereas increases in income are larger and compound better.

tomrod 5y ago
The higher ups need some basic economics education, it appears. Certainly you shouldn't invest everything in long term returns, but you should be open to it. Instead, when something has a payoff of 3 years, executives get antsy in orgs that have a 2-year cycle on exec positions.
avianlyric 5y ago
The big difference between cost reduction and income increase, is that one has a hard limit on possible upside, whereas the other does not. You can reduce your costs by more than your total costs, but it’s quite possible to increase your income by many multiples of your existing income. Result is that maximising income is generally better than reducing cost. Of course, as with all generalisation, there are situations where this approach doesn’t hold true. But as a high level first order strategy, it’s a good one to adopt.
- wolf550e 5y ago
  You meant to write "You cannot reduce your costs"
- ClumsyPilot 5y ago
  "one has a hard limit on possible upside, whereas the other does not." Thats plain wrong, the gloval market for cars, bicycles and what have you has a limited size. Every large company that's a market leader understands that.
  - beiller 5y ago
    Would capturing 100% of the market granting a monopoly essentially grant unlimited upside? Cause you can just jack the price to absurdity? Also wouldn't reducing cost to zero have an infinite upside as well? Basically zero cost you can produce infinite output. Gettin real pedantic here heh.

mnutt 5y ago

There was a startup I talked to at one point that had a service where they’d run an agent on your instances that would collect performance data and live tune kernel parameters, and they had some AI to find the best parameters for your workload. No idea how well it worked, but it seems like a potentially good application of AI.

servytor 5y ago
Do you remember the name for it? Sounds really useful.
- syngrog66 5y ago
  Log4Shell.com
- wmf 5y ago
  Was it Granulate?
  - mnutt 5y ago
    Yes, that was it.

spockz 5y ago

Afaik, Twitter already has significant and mature infrastructure in place to run a plethora of different instances on shadowed traffic and compare settings. It is used at least by the people working on the optimised jre.

joshuamorton 5y ago

> With numbers like that Twitter could, profitably, hire dozens of engineers that do literally nothing else. Just tweak thread pool sizes all day, every day, for service after service. Even though it's a boring, manual, "noncomplex" thing, this type of work is clearly valuable and should have happened years ago. The issue is that once you hire a dozen engineers to do this (say for 5M a year in total), and they do it for a year, they save mid 8 figures (keep in mind this was the largest service, so the savings across other services will be smaller). Then can they keep saving mid 8 figures every year? I'll paraphrase something I previously wrote privately, but imagine you have some team that's able to save 10% of your fleetwide resources this year. They densify and optimize and improve defaults. So you can now grow your services by 10% without any additional cost increase, and you do! The next year, they've already saved the easiest 10%. Can they save 10% again? Can they keep it up every year? How long until they're saving 3% a year, or 1% a year? And that's if you keep the team the same size, where its clearly loosing money! If you could afford a dozen people to save 10%, you can only really afford 1-2 to save 1%, but then you're likely to get an even smaller return. Unless you expect to be able to maintain the same relative value of optimizations every year, 3 or 5 years out, its not worth it to hire an FTE to work on them. I should note that I've experienced this myself: I was working in an area where resource optimization could lead to "significant" savings (not 8 figures, but 6 or maybe 7). My first 6 months working in this area, I found all sorts of low hanging fruit and saved a fair amount. The second six months, 5-10x less ROI. I gave up even trying in the third six moths, if I come across a thing, I'll fix it, but its no longer worthwhile to look.

ZephyrBlu 5y ago
If you're looking in the same area/domain, what you're saying is almost certainly true. If you're looking across the business as a whole, it seems likely that there is a lot of this kind of work lying around because there is not much incentive for people to tackle it as described in this comment: https://news.ycombinator.com/item?id=29691847 https://news.ycombinator.com/item?id=29691847.
- joshuamorton 5y ago
  I'm replying to that comment. The issue is still the same: what do you do in the second year (or the third), after you've fixed all the low hanging fruit. If it takes a constant amount of time to investigate a particular service, it's not even worth examining the long tail, because the investigation is costlier than the savings. So once you've saved the high 8 figures by fixing the low hanging fruit in the biggest 20 services or whatever, what next? The broader point is that while it is often very worthwhile to have individual employees work on optimizations, it is much less often worthwhile to task teams entirely with optimizations, especially in the way you describe. Having some group in charge of generalizable fleetwide optimizations is possibly useful. Doing some kind of resource quota thing where you get fewer resources than you're forecast to need (to force you to do some kind of local optimization) may make sense, but having a strike team whose job it is to tune JVM parameters isn't useful. A central team writing a document on best practices, setting better defaults, or using/building some kind of grid-search optimizer are all probably better investments.