3 ms·

> With numbers like that Twitter could, profitably, hire dozens of engineers that do literally nothing else. Just tweak thread pool sizes all day, every day, fo

> With numbers like that Twitter could, profitably, hire dozens of engineers that do literally nothing else. Just tweak thread pool sizes all day, every day, for service after service. Even though it's a boring, manual, "noncomplex" thing, this type of work is clearly valuable and should have happened years ago. The issue is that once you hire a dozen engineers to do this (say for 5M a year in total), and they do it for a year, they save mid 8 figures (keep in mind this was the largest service, so the savings across other services will be smaller). Then can they keep saving mid 8 figures every year? I'll paraphrase something I previously wrote privately, but imagine you have some team that's able to save 10% of your fleetwide resources this year. They densify and optimize and improve defaults. So you can now grow your services by 10% without any additional cost increase, and you do! The next year, they've already saved the easiest 10%. Can they save 10% again? Can they keep it up every year? How long until they're saving 3% a year, or 1% a year? And that's if you keep the team the same size, where its clearly loosing money! If you could afford a dozen people to save 10%, you can only really afford 1-2 to save 1%, but then you're likely to get an even smaller return. Unless you expect to be able to maintain the same relative value of optimizations every year, 3 or 5 years out, its not worth it to hire an FTE to work on them. I should note that I've experienced this myself: I was working in an area where resource optimization could lead to "significant" savings (not 8 figures, but 6 or maybe 7). My first 6 months working in this area, I found all sorts of low hanging fruit and saved a fair amount. The second six months, 5-10x less ROI. I gave up even trying in the third six moths, if I come across a thing, I'll fix it, but its no longer worthwhile to look.

ZephyrBlu 5y ago
If you're looking in the same area/domain, what you're saying is almost certainly true. If you're looking across the business as a whole, it seems likely that there is a lot of this kind of work lying around because there is not much incentive for people to tackle it as described in this comment: https://news.ycombinator.com/item?id=29691847 https://news.ycombinator.com/item?id=29691847.
- joshuamorton 5y ago
  I'm replying to that comment. The issue is still the same: what do you do in the second year (or the third), after you've fixed all the low hanging fruit. If it takes a constant amount of time to investigate a particular service, it's not even worth examining the long tail, because the investigation is costlier than the savings. So once you've saved the high 8 figures by fixing the low hanging fruit in the biggest 20 services or whatever, what next? The broader point is that while it is often very worthwhile to have individual employees work on optimizations, it is much less often worthwhile to task teams entirely with optimizations, especially in the way you describe. Having some group in charge of generalizable fleetwide optimizations is possibly useful. Doing some kind of resource quota thing where you get fewer resources than you're forecast to need (to force you to do some kind of local optimization) may make sense, but having a strike team whose job it is to tune JVM parameters isn't useful. A central team writing a document on best practices, setting better defaults, or using/building some kind of grid-search optimizer are all probably better investments.
  - ptc 5y ago
    The answer is you build a team that builds tools to make optimization easier for everyone in the engineering organization. One individual team can't optimize every application, but if you make it easy for every engineering team to profile their applications and debug performance problems, you've enabled every team to continuously optimize the low-hanging fruit. In effect, you're no longer solving one-off 8-figure problems, you're optimizing the time of other engineers which is something that pays off in perpetuity. Good large engineering organizations have these teams, and understand their value. For smaller companies, it's not as clear if you can reasonably fund a team for that kind of work.
  - YokoZar 5y ago
    > The issue is still the same: what do you do in the second year (or the third), after you've fixed all the low hanging fruit. Then you destaff the team, collect your tens of millions of dollars, and be happy about a successfully finished project? The problem with going too far down this line of reasoning is that you leave even low-hanging fruit laying around forever.

ZephyrBlu 5y ago

If you're looking in the same area/domain, what you're saying is almost certainly true. If you're looking across the business as a whole, it seems likely that there is a lot of this kind of work lying around because there is not much incentive for people to tackle it as described in this comment: https://news.ycombinator.com/item?id=29691847 https://news.ycombinator.com/item?id=29691847.

joshuamorton 5y ago
I'm replying to that comment. The issue is still the same: what do you do in the second year (or the third), after you've fixed all the low hanging fruit. If it takes a constant amount of time to investigate a particular service, it's not even worth examining the long tail, because the investigation is costlier than the savings. So once you've saved the high 8 figures by fixing the low hanging fruit in the biggest 20 services or whatever, what next? The broader point is that while it is often very worthwhile to have individual employees work on optimizations, it is much less often worthwhile to task teams entirely with optimizations, especially in the way you describe. Having some group in charge of generalizable fleetwide optimizations is possibly useful. Doing some kind of resource quota thing where you get fewer resources than you're forecast to need (to force you to do some kind of local optimization) may make sense, but having a strike team whose job it is to tune JVM parameters isn't useful. A central team writing a document on best practices, setting better defaults, or using/building some kind of grid-search optimizer are all probably better investments.
- ptc 5y ago
  The answer is you build a team that builds tools to make optimization easier for everyone in the engineering organization. One individual team can't optimize every application, but if you make it easy for every engineering team to profile their applications and debug performance problems, you've enabled every team to continuously optimize the low-hanging fruit. In effect, you're no longer solving one-off 8-figure problems, you're optimizing the time of other engineers which is something that pays off in perpetuity. Good large engineering organizations have these teams, and understand their value. For smaller companies, it's not as clear if you can reasonably fund a team for that kind of work.
- YokoZar 5y ago
  > The issue is still the same: what do you do in the second year (or the third), after you've fixed all the low hanging fruit. Then you destaff the team, collect your tens of millions of dollars, and be happy about a successfully finished project? The problem with going too far down this line of reasoning is that you leave even low-hanging fruit laying around forever.