4 ms·

>> This was costing us ~$300K/year in compute, and the number kept growing as more customers and detection rules were added. > For something so core to the bus

by ezst 7mo ago

>> This was costing us ~$300K/year in compute, and the number kept growing as more customers and detection rules were added. > For something so core to the business, I'm baffled that they let it get to the point where it was costing $300K per year. And this, this is the core/true/insightful story the executives will never hear about.

sigmoid10 7mo ago
Eh. If you get into enterprise business, this is the accepted management style. AI will now mix this up a little, but before you basically needed to ask if you want to blow 300k on developer salaries to maybe fix something that is already working and generating money, or add more features to the roadmap you can pin on your chest. Scaling infrastructure is the best choice for 90% of managers, especially since they are not the ones paying for it and this kind of technical debt doesn't matter on typical bonus check timeframes.
- xnx 7mo ago
  Managers love big cloud spend so the vendors take them on fancy golf trips ... er ... "Conferences".
- stevepotter 7mo ago
  I used to work for AWS on a service team. I noticed we were spending way too much on provisioned concurrency for dynamo and would benefit from on-demand provisioning. After proving it worked, making the change, deploying, was rather pleased with myself. "Saved $2M in costs by switching to on-demand provisioning" barely made it onto my performance review lol.
  - johnisgood 7mo ago
    In an ideal world, you would have gotten those extra bucks. :P
    - wongarsu 7mo ago
      Or even just 10% of them, or 50% of the first year savings In the world of manufacturing this is known as a gain-sharing plan. Not sure I'd call it common, but it certainly isn't unheard of
  - nitwit005 7mo ago
    They might just not have believed it. At the management level everyone is busy claiming to be delivering huge numbers all the time, and people stop trusting that sort of claim.
- insensible 7mo ago
  Yes, the factors to consider include: - cost of the effort - probability of success - trade-offs in the case of success or of failure - the possibility of only partial success creating an even messier situation than the existing one Having a way to do the whole thing on a much smaller timescale and budget lets decision makers focus more on those externalities, and also can simplify them. This kind of bit rot is somewhere (often everywhere) in many fast-moving businesses, as a natural consequence of the value tradeoffs we have had up to now. Now there are machines that can speedrun the grunt work of clearing them.
mbrumlow 7mo ago
I am always having these arguments. We are paying this other company x a year for something we should build if we really need it. The rebuttals I always get are “I want you working on something that I can’t pay another company for”. I think it sounds good, but in the long run we always end up a budget conversations and head count limits because we spend so much money on external services and software we should just build. Every company ever has this problem. But now with AI. The cost of showing the company “yes we can” is so cheap. I worry for companies who have promotable replacements.

by ezst 7mo ago

sigmoid10 7mo ago

Eh. If you get into enterprise business, this is the accepted management style. AI will now mix this up a little, but before you basically needed to ask if you want to blow 300k on developer salaries to maybe fix something that is already working and generating money, or add more features to the roadmap you can pin on your chest. Scaling infrastructure is the best choice for 90% of managers, especially since they are not the ones paying for it and this kind of technical debt doesn't matter on typical bonus check timeframes.

xnx 7mo ago
Managers love big cloud spend so the vendors take them on fancy golf trips ... er ... "Conferences".
stevepotter 7mo ago
I used to work for AWS on a service team. I noticed we were spending way too much on provisioned concurrency for dynamo and would benefit from on-demand provisioning. After proving it worked, making the change, deploying, was rather pleased with myself. "Saved $2M in costs by switching to on-demand provisioning" barely made it onto my performance review lol.
- johnisgood 7mo ago
  In an ideal world, you would have gotten those extra bucks. :P
  - wongarsu 7mo ago
    Or even just 10% of them, or 50% of the first year savings In the world of manufacturing this is known as a gain-sharing plan. Not sure I'd call it common, but it certainly isn't unheard of
- nitwit005 7mo ago
  They might just not have believed it. At the management level everyone is busy claiming to be delivering huge numbers all the time, and people stop trusting that sort of claim.
insensible 7mo ago
Yes, the factors to consider include: - cost of the effort - probability of success - trade-offs in the case of success or of failure - the possibility of only partial success creating an even messier situation than the existing one Having a way to do the whole thing on a much smaller timescale and budget lets decision makers focus more on those externalities, and also can simplify them. This kind of bit rot is somewhere (often everywhere) in many fast-moving businesses, as a natural consequence of the value tradeoffs we have had up to now. Now there are machines that can speedrun the grunt work of clearing them.

mbrumlow 7mo ago

I am always having these arguments. We are paying this other company x a year for something we should build if we really need it. The rebuttals I always get are “I want you working on something that I can’t pay another company for”. I think it sounds good, but in the long run we always end up a budget conversations and head count limits because we spend so much money on external services and software we should just build. Every company ever has this problem. But now with AI. The cost of showing the company “yes we can” is so cheap. I worry for companies who have promotable replacements.