4 ms·

I've mentioned this before, in my company (big media company) I saw some S3 costs creeping up each month. I looked into it and it was a system we abandoned that

I've mentioned this before, in my company (big media company) I saw some S3 costs creeping up each month. I looked into it and it was a system we abandoned that was still copying files to this bucket. I reached out to the team and they turned it off, it saved us $1m a year. The higher-ups rewarded me by telling me that a team should have caught this so I should meet with them now.

tuananh 3y ago
true. the first step is to have visibility into what's eating the bill. it's just like you need to profile the program before optimizing it. that's why we did finops dashboard first thing when we first started the cloud journey. can't optimize if you dont know.
Salgat 3y ago
It's truly fascinating how companies won't bat at an eye at spending ungodly amounts of money on things they don't need, but will sweat profusely at the thought of a tiny fraction of that going towards additional compensation.
- anotherhue 3y ago
  Enriching AWS does not threaten their social standing.
- preommr 3y ago
  Something something snakes, something something unintended consequences.
  - Salgat 3y ago
    Sabotaging your company's infrastructure for a bonus is a great way to end up in prison.
    - Marrand 3y ago
      How would you prove intent to sabotage?
      - Salgat 3y ago
        Hopefully your company doesn't have its head up its ass and has basic things like logs setup. The vast majority of what I do in production is logged through AWS (Cloudtrail) or git.
max_ 3y ago
You got rewarded with more meetings? Not a bonus?
- thefourthchime 3y ago
  Yep
stevejb 3y ago
I work for a company called CloudFix, and we are solely focused on AWS costs. We do automated AWS cost optimization. We find one of two reactions when we deliver savings to customers: (A) "Hey wow, this is great! We are so excited to be saving from here on out." OR, (B) "This should have been caught earlier. $TEAM was supposed to be experts..." and then blame game starts. It is really unfortunate when institutions react in the latter way. Often the engineers are assigned to cost optimization, along with a million other things. And, the incentives aren't really aligned well to reward savings. For example, S3 Intelligent Tiering is the right thing in 99.9% of cases - so it should be your default bucket type. BUT, engineers often face only downside risk for the change, and very little upside reward. And, it isn't their money so they just leave it. The cost of overprovisioned S3 can be staggering! What is really needed is to establish a proper FinOps discipline, put someone in charge of cost savings, and make sure incentives are aligned properly. And of course check out CloudFix if you can!
- thefourthchime 3y ago
  We work with a competitor called Vega, the product seems OK although the UI is very slow and confusing. The biggest problem they have is they have no business insight into what these costs are, and if we can reduce the cost without any kind of engineering, effort, or loss of performance.

tuananh 3y ago

true. the first step is to have visibility into what's eating the bill. it's just like you need to profile the program before optimizing it. that's why we did finops dashboard first thing when we first started the cloud journey. can't optimize if you dont know.

Salgat 3y ago

It's truly fascinating how companies won't bat at an eye at spending ungodly amounts of money on things they don't need, but will sweat profusely at the thought of a tiny fraction of that going towards additional compensation.

anotherhue 3y ago
Enriching AWS does not threaten their social standing.
preommr 3y ago
Something something snakes, something something unintended consequences.
- Salgat 3y ago
  Sabotaging your company's infrastructure for a bonus is a great way to end up in prison.
  - Marrand 3y ago
    How would you prove intent to sabotage?
    - Salgat 3y ago
      Hopefully your company doesn't have its head up its ass and has basic things like logs setup. The vast majority of what I do in production is logged through AWS (Cloudtrail) or git.

max_ 3y ago

You got rewarded with more meetings? Not a bonus?

thefourthchime 3y ago
Yep

stevejb 3y ago

I work for a company called CloudFix, and we are solely focused on AWS costs. We do automated AWS cost optimization. We find one of two reactions when we deliver savings to customers: (A) "Hey wow, this is great! We are so excited to be saving from here on out." OR, (B) "This should have been caught earlier. $TEAM was supposed to be experts..." and then blame game starts. It is really unfortunate when institutions react in the latter way. Often the engineers are assigned to cost optimization, along with a million other things. And, the incentives aren't really aligned well to reward savings. For example, S3 Intelligent Tiering is the right thing in 99.9% of cases - so it should be your default bucket type. BUT, engineers often face only downside risk for the change, and very little upside reward. And, it isn't their money so they just leave it. The cost of overprovisioned S3 can be staggering! What is really needed is to establish a proper FinOps discipline, put someone in charge of cost savings, and make sure incentives are aligned properly. And of course check out CloudFix if you can!

thefourthchime 3y ago
We work with a competitor called Vega, the product seems OK although the UI is very slow and confusing. The biggest problem they have is they have no business insight into what these costs are, and if we can reduce the cost without any kind of engineering, effort, or loss of performance.