3 ms·

I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possib

I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse? It's rhetorical really as I know why. Terrible architecture from "enterprise". Stick everything in a single account so it's hard to figure out how much is your spend. All 3000 databases, and make sure your k8s cluster is 5 8XL boxes so no one can scale down excess capacity. Classic I Burn Money consultancy!

darth_avocado 3y ago
Locking down the UI is definitely a problem. Another problem I have seen is, not being able to accurately tell even if you have the UI.
- happymellon 3y ago
  We had a battle to get AWS console access in the first place, after that I had to deal with: > You need to request access every 60 days. Luckily it's now added to my permanent role, but even then no billing access? FFS.
tuananh 3y ago
> 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse? This is so true. billing transparency is very important. in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill
malfist 3y ago
Lots of that is just bad design by going the easiest route. It's easy not to grant engineers access to the billing dashboard. It's easy to put everything in the same aws account. Inside Amazon, we're supposed to set up new aws accounts for every service and realm, so we know how much X service's beta environment is costing
- happymellon 3y ago
  Indeed, you need to share resources? Plenty of ways of doing that, like making a cross account shared VPC for example. Everything is still accountable.
macNchz 3y ago
> getting as much as possible out of a system for cheap is great fun In certain circumstances, absolutely, however it's extremely aggravating to be in the position of being constantly pestered to ship features faster without the authority to overprovision some of the infrastructure the software runs on. Waking up in the middle of the night because we saved money by allocating too little disk for the primary database or because the latest release included new dependencies that increased memory usage and the OOMKiller is picking off web servers like a wolf in the lamb's pen, or we're just swapping our way to hell while web requests 502...eh. Not for me. More visibility into costs, though, absolutely agreed. Engineers should know that when they turn on some new cool serverless gizmo and then forget about it, it's costing $ each month.
- happymellon 3y ago
  Completely agree. I didn't mean that engineers love having no control over their systems, I just see the labours of love that get posted here about getting nginx to throw out 1000 pages a second on an Atari 800, or getting LLMs designed for $2000 GPUs running on a phone. The question should be, we currently cost $X a month and we need to half it because [reason], what can we do to bring it down? Which might be reducing hardware, or maybe something else, might be both. Puzzles can be fun.

darth_avocado 3y ago

Locking down the UI is definitely a problem. Another problem I have seen is, not being able to accurately tell even if you have the UI.

happymellon 3y ago
We had a battle to get AWS console access in the first place, after that I had to deal with: > You need to request access every 60 days. Luckily it's now added to my permanent role, but even then no billing access? FFS.

tuananh 3y ago

> 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse? This is so true. billing transparency is very important. in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill

malfist 3y ago

Lots of that is just bad design by going the easiest route. It's easy not to grant engineers access to the billing dashboard. It's easy to put everything in the same aws account. Inside Amazon, we're supposed to set up new aws accounts for every service and realm, so we know how much X service's beta environment is costing

happymellon 3y ago
Indeed, you need to share resources? Plenty of ways of doing that, like making a cross account shared VPC for example. Everything is still accountable.

macNchz 3y ago

> getting as much as possible out of a system for cheap is great fun In certain circumstances, absolutely, however it's extremely aggravating to be in the position of being constantly pestered to ship features faster without the authority to overprovision some of the infrastructure the software runs on. Waking up in the middle of the night because we saved money by allocating too little disk for the primary database or because the latest release included new dependencies that increased memory usage and the OOMKiller is picking off web servers like a wolf in the lamb's pen, or we're just swapping our way to hell while web requests 502...eh. Not for me. More visibility into costs, though, absolutely agreed. Engineers should know that when they turn on some new cool serverless gizmo and then forget about it, it's costing $ each month.

happymellon 3y ago
Completely agree. I didn't mean that engineers love having no control over their systems, I just see the labours of love that get posted here about getting nginx to throw out 1000 pages a second on an Atari 800, or getting LLMs designed for $2000 GPUs running on a phone. The question should be, we currently cost $X a month and we need to half it because [reason], what can we do to bring it down? Which might be reducing hardware, or maybe something else, might be both. Puzzles can be fun.