12 ms·

Cutting down AWS cost by $150k per year simply by shutting things off

191 points by tuananh 3y ago

anuptalwalkar 3y ago
Couple of years ago I saved about 14 mn of revenue per year for my company. I got a 250$ bonus for it.
- schnebbau 3y ago
  This is why you should start the conversation with "I have drawn up a plan to save the company $14M per year. I will execute this plan in exchange for $7M upon completion." If they say no then just go back to your regular duties.
  - munchler 3y ago
    Because blackmail is an effective salary negotiation tactic?
  - listenallyall 3y ago
    Very few companies would make this deal at 0.7M, 0.07M, or even 0.007M. Direct sharing a % of savings with the responsible employee is simply not the way most companies work. Consultants, now, that's different...
    - charlie0 3y ago
      For a slice of 7M, just quit and come back as consultant.
- llanowarelves 3y ago
  Learned a lesson that you only have to "spend" (forgo) $250 to cause that company $14m in losses (that you could have prevented)
- nvm0n2 3y ago
  But was that your job? Because if so, you really got salary+bonus. And if you'd found nothing you'd still have got salary. So you can look at it several ways.
zikduruqe 3y ago
Shit... I reduced our spend on one AWS account from $270K a month to $75K a month.
- ponector 3y ago
  Great! Shareholders will be happy. Did you receive any bonus?
  - marvin 3y ago
    It's a good shoo-in for becoming a freelance consultant charging a percentage of annual savings rather than hourly time billed.
  - Chico75 3y ago
    The problem is always around abuse. If it becomes known that you can get a big bonus by wasting a lot of money on useless infra first and then reducing it, other people will start playing the game. How do you reward cloud cost awareness without creating perverse incentives?
    - nvm0n2 3y ago
      It's always the same answer: managers who pay attention to the details. People familiar with your work should be able to tell if you're gaming the system or not.
  - zen928 3y ago
    Will they also be paying what they owe on the added costs that could have been noticed earlier with due diligence? I'm imagining that what they'll receive instead is the compensation expected and agreed upon by both parties negotiated during either initial hiring or the multiple points in the year that allow for easy communication about changing payroll expectations, instead of hawking for dimes at first sight.
  - zikduruqe 3y ago
    Hahaha, for doing my job? I wish.
sremani 3y ago
Cloud optimization is the next Kubernetes.
- hkt 3y ago
  Goodness I hope so, that'll be easy money
  - bdcravens 3y ago
    <quinnypig has entered the chat>
    - QuinnyPig 3y ago
      I HAVE BEEN SUMMONED
thefourthchime 3y ago
I've mentioned this before, in my company (big media company) I saw some S3 costs creeping up each month. I looked into it and it was a system we abandoned that was still copying files to this bucket. I reached out to the team and they turned it off, it saved us $1m a year. The higher-ups rewarded me by telling me that a team should have caught this so I should meet with them now.
- tuananh 3y ago
  true. the first step is to have visibility into what's eating the bill. it's just like you need to profile the program before optimizing it. that's why we did finops dashboard first thing when we first started the cloud journey. can't optimize if you dont know.
- Salgat 3y ago
  It's truly fascinating how companies won't bat at an eye at spending ungodly amounts of money on things they don't need, but will sweat profusely at the thought of a tiny fraction of that going towards additional compensation.
  - anotherhue 3y ago
    Enriching AWS does not threaten their social standing.
  - preommr 3y ago
    Something something snakes, something something unintended consequences.
    - Salgat 3y ago
      Sabotaging your company's infrastructure for a bonus is a great way to end up in prison.
      - Marrand 3y ago
        How would you prove intent to sabotage?
datadrivenangel 3y ago
If you're spending $425k per year on non-production AWS resources, you have an interesting setup.
- qaq 3y ago
  Dev env. for complicated products can be fairly involved and large companies might have a lot of them.
  - pixl97 3y ago
    If you're dev doesn't look like production, then you're not testing in reality.
- tuananh 3y ago
  have you met SAP? :)
- ponector 3y ago
  In one project we had testing setup which costs 600k USD allually. It was three times more expensive than production setup we had for product which was more than 3 year old. Nothing special, just mongo and Kafka with enormous size. If you run automation tests Manu times per day but do not clean anything - you'll get mongo with terabytes of test data. And then, on top there was elastic search which multiply bills.
hkt 3y ago
Small fry, I saved a public sector body $1m/year by doing this and rightsizing the kubernetes hosts. :D
mkl95 3y ago
Sounds like a great way to find out who's working late or is an early bird
ejs 3y ago
This is especially easy if you can shutdown environments that are only used for dev/staging tasks. With 168 hours in a week - how many hours do those things need to be running? I run a little tool for Heroku to make it easy to do this kinda thing.
- cosmotic 3y ago
  Often the continous use discounts make regularly turning on and off a wash.
  - bdcravens 3y ago
    This assumes they have something like RI etc for those resources. Those are typically used for production, but far too often, dev/test resources are usually turned on ad hoc.
latchkey 3y ago
It is interesting to note that the author works at VPBank, which is one of the larger Vietnamese banks. Saving $150k per year on an AWS bill, is really nothing to them. The fact that they even outsource their compute to AWS is kind of surprising when they could just fill up their existing data centers (like VNTT https://vntt.com.vn/ https://vntt.com.vn/) with equipment, and save a whole lot more money.
- JCharante 3y ago
  And it's also interesting that they can outsource their compute to AWS because AWS's nearest data centers are in Hong Kong & Singapore. I didn't realize a bank would allow that.
  - latchkey 3y ago
    I thought it, but I wasn't going to say it. Vietnam's internet connection is notoriously unstable. The running joke is that sharks attack the fiber connections [0] because pointing fingers is a national past time. The fact that a major bank is relying on an external AWS like that, makes it even more comical. My guess is that nobody in corporate approved this guys posting and if word got back, it would disappear quickly. Reminds me to forward this to my buddies who run Timo, which VPBank used to own, but then dropped [1]. Timo was the first forward thinking bank in Vietnam with a great tech platform, likely because it was started and run by foreigners... ¯\_(ツ)_/¯. [0] https://www.reddit.com/r/VietNam/comments/zvo553/sharks_ate_vietnams_undersea_internet_cable_again/ https://www.reddit.com/r/VietNam/comments/zvo553/sharks_ate_... [1] https://fintechnews.sg/42738/vietnam/vietnams-challenger-timo-changes-banking-partner-cuts-ties-with-vpbank/ https://fintechnews.sg/42738/vietnam/vietnams-challenger-tim...
darth_avocado 3y ago
It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were paying 300k yearly for it before discounts. It took me a literal click to shut it down. And I’m not proud of it but, had to send out an org wide email on the savings achieved (corporate politics :shrug:).
- tuananh 3y ago
  that's one of the main reason we made this feature "opt-out". if you want to keep it up, you have to tag it. once you tag it, it can opt-out 7 days. then you have to extend it (simply chat with our bot)
  - zikduruqe 3y ago
    https://cloudcustodian.io https://cloudcustodian.io Create a rule and shame them on Slack.
- dangus 3y ago
  Well, it’s definitely fathomable. Does my employer have cost control baked into their proceses, tooling, and culture or are they rushing me to get projects out the door leaving barely enough time to make sure they’re production-ready? Most places I’ve worked had no formal production readiness review before launching infrastructure.
- lnxg33k1 3y ago
  I get paid the same whether or not I spend time going around to save company cost, it’s not like they’re going to share their savings, then shareholders get their juice, management gets their juice and Im the clown who went out of my way for them, who cares I do software not cost management
  - 3y ago
EvanAnderson 3y ago
I worked adjacent some telecom consultants in the 90s whose income was solely driven by a percentage of cost savings they could trim from telephone bills. Seemed like a very brash business model but they clearly knew there was gold to be mined. I keep thinking I should be doing "cloud optimization" work and being compensated this way. Slicing and dicing output from usage/billing APIs and providing an "optimized spend" probably has the potential for a lot of low hanging fruit.
- bdcravens 3y ago
  My employer has the same business model (we audit Fedex and UPS invoices for late deliveries, bogus surcharges, etc)
- philsnow 3y ago
  Like https://www.duckbillgroup.com/about/ https://www.duckbillgroup.com/about/ ? There’s probably room in the market.
- QuinnyPig 3y ago
  As someone who's been doing this for the better part of a decade: it's a mirage. No client is going to sign for a "percentage of savings" model when it comes to cloud. Believe you me, I've looked at this up, down, and sideways; neither the math nor the psychology work out the way you'd hope.
JohnMakin 3y ago
Looking forward to the kubernetes one - Most kubernetes clusters are designed for high availability, not necessarily for being able to quickly spin up/down and there’s often a lot of hidden complexity there (at least on aws).
- tuananh 3y ago
  hint: it's going to use the same platform. allow us having the ability to inject certain manifest into any eks cluster within the org.
  - JohnMakin 3y ago
    Have you done this or attempted this yet? Every kubernetes cluster is different, but in my time working with them the last several years I anticipate the following issues: - dependent services not coming up in the order you expect/want - issues draining nodes due to crashlooping/erroring pods (can also be caused by dependent upstream services going down in wrong order) - Persistent Volume retention/synchronization - IAC not cooperating - Configuration annoyances with deployments’ availability/replica settings - Thundering herd types of problems I can think of tons of things that can make this extraordinarily difficult. I’ve had many managers over the years pitch this idea of “rapidly deployable/destructable EKS clusters” and the projects always get killed due to the complexity around this. IMHO they simply aren’t really designed for this type of thing, however, I could be misunderstanding exactly what you’re trying to do.
    - tuananh 3y ago
      > I’ve had many managers over the years pitch this idea of “rapidly deployable/destructable EKS clusters” and the projects always get killed due to the complexity around this. This is exactly what we do: blue green eks cluster. We just thought if we do it on monthly basis, DRP will be piece of cake :)
      - JohnMakin 3y ago
        Look forward to the writeup! thanks
tehlike 3y ago
Cutting down AWS cost by 90% by simply moving to hetzner.
- bdcravens 3y ago
  Was doing some research this weekend on cloud exit. Hetzner is attractive, but our company is pretty much limited to the US (no international companies due to our current business model). How practical would it be? Also, I've seen a lot of concern over blocked IPs, especially for lower-cost hosts. Is that an issue with Hetzner?
  - tehlike 3y ago
    Hetzner cloud supports us-west and us-east.
    - bdcravens 3y ago
      I was looking at dedicated hardware. If I decide to stay in anyone's cloud, I'd stick with AWS.
      - tehlike 3y ago
        Yeah Hetzner doesn't have dedicated in the US, and not sure if it will in near/mid future. Still, Hetzner cloud is pretty good option, and there's more support coming on building on Hetzner. https://www.ubicloud.com/ https://www.ubicloud.com/ (from founders of citus) is mainly/currently targeting hetzner, for example.
      - nvm0n2 3y ago
        There's alternative companies with similar offers, like Deft: https://deft.com/dedicated-servers/ https://deft.com/dedicated-servers/
        tehlike 3y ago
        I imagine one reason I didn't hear about them before is that they don't seem to offer self service. I absolutely don't want to talk to anyone when I am buying object storage, cloud vms, or even dedicated.
brycewray 3y ago
In a similar vein: https://usefathom.com/blog/reduce-aws-bill https://usefathom.com/blog/reduce-aws-bill
danfritz 3y ago
Cut costs down to 90% by going serverless and run everything on lambdas? Nothing keeps humming if it's not being used
- nikita 3y ago
  (CEO of Neon) We routinely see 10x savings when switching from RDS or Aurora. Especially if you start adding dev environments.
- bdcravens 3y ago
  Sure, after you reengineer your application. Even then, "serverless" apps often use persistent resources like databases, and your developers will likely spin up those resources for the same reasons as indicated in the article.
- bob1029 3y ago
  Cost savings can be incredible if you use the FaaS product in the most aggressive way possible. For us, this means using functions as a simple translation layer between SSR web forms served directly as text/html and whatever SQL provider (ideally also on a consumption-based tier). 90% sounds just about right. We are seeing figures going from $120/m for a VM-based QA environment to $10/m for a consumption-based / serverless stack.
- hnav 3y ago
  depends on what your utilization looks like, serverless is usually +/- an order of magnitude more expensive. Ideally your workloads are stateless and containerized so you can shuffle them between serverless, container orchestration that you own and dedicated VMs.
gnarlouse 3y ago
“Have you tried turning it off and then turning it back on again?”
TrianguloY 3y ago
> You go home. So you shut things down. Sorry for the rant, but this is usually wrong. The amount of people that just keeps their computer on is noticeable. And when I ask it's usually "just to avoid having to wait" or "I've always done that". I personally always hibernate my computer. When I turn it off it takes more time, but I'm already on the other side of the building so I don't care. When I turn it on it takes basically the same amount of time, and it is exactly as I left it. People keep the computer on just because convenience...and I don't think it's a good thing.
- saylisteins 3y ago
  I always leave my PC on, but for different reasons: - I have a plex server running on it - I can remote into it from my phone, this comes in handy a lot of the time. - I can remote into it when traveling through my Fire stick using parsec, which means I don't have to carry a laptop with me everywhere I go ( I also setup my phone so I can use it as keyboard/mouse when I do this). Regarding energy costs, it's negligible for the benefits it gives me
- SoftTalker 3y ago
  I keep mine on because it's my jumphost for working remotely. But I agree many people don't need to do this. My company, though, wants people to leave their PCs on so they can get automatic updates and be centrally managed.
- charlie0 3y ago
  I always shutdown my machine at night and sometimes restart when I leave for lunch. I've noticed running Docker and other apps for a while makes my machine slower. I'm convinced there's a memory leak somewhere and restarting away fixes those issues.
  - Too 3y ago
    Restart the computer on a daily basis? Like it’s 1998? You need a different system man. Ubuntu can go strong for a month easily, with only sleep for leaving it unused. Only reason for restarts is security updates or that your battery ran totally dry.
kevin_nisbet 3y ago
I've for a long time set my cloud VMs to shutdown on idle. I usually use it to also justify running a much larger VM to cut down on build and test times. Just set a cron to run the shutdown command with a grace period. And then if you're working late, you just cancel the shutdown and the shutdown will be retried in a couple of hours. And have a script or command to just run the cloud API calls to boot the VM in the morning / when needed, and the environment boots in a minute or two. For other stuff I've been tempted to do a more complicated setup, with something like a micro-vm as a proxy, that will do the shutdown / activation on TCP connection, but haven't gotten around to it.
from-nibly 3y ago
Reacting to events to install security defaults (or any kind of defaults) sounds really error prone. Are people running AWS where devs just click buttons in aws and spin up random stuff? I thought we all decided that was dumb and switched to gitops/terraform? Did I miss a new trend or something?
joshstrange 3y ago
One small issue I have as a developer who can spin up just about anything on AWS is this: I have zero insight into the costs. Yes, my company could turn that on for me but it's rare that they do so it's nearly impossible to know if I did something that costs a lot of money (relatively or in general) without access to the cost explorer/billing dashboard. And before "well can look up what a t2.2xlarge costs and calculate it", sure. In a very contrived example I might be able to see what it costs but so many things are hidden/hard to see in AWS. For example, I recently spun up an RDS customer on my own AWS account. After testing for a while I decided it wasn't what I wanted and I deleted the cluster. Fast forward a month and my bill is well over what I expected (Like $30, no it's not a ton of money but it's my personal account and I wasn't expecting that charge). Come to find out it created a VPC as part of the RDS cluster (I think maybe it was for the RDS proxy? Still not sure) that didn't get deleted. I had to go chase that down and even that process wasn't easy. I had to make sure that it wasn't be used by anything else and then delete other things that were created when I made the RDS cluster before I could remove the VPC. I was only able to do the above because I had access to the billing info. I would have left that VPC indefinitely on my work's AWS account by accident and been none the wiser. I'm more than happy to take costs into account but without access to what things are actually costing us I can't help that much. Mostly because I need to know the costs to know what's worth optimizing. Sure I know I could improve X feature but if that costs us pennies a day (or month sometimes) then it's not worth it. Similarly if I know feature/infra Y is costing $XX,000/mo then I know I should rethink or investigate if that's correct/worth it.
- tuananh 3y ago
  billing transparency is very important. in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill
  - joshstrange 3y ago
    Exactly, sometimes it's not clear at all what something will cost (and/or if the costs will go up). I'm happy to glance at the monthly costs here and there and if I see a jump I can dive in and see where it's coming from. We all make silly mistakes, like leaving logs on infinite retention in CloudWatch, and that's something I can easily fix/address but only if I have the info. I've asked, off-hand, a couple times for billing access but nothing has come of it. I don't want to seem pushy but also it feels like data I need to perform my job to the best of my ability (especially at a small company). I don't think it comes from a place of "We don't want to give Josh access" or secrecy as much as it not being a priority but I need to bring it up again.
gurchik 3y ago
I recently helped save $150k per year by deleting node_modules. I noticed that one of our S3 buckets had high data transfer costs, a bucket that our app downloads HTML+JS assets from when we push out a new release. I downloaded the "directory" of files for our latest release and saw it was mostly node_modules. I checked the code and confirmed that, yes, if this file exists in the bucket then it'll be downloaded by the user. I wrote a quick Python script to list out each directory that had this problem, and a quick Slack message to the appropriate team later, we discovered the specific commit that was the cause, a change to our CI that inadvertently uploaded that directory when we wanted to ignore it. A few months later, I checked the billing metrics, the effect was an avg of $12,500 reduction in cost for this bucket, or around $150k per year, or 4% of our bill. Not bad for one hour of work. Over the course of a quarter I reduced our bill by over $1m, or around 30% of our bill. I might write a blog post explaining how to go about something like that. A lot of people are not familiar with tools like Trusted Advisor which can easily tell you if you have, for example, unused EC2 instances that can be terminated.
- tuananh 3y ago
  please do :) i would love to learn more about this
- Detrytus 3y ago
  Now, the inconvenient question: how much of that $1m savings ended up in your bank account as a bonus? Because certainly some of it should :-D
  - gurchik 3y ago
    Not sure yet, but probably nothing. I completely understand the expectations written in this thread to receive something in return, but I've given this thought and I'm not sure how to do this in a fair way in this situation. First, I was given dedicated time that quarter to work on cost savings and other people weren't, if I received a bonus is that fair to other people who didn't have the same opportunity? Not to mention the possibility of people abusing this process. I would be happy to receive some extra cash, don't get me wrong, but I work for non-monetary benefits as well, and I have received some of those as part of this work. If I worked at a company with a different culture and I was being punished for doing the work, I would demand some bonus.
ary 3y ago
> The best optimization is simply shutting things off This is the way. A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following: - Execute via Lambda (serverless). - Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags. - Enable resources to be brought back up out of schedule when demand dictates. - Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced. Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it. So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model. [1] https://cloudcustodian.io/ https://cloudcustodian.io/
- pid-1 3y ago
  > Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced. My team mostly builds internal stuff and we save tons of $$$ by using Knative + Karpenter, which basically does that on container + EC2 levels.
- akira2501 3y ago
  Everything I've built in AWS is strictly serverless. You can do an incredible amount with a clever DynamoDB pay-per-request setup, S3 and CloudFront. I haven't once felt the need to reach out to EC2 or RDS and I can't imagine building any sort of control plane to spool them up and down for me.
nathanwallace 3y ago
Readers may find Steampipe's [1] AWS Thrifty Mod [2] useful. It will automatically scan multiple accounts and regions for 50 cost saving opportunities - many of which are looking for over-provisioned or unused resources. For example, it's crazy how much you can save by doing things like just converting your EBS volumes to the newer gp3 type. Combine with Flowpipe [3] to automate checks and actions. It's all open source and extensible. 1 - https://github.com/turbot/steampipe https://github.com/turbot/steampipe 2 - https://github.com/turbot/steampipe-mod-aws-thrifty https://github.com/turbot/steampipe-mod-aws-thrifty 3 - https://github.com/turbot/flowpipe https://github.com/turbot/flowpipe
cddotdotslash 3y ago
I’m convinced that once a company reaches ~$10m/year in AWS spend it becomes entirely reasonable to hire an in-house engineer whose sole job is to find cost savings opportunities. Literally a “find unused stuff and turn it off” engineer.
- 8organicbits 3y ago
  I've spent some time doing this. There's always old systems people don't really understand, ownership is poorly defined, and no one knows what happens if you turn it off. It's archeology. Understand what the system is doing and how it interacts with other systems and the business. If it looks unneeded back it up, stop the VM, wait and watch for fallout, and eventually terminate it.
  - cddotdotslash 3y ago
    There’s definitely a science to it. To complicate matters, the way you explore those connections, take backups, identify owners, and perform restores is different across pretty much every cloud service.
StratusBen 3y ago
Disclosure: I'm CEO of https://www.vantage.sh/ https://www.vantage.sh/ -- a cloud cost observability platform. I previously worked at both AWS and DigitalOcean. For people looking for how to save money on AWS - I'd [selfishly] recommend connecting up to Vantage. We profile AWS for all sorts of savings and give you the information on how much we can save prior to you paying us. It can be a good gut-check if nothing else on how well optimized you are.
- pizzafeelsright 3y ago
  Silly request. The amount of paperwork, lawyers, and time required to connect our service makes it impossible to validate savings. Is there an offline method? I have not looked at vantage to see if it's possible.
  - StratusBen 3y ago
    Unfortunately we don't have an option for that route -- but we're happy to help support any paperwork for getting things up and running if you contact me or my team: ben [at] vantage [dot] sh.
ralfcheung 3y ago
meanwhile, I'm saving the company 7-digit/year from Google.
throwitaway222 3y ago
It looks like this is surfacing because a lot of companies are laying off but also finding every possible way to save on costs right now.
8organicbits 3y ago
You should always calculate if you're actually going to see cost savings. Counterintuitively, running for fewer hours can increase your bill if it causes you to switch to on-demand pricing [1]. There's a break even point you need to get past. [1] https://alexsci.com/blog/modeling-on-demand-pricing/ https://alexsci.com/blog/modeling-on-demand-pricing/
nodesocket 3y ago
I’ve mentioned this before, but probably one of the most egregious costs on AWS are NAT gateways and NAT bandwidth pricing. Typically I deploy one NAT gateway per AZ so looking at $99 a month just for three NAT gateway instances with zero traffic.
partiallypro 3y ago
I saved my previous company $4000-5000/mo on AWS billing just by auditing the AWS account and turning off unused machines that that old devs has spun up and deleting hard drives after backing them up to S3, just in case. No one had really even bothered looking at it for years and I did it in my "free time" at work without being tasked with it. Ironically, I asked for a raise a year later and was denied, despite single handedly saving the company nearly $50000/yr. The raise I asked for wasn't close the cost savings I had brought. I left the company shortly after. I saw someone else have a similar experience here and a comment to it was saying rewarding this produces a bad incentive...well, honestly why would I have even bothered cutting costs if I felt I wouldn't be rewarded? Not rewarding it just makes me half regret doing it at all.
octopoc 3y ago
Here's a startup idea: a profiler for infrastructure-as-code that shows how much each line of code is costing per month, instead of showing where the CPU spends most of its time.
- msmith 3y ago
  https://www.infracost.io/ https://www.infracost.io/ might do what you're imagining
- stusmall 3y ago
  Have you looked at OpenCost? It's more k8s focused but of a similar idea
davidgerard 3y ago
We do this. It was a bit faffy to set up, but having dev and staging shut down at night and only be started as needed that day saved us a fortune.
paxys 3y ago
I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the savings for myself as a reward, the answer would be a very clear "no" to both. So overall, as an end developer I really have no incentive to work harder and ensure lower operating costs for my company, and I'm sure most developers in the industry are in the same position as me.
- jfim 3y ago
  Part of it is that it creates an incentive to create wasteful systems, only to "optimize" them later to rack up a bonus. Even if it gets changed to only pay out for reducing spend incurred by other engineers, it's possible to collude in such a way to extract bonuses from the company. A better way to have aligned incentives for the company and the employees would be to allocate a bonus pool for the entire company, from which AWS expenses are taken out of, but that might be a bit unorthodox.
  - avidiax 3y ago
    > allocate a bonus pool for the entire company, from which AWS expenses are taken out of Also a perverse incentive. If we use ec2.small, the customer's query will take 3x longer but be half the price. Let's turn off the nightly security audits. We can live with quarterly backups, right? What do we need all these logs for, anyway? We could hack something that works together in 2 weeks, but if we spend 3 months, it could be really efficient, let's do that...
- cduzz 3y ago
  How are they going to be sure you're not just farming cobras[1]? [1]https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive
- nick_ 3y ago
  This is one of many insights that hint at why biz-facing cloud architecture is so popular, wasteful, and profitable. The incentives are designed to form an enormous cash siphon. From aggressively marketing toward fearful & liable (or maybe just tech-cost-illiterate) upper-management to the silencing effect that the low-rung experts experience when sounding the alarm.
TheIronMark 3y ago
I did this at a previous employer. I leveraged a Lambda functions and tags applied to instances to determine when they should be and when they should be off.
bornfreddy 3y ago
Does anyone have a good experience with tools / services that track and analyze cloud usage? We don't use any, but could benefit from better visibility in spending patterns.
whummer 3y ago
To give this a slightly different spin: --> "The best optimization is simply not spinning things up!" At least for local development and testing, as made possible by LocalStack (https://localstack.cloud https://localstack.cloud), among other local testing solutions and emulators. We've seen so many teams fall into the trap of "someone forgot to shut down dev resource X for a week and now we've racked up a $$$ bill on AWS". What is everyone's strategy to avoid this kind of situation? Tools like `aws-nuke` (https://github.com/rebuy-de/aws-nuke https://github.com/rebuy-de/aws-nuke) are awesome (!) to clean up unused resources, but frankly they should not be necessary in the first place...
kthejoker2 3y ago
When I joined my current company I quickly found our internal Azure environment was effectively unmanaged. Four weeks in I shaved nearly 55k in monthly spend just scripting out VM shutdowns and service pauses. Cloud revenue in most large companies is at least 25$, maybe up to 40%, pure developer waste because nobody upstairs knows the difference.

191 points by tuananh 3y ago

anuptalwalkar 3y ago

Couple of years ago I saved about 14 mn of revenue per year for my company. I got a 250$ bonus for it.

schnebbau 3y ago
This is why you should start the conversation with "I have drawn up a plan to save the company $14M per year. I will execute this plan in exchange for $7M upon completion." If they say no then just go back to your regular duties.
- munchler 3y ago
  Because blackmail is an effective salary negotiation tactic?
- listenallyall 3y ago
  Very few companies would make this deal at 0.7M, 0.07M, or even 0.007M. Direct sharing a % of savings with the responsible employee is simply not the way most companies work. Consultants, now, that's different...
  - charlie0 3y ago
    For a slice of 7M, just quit and come back as consultant.
llanowarelves 3y ago
Learned a lesson that you only have to "spend" (forgo) $250 to cause that company $14m in losses (that you could have prevented)
nvm0n2 3y ago
But was that your job? Because if so, you really got salary+bonus. And if you'd found nothing you'd still have got salary. So you can look at it several ways.

zikduruqe 3y ago

Shit... I reduced our spend on one AWS account from $270K a month to $75K a month.

ponector 3y ago
Great! Shareholders will be happy. Did you receive any bonus?
- marvin 3y ago
  It's a good shoo-in for becoming a freelance consultant charging a percentage of annual savings rather than hourly time billed.
- Chico75 3y ago
  The problem is always around abuse. If it becomes known that you can get a big bonus by wasting a lot of money on useless infra first and then reducing it, other people will start playing the game. How do you reward cloud cost awareness without creating perverse incentives?
  - nvm0n2 3y ago
    It's always the same answer: managers who pay attention to the details. People familiar with your work should be able to tell if you're gaming the system or not.
- zen928 3y ago
  Will they also be paying what they owe on the added costs that could have been noticed earlier with due diligence? I'm imagining that what they'll receive instead is the compensation expected and agreed upon by both parties negotiated during either initial hiring or the multiple points in the year that allow for easy communication about changing payroll expectations, instead of hawking for dimes at first sight.
- zikduruqe 3y ago
  Hahaha, for doing my job? I wish.

sremani 3y ago

Cloud optimization is the next Kubernetes.

hkt 3y ago
Goodness I hope so, that'll be easy money
- bdcravens 3y ago
  <quinnypig has entered the chat>
  - QuinnyPig 3y ago
    I HAVE BEEN SUMMONED

thefourthchime 3y ago

I've mentioned this before, in my company (big media company) I saw some S3 costs creeping up each month. I looked into it and it was a system we abandoned that was still copying files to this bucket. I reached out to the team and they turned it off, it saved us $1m a year. The higher-ups rewarded me by telling me that a team should have caught this so I should meet with them now.

tuananh 3y ago
true. the first step is to have visibility into what's eating the bill. it's just like you need to profile the program before optimizing it. that's why we did finops dashboard first thing when we first started the cloud journey. can't optimize if you dont know.
Salgat 3y ago
It's truly fascinating how companies won't bat at an eye at spending ungodly amounts of money on things they don't need, but will sweat profusely at the thought of a tiny fraction of that going towards additional compensation.
- anotherhue 3y ago
  Enriching AWS does not threaten their social standing.
- preommr 3y ago
  Something something snakes, something something unintended consequences.
  - Salgat 3y ago
    Sabotaging your company's infrastructure for a bonus is a great way to end up in prison.
    - Marrand 3y ago
      How would you prove intent to sabotage?

datadrivenangel 3y ago

If you're spending $425k per year on non-production AWS resources, you have an interesting setup.

qaq 3y ago
Dev env. for complicated products can be fairly involved and large companies might have a lot of them.
- pixl97 3y ago
  If you're dev doesn't look like production, then you're not testing in reality.
tuananh 3y ago
have you met SAP? :)
ponector 3y ago
In one project we had testing setup which costs 600k USD allually. It was three times more expensive than production setup we had for product which was more than 3 year old. Nothing special, just mongo and Kafka with enormous size. If you run automation tests Manu times per day but do not clean anything - you'll get mongo with terabytes of test data. And then, on top there was elastic search which multiply bills.

hkt 3y ago

Small fry, I saved a public sector body $1m/year by doing this and rightsizing the kubernetes hosts. :D

mkl95 3y ago

Sounds like a great way to find out who's working late or is an early bird

ejs 3y ago

This is especially easy if you can shutdown environments that are only used for dev/staging tasks. With 168 hours in a week - how many hours do those things need to be running? I run a little tool for Heroku to make it easy to do this kinda thing.

cosmotic 3y ago
Often the continous use discounts make regularly turning on and off a wash.
- bdcravens 3y ago
  This assumes they have something like RI etc for those resources. Those are typically used for production, but far too often, dev/test resources are usually turned on ad hoc.

latchkey 3y ago

It is interesting to note that the author works at VPBank, which is one of the larger Vietnamese banks. Saving $150k per year on an AWS bill, is really nothing to them. The fact that they even outsource their compute to AWS is kind of surprising when they could just fill up their existing data centers (like VNTT https://vntt.com.vn/ https://vntt.com.vn/) with equipment, and save a whole lot more money.

JCharante 3y ago
And it's also interesting that they can outsource their compute to AWS because AWS's nearest data centers are in Hong Kong & Singapore. I didn't realize a bank would allow that.
- latchkey 3y ago
  I thought it, but I wasn't going to say it. Vietnam's internet connection is notoriously unstable. The running joke is that sharks attack the fiber connections [0] because pointing fingers is a national past time. The fact that a major bank is relying on an external AWS like that, makes it even more comical. My guess is that nobody in corporate approved this guys posting and if word got back, it would disappear quickly. Reminds me to forward this to my buddies who run Timo, which VPBank used to own, but then dropped [1]. Timo was the first forward thinking bank in Vietnam with a great tech platform, likely because it was started and run by foreigners... ¯\_(ツ)_/¯. [0] https://www.reddit.com/r/VietNam/comments/zvo553/sharks_ate_vietnams_undersea_internet_cable_again/ https://www.reddit.com/r/VietNam/comments/zvo553/sharks_ate_... [1] https://fintechnews.sg/42738/vietnam/vietnams-challenger-timo-changes-banking-partner-cuts-ties-with-vpbank/ https://fintechnews.sg/42738/vietnam/vietnams-challenger-tim...

darth_avocado 3y ago

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were paying 300k yearly for it before discounts. It took me a literal click to shut it down. And I’m not proud of it but, had to send out an org wide email on the savings achieved (corporate politics :shrug:).

tuananh 3y ago
that's one of the main reason we made this feature "opt-out". if you want to keep it up, you have to tag it. once you tag it, it can opt-out 7 days. then you have to extend it (simply chat with our bot)
- zikduruqe 3y ago
  https://cloudcustodian.io https://cloudcustodian.io Create a rule and shame them on Slack.
dangus 3y ago
Well, it’s definitely fathomable. Does my employer have cost control baked into their proceses, tooling, and culture or are they rushing me to get projects out the door leaving barely enough time to make sure they’re production-ready? Most places I’ve worked had no formal production readiness review before launching infrastructure.
lnxg33k1 3y ago
I get paid the same whether or not I spend time going around to save company cost, it’s not like they’re going to share their savings, then shareholders get their juice, management gets their juice and Im the clown who went out of my way for them, who cares I do software not cost management
- 3y ago

EvanAnderson 3y ago

I worked adjacent some telecom consultants in the 90s whose income was solely driven by a percentage of cost savings they could trim from telephone bills. Seemed like a very brash business model but they clearly knew there was gold to be mined. I keep thinking I should be doing "cloud optimization" work and being compensated this way. Slicing and dicing output from usage/billing APIs and providing an "optimized spend" probably has the potential for a lot of low hanging fruit.

bdcravens 3y ago
My employer has the same business model (we audit Fedex and UPS invoices for late deliveries, bogus surcharges, etc)
philsnow 3y ago
Like https://www.duckbillgroup.com/about/ https://www.duckbillgroup.com/about/ ? There’s probably room in the market.
QuinnyPig 3y ago
As someone who's been doing this for the better part of a decade: it's a mirage. No client is going to sign for a "percentage of savings" model when it comes to cloud. Believe you me, I've looked at this up, down, and sideways; neither the math nor the psychology work out the way you'd hope.

JohnMakin 3y ago

Looking forward to the kubernetes one - Most kubernetes clusters are designed for high availability, not necessarily for being able to quickly spin up/down and there’s often a lot of hidden complexity there (at least on aws).

tuananh 3y ago
hint: it's going to use the same platform. allow us having the ability to inject certain manifest into any eks cluster within the org.
- JohnMakin 3y ago
  Have you done this or attempted this yet? Every kubernetes cluster is different, but in my time working with them the last several years I anticipate the following issues: - dependent services not coming up in the order you expect/want - issues draining nodes due to crashlooping/erroring pods (can also be caused by dependent upstream services going down in wrong order) - Persistent Volume retention/synchronization - IAC not cooperating - Configuration annoyances with deployments’ availability/replica settings - Thundering herd types of problems I can think of tons of things that can make this extraordinarily difficult. I’ve had many managers over the years pitch this idea of “rapidly deployable/destructable EKS clusters” and the projects always get killed due to the complexity around this. IMHO they simply aren’t really designed for this type of thing, however, I could be misunderstanding exactly what you’re trying to do.
  - tuananh 3y ago
    > I’ve had many managers over the years pitch this idea of “rapidly deployable/destructable EKS clusters” and the projects always get killed due to the complexity around this. This is exactly what we do: blue green eks cluster. We just thought if we do it on monthly basis, DRP will be piece of cake :)
    - JohnMakin 3y ago
      Look forward to the writeup! thanks

tehlike 3y ago

Cutting down AWS cost by 90% by simply moving to hetzner.

bdcravens 3y ago
Was doing some research this weekend on cloud exit. Hetzner is attractive, but our company is pretty much limited to the US (no international companies due to our current business model). How practical would it be? Also, I've seen a lot of concern over blocked IPs, especially for lower-cost hosts. Is that an issue with Hetzner?
- tehlike 3y ago
  Hetzner cloud supports us-west and us-east.
  - bdcravens 3y ago
    I was looking at dedicated hardware. If I decide to stay in anyone's cloud, I'd stick with AWS.
    - tehlike 3y ago
      Yeah Hetzner doesn't have dedicated in the US, and not sure if it will in near/mid future. Still, Hetzner cloud is pretty good option, and there's more support coming on building on Hetzner. https://www.ubicloud.com/ https://www.ubicloud.com/ (from founders of citus) is mainly/currently targeting hetzner, for example.
    - nvm0n2 3y ago
      There's alternative companies with similar offers, like Deft: https://deft.com/dedicated-servers/ https://deft.com/dedicated-servers/
      - tehlike 3y ago
        I imagine one reason I didn't hear about them before is that they don't seem to offer self service. I absolutely don't want to talk to anyone when I am buying object storage, cloud vms, or even dedicated.

brycewray 3y ago

In a similar vein: https://usefathom.com/blog/reduce-aws-bill https://usefathom.com/blog/reduce-aws-bill

danfritz 3y ago

Cut costs down to 90% by going serverless and run everything on lambdas? Nothing keeps humming if it's not being used

nikita 3y ago
(CEO of Neon) We routinely see 10x savings when switching from RDS or Aurora. Especially if you start adding dev environments.
bdcravens 3y ago
Sure, after you reengineer your application. Even then, "serverless" apps often use persistent resources like databases, and your developers will likely spin up those resources for the same reasons as indicated in the article.
bob1029 3y ago
Cost savings can be incredible if you use the FaaS product in the most aggressive way possible. For us, this means using functions as a simple translation layer between SSR web forms served directly as text/html and whatever SQL provider (ideally also on a consumption-based tier). 90% sounds just about right. We are seeing figures going from $120/m for a VM-based QA environment to $10/m for a consumption-based / serverless stack.
hnav 3y ago
depends on what your utilization looks like, serverless is usually +/- an order of magnitude more expensive. Ideally your workloads are stateless and containerized so you can shuffle them between serverless, container orchestration that you own and dedicated VMs.

gnarlouse 3y ago

“Have you tried turning it off and then turning it back on again?”

TrianguloY 3y ago

> You go home. So you shut things down. Sorry for the rant, but this is usually wrong. The amount of people that just keeps their computer on is noticeable. And when I ask it's usually "just to avoid having to wait" or "I've always done that". I personally always hibernate my computer. When I turn it off it takes more time, but I'm already on the other side of the building so I don't care. When I turn it on it takes basically the same amount of time, and it is exactly as I left it. People keep the computer on just because convenience...and I don't think it's a good thing.

saylisteins 3y ago
I always leave my PC on, but for different reasons: - I have a plex server running on it - I can remote into it from my phone, this comes in handy a lot of the time. - I can remote into it when traveling through my Fire stick using parsec, which means I don't have to carry a laptop with me everywhere I go ( I also setup my phone so I can use it as keyboard/mouse when I do this). Regarding energy costs, it's negligible for the benefits it gives me
SoftTalker 3y ago
I keep mine on because it's my jumphost for working remotely. But I agree many people don't need to do this. My company, though, wants people to leave their PCs on so they can get automatic updates and be centrally managed.
charlie0 3y ago
I always shutdown my machine at night and sometimes restart when I leave for lunch. I've noticed running Docker and other apps for a while makes my machine slower. I'm convinced there's a memory leak somewhere and restarting away fixes those issues.
- Too 3y ago
  Restart the computer on a daily basis? Like it’s 1998? You need a different system man. Ubuntu can go strong for a month easily, with only sleep for leaving it unused. Only reason for restarts is security updates or that your battery ran totally dry.

kevin_nisbet 3y ago

I've for a long time set my cloud VMs to shutdown on idle. I usually use it to also justify running a much larger VM to cut down on build and test times. Just set a cron to run the shutdown command with a grace period. And then if you're working late, you just cancel the shutdown and the shutdown will be retried in a couple of hours. And have a script or command to just run the cloud API calls to boot the VM in the morning / when needed, and the environment boots in a minute or two. For other stuff I've been tempted to do a more complicated setup, with something like a micro-vm as a proxy, that will do the shutdown / activation on TCP connection, but haven't gotten around to it.

from-nibly 3y ago

Reacting to events to install security defaults (or any kind of defaults) sounds really error prone. Are people running AWS where devs just click buttons in aws and spin up random stuff? I thought we all decided that was dumb and switched to gitops/terraform? Did I miss a new trend or something?

joshstrange 3y ago

One small issue I have as a developer who can spin up just about anything on AWS is this: I have zero insight into the costs. Yes, my company could turn that on for me but it's rare that they do so it's nearly impossible to know if I did something that costs a lot of money (relatively or in general) without access to the cost explorer/billing dashboard. And before "well can look up what a t2.2xlarge costs and calculate it", sure. In a very contrived example I might be able to see what it costs but so many things are hidden/hard to see in AWS. For example, I recently spun up an RDS customer on my own AWS account. After testing for a while I decided it wasn't what I wanted and I deleted the cluster. Fast forward a month and my bill is well over what I expected (Like $30, no it's not a ton of money but it's my personal account and I wasn't expecting that charge). Come to find out it created a VPC as part of the RDS cluster (I think maybe it was for the RDS proxy? Still not sure) that didn't get deleted. I had to go chase that down and even that process wasn't easy. I had to make sure that it wasn't be used by anything else and then delete other things that were created when I made the RDS cluster before I could remove the VPC. I was only able to do the above because I had access to the billing info. I would have left that VPC indefinitely on my work's AWS account by accident and been none the wiser. I'm more than happy to take costs into account but without access to what things are actually costing us I can't help that much. Mostly because I need to know the costs to know what's worth optimizing. Sure I know I could improve X feature but if that costs us pennies a day (or month sometimes) then it's not worth it. Similarly if I know feature/infra Y is costing $XX,000/mo then I know I should rethink or investigate if that's correct/worth it.

tuananh 3y ago
billing transparency is very important. in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill
- joshstrange 3y ago
  Exactly, sometimes it's not clear at all what something will cost (and/or if the costs will go up). I'm happy to glance at the monthly costs here and there and if I see a jump I can dive in and see where it's coming from. We all make silly mistakes, like leaving logs on infinite retention in CloudWatch, and that's something I can easily fix/address but only if I have the info. I've asked, off-hand, a couple times for billing access but nothing has come of it. I don't want to seem pushy but also it feels like data I need to perform my job to the best of my ability (especially at a small company). I don't think it comes from a place of "We don't want to give Josh access" or secrecy as much as it not being a priority but I need to bring it up again.

gurchik 3y ago

I recently helped save $150k per year by deleting node_modules. I noticed that one of our S3 buckets had high data transfer costs, a bucket that our app downloads HTML+JS assets from when we push out a new release. I downloaded the "directory" of files for our latest release and saw it was mostly node_modules. I checked the code and confirmed that, yes, if this file exists in the bucket then it'll be downloaded by the user. I wrote a quick Python script to list out each directory that had this problem, and a quick Slack message to the appropriate team later, we discovered the specific commit that was the cause, a change to our CI that inadvertently uploaded that directory when we wanted to ignore it. A few months later, I checked the billing metrics, the effect was an avg of $12,500 reduction in cost for this bucket, or around $150k per year, or 4% of our bill. Not bad for one hour of work. Over the course of a quarter I reduced our bill by over $1m, or around 30% of our bill. I might write a blog post explaining how to go about something like that. A lot of people are not familiar with tools like Trusted Advisor which can easily tell you if you have, for example, unused EC2 instances that can be terminated.

tuananh 3y ago
please do :) i would love to learn more about this
Detrytus 3y ago
Now, the inconvenient question: how much of that $1m savings ended up in your bank account as a bonus? Because certainly some of it should :-D
- gurchik 3y ago
  Not sure yet, but probably nothing. I completely understand the expectations written in this thread to receive something in return, but I've given this thought and I'm not sure how to do this in a fair way in this situation. First, I was given dedicated time that quarter to work on cost savings and other people weren't, if I received a bonus is that fair to other people who didn't have the same opportunity? Not to mention the possibility of people abusing this process. I would be happy to receive some extra cash, don't get me wrong, but I work for non-monetary benefits as well, and I have received some of those as part of this work. If I worked at a company with a different culture and I was being punished for doing the work, I would demand some bonus.

ary 3y ago

> The best optimization is simply shutting things off This is the way. A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following: - Execute via Lambda (serverless). - Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags. - Enable resources to be brought back up out of schedule when demand dictates. - Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced. Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it. So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model. [1] https://cloudcustodian.io/ https://cloudcustodian.io/

pid-1 3y ago
> Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced. My team mostly builds internal stuff and we save tons of $$$ by using Knative + Karpenter, which basically does that on container + EC2 levels.
akira2501 3y ago
Everything I've built in AWS is strictly serverless. You can do an incredible amount with a clever DynamoDB pay-per-request setup, S3 and CloudFront. I haven't once felt the need to reach out to EC2 or RDS and I can't imagine building any sort of control plane to spool them up and down for me.

nathanwallace 3y ago

Readers may find Steampipe's [1] AWS Thrifty Mod [2] useful. It will automatically scan multiple accounts and regions for 50 cost saving opportunities - many of which are looking for over-provisioned or unused resources. For example, it's crazy how much you can save by doing things like just converting your EBS volumes to the newer gp3 type. Combine with Flowpipe [3] to automate checks and actions. It's all open source and extensible. 1 - https://github.com/turbot/steampipe https://github.com/turbot/steampipe 2 - https://github.com/turbot/steampipe-mod-aws-thrifty https://github.com/turbot/steampipe-mod-aws-thrifty 3 - https://github.com/turbot/flowpipe https://github.com/turbot/flowpipe

cddotdotslash 3y ago

I’m convinced that once a company reaches ~$10m/year in AWS spend it becomes entirely reasonable to hire an in-house engineer whose sole job is to find cost savings opportunities. Literally a “find unused stuff and turn it off” engineer.

8organicbits 3y ago
I've spent some time doing this. There's always old systems people don't really understand, ownership is poorly defined, and no one knows what happens if you turn it off. It's archeology. Understand what the system is doing and how it interacts with other systems and the business. If it looks unneeded back it up, stop the VM, wait and watch for fallout, and eventually terminate it.
- cddotdotslash 3y ago
  There’s definitely a science to it. To complicate matters, the way you explore those connections, take backups, identify owners, and perform restores is different across pretty much every cloud service.

StratusBen 3y ago

Disclosure: I'm CEO of https://www.vantage.sh/ https://www.vantage.sh/ -- a cloud cost observability platform. I previously worked at both AWS and DigitalOcean. For people looking for how to save money on AWS - I'd [selfishly] recommend connecting up to Vantage. We profile AWS for all sorts of savings and give you the information on how much we can save prior to you paying us. It can be a good gut-check if nothing else on how well optimized you are.

pizzafeelsright 3y ago
Silly request. The amount of paperwork, lawyers, and time required to connect our service makes it impossible to validate savings. Is there an offline method? I have not looked at vantage to see if it's possible.
- StratusBen 3y ago
  Unfortunately we don't have an option for that route -- but we're happy to help support any paperwork for getting things up and running if you contact me or my team: ben [at] vantage [dot] sh.

ralfcheung 3y ago

meanwhile, I'm saving the company 7-digit/year from Google.

throwitaway222 3y ago

It looks like this is surfacing because a lot of companies are laying off but also finding every possible way to save on costs right now.

8organicbits 3y ago

You should always calculate if you're actually going to see cost savings. Counterintuitively, running for fewer hours can increase your bill if it causes you to switch to on-demand pricing [1]. There's a break even point you need to get past. [1] https://alexsci.com/blog/modeling-on-demand-pricing/ https://alexsci.com/blog/modeling-on-demand-pricing/

nodesocket 3y ago

I’ve mentioned this before, but probably one of the most egregious costs on AWS are NAT gateways and NAT bandwidth pricing. Typically I deploy one NAT gateway per AZ so looking at $99 a month just for three NAT gateway instances with zero traffic.

partiallypro 3y ago

I saved my previous company $4000-5000/mo on AWS billing just by auditing the AWS account and turning off unused machines that that old devs has spun up and deleting hard drives after backing them up to S3, just in case. No one had really even bothered looking at it for years and I did it in my "free time" at work without being tasked with it. Ironically, I asked for a raise a year later and was denied, despite single handedly saving the company nearly $50000/yr. The raise I asked for wasn't close the cost savings I had brought. I left the company shortly after. I saw someone else have a similar experience here and a comment to it was saying rewarding this produces a bad incentive...well, honestly why would I have even bothered cutting costs if I felt I wouldn't be rewarded? Not rewarding it just makes me half regret doing it at all.

octopoc 3y ago

Here's a startup idea: a profiler for infrastructure-as-code that shows how much each line of code is costing per month, instead of showing where the CPU spends most of its time.

msmith 3y ago
https://www.infracost.io/ https://www.infracost.io/ might do what you're imagining
stusmall 3y ago
Have you looked at OpenCost? It's more k8s focused but of a similar idea

davidgerard 3y ago

We do this. It was a bit faffy to set up, but having dev and staging shut down at night and only be started as needed that day saved us a fortune.

paxys 3y ago

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the savings for myself as a reward, the answer would be a very clear "no" to both. So overall, as an end developer I really have no incentive to work harder and ensure lower operating costs for my company, and I'm sure most developers in the industry are in the same position as me.

jfim 3y ago
Part of it is that it creates an incentive to create wasteful systems, only to "optimize" them later to rack up a bonus. Even if it gets changed to only pay out for reducing spend incurred by other engineers, it's possible to collude in such a way to extract bonuses from the company. A better way to have aligned incentives for the company and the employees would be to allocate a bonus pool for the entire company, from which AWS expenses are taken out of, but that might be a bit unorthodox.
- avidiax 3y ago
  > allocate a bonus pool for the entire company, from which AWS expenses are taken out of Also a perverse incentive. If we use ec2.small, the customer's query will take 3x longer but be half the price. Let's turn off the nightly security audits. We can live with quarterly backups, right? What do we need all these logs for, anyway? We could hack something that works together in 2 weeks, but if we spend 3 months, it could be really efficient, let's do that...
cduzz 3y ago
How are they going to be sure you're not just farming cobras[1]? [1]https://en.wikipedia.org/wiki/Perverse_incentive https://en.wikipedia.org/wiki/Perverse_incentive
nick_ 3y ago
This is one of many insights that hint at why biz-facing cloud architecture is so popular, wasteful, and profitable. The incentives are designed to form an enormous cash siphon. From aggressively marketing toward fearful & liable (or maybe just tech-cost-illiterate) upper-management to the silencing effect that the low-rung experts experience when sounding the alarm.

TheIronMark 3y ago

I did this at a previous employer. I leveraged a Lambda functions and tags applied to instances to determine when they should be and when they should be off.

bornfreddy 3y ago

Does anyone have a good experience with tools / services that track and analyze cloud usage? We don't use any, but could benefit from better visibility in spending patterns.

whummer 3y ago

To give this a slightly different spin: --> "The best optimization is simply not spinning things up!" At least for local development and testing, as made possible by LocalStack (https://localstack.cloud https://localstack.cloud), among other local testing solutions and emulators. We've seen so many teams fall into the trap of "someone forgot to shut down dev resource X for a week and now we've racked up a $$$ bill on AWS". What is everyone's strategy to avoid this kind of situation? Tools like `aws-nuke` (https://github.com/rebuy-de/aws-nuke https://github.com/rebuy-de/aws-nuke) are awesome (!) to clean up unused resources, but frankly they should not be necessary in the first place...

kthejoker2 3y ago

When I joined my current company I quickly found our internal Azure environment was effectively unmanaged. Four weeks in I shaved nearly 55k in monthly spend just scripting out VM shutdowns and service pauses. Cloud revenue in most large companies is at least 25$, maybe up to 40%, pure developer waste because nobody upstairs knows the difference.