6 ms·
Show HN: Tiny CLI to save AWS costs in dev environments when you're sleeping
- aramalipoor 6y agoJust published a tiny CLI utility to save some bucks on AWS development environment while I'm not really working. https://github.com/aramalipoor/aws-cost-saver https://github.com/aramalipoor/aws-cost-saver The idea is to stop/shutdown any resources (such as EC2 instances, RDS databases, ECS tasks, etc.) when you're done for the day and restore them back up next time you continue. Currently it's in a very early stage, a hobby side-project :) Ideally I'm thinking to provide a CloudFormation template or terraform config for example, to provision a scheduled Lambda with predefined "start of work" and "end of work" times and have it automatically do that for you. I'd like to know if something like this would be useful for you too and what do you think about it?
- adamlklein 6y agoThis sounds like a good idea! I had been looking at creating a lamba to check for the existence of pods, services, alternate namespaces etc., as I learn K8s on AWS, to shut down the cluster when I'm not working on it.
- aramalipoor 6y agoThanks. Shutting down K8S cluster when not using is a really good idea. Maybe one day I play around with AWS's k8s cluster and add the trick to aws-cost-saver. Pull requests are also welcome if you have time :) the logic must be similar to EC2 or RDS tricks: https://github.com/aramalipoor/aws-cost-saver/blob/master/src/tricks/stop-rds-database-instances.trick.ts https://github.com/aramalipoor/aws-cost-saver/blob/master/sr...
- WrtCdEvrydy 6y agoThis is amazing.... Edit: I wonder if you could put this inside of a lambda with just a role.
- aramalipoor 6y agoThanks. Agree with you, having it in a Lambda with a scheduled trigger or even manual trigger will be really useful.
- banana_giraffe 6y agoOr an API gateway. I have something similar (though very bespoke for me) that has a simple API Gateway web page as it's front-end. It also opens up a firewall entry for my IP so I can connect to the machine. Depending on my mood I either use the web page or some simple curl aliases to start and stop an instance.
- redacted3212 6y agoCloudCustodian has offered this for quite some time. We use it at our company in all our accounts.
- swyx 6y agointeresting work! i feel like this can't be a new problem. what comparable tools exist for doing this today? its basically just spinning up a staging env while you work and spinning down when you knock off work right? almost like a heroku dyno spinning up for you.
- aliswe 6y agoI would regard it as a fairly new problem actually, the same age as the "developer resources on a per-minute basis" phenomenon.
- z3t4 6y agoIm very interested in learning about your work flows, like how does a typical day look like? I do all my work on a local pc and only use servers for production. So im interesting in how/why you use the cloud stack
- jen20 6y agoHow do you test or develop against cloud resources like ECS using a local PC?
- ununoctium87 6y agohttps://github.com/localstack/localstack https://github.com/localstack/localstack Though I find a stronger argument in using containerisation to ensure an identical application run-time environment no matter the infra it’s running on. If we are talking about infra engineering, then sure. That’s where I find localstack helps
- krono 6y agoOk, I don't know how else to ask and I'm seriously not trying to be an arse but how do you? I'm genuinely interested as it's something I've been struggling with myself lately.
- vageli 6y agoAre you aware of cloud custodian out of Capital One? [0] Does this and much, much more (like delete noncompliant security groups, etc). [0]: https://cloudcustodian.io/ https://cloudcustodian.io/
- 1vuio0pswjnm7 6y agoI'm aware that Capital One was found guilty of not properly securing customer information when they started moving it to "the cloud" (AWS). I'm also aware that the person who orchestrated the data breach, by exploiting the bank's AWS (mis)configuration, had been hired by Amazon as a developer. Maybe the bank's IT staff created "cloud custodian" after the incident. https://www.marketwatch.com/story/capital-one-fined-80-million-for-breach-that-accessed-data-of-106-million-card-holders-2020-08-06 https://www.marketwatch.com/story/capital-one-fined-80-milli...
- destory-everyth 6y agoCustodian was already there , they have a strict pipeline but what I hear is that the environment where the hack took place was bypassing the normal organisational pipe governance and did not even have coverage of the custodian on the account. Basically shadow it. I still put a lot of blame on the way AWS Iam makes it incredibly hard to stop the use of the credentials outside the vpc in the event they are stolen , for example source ip restrictions do not work if the bucked it using kms encryption because kms will decrypt on users behalf appearing to come from a different ip than the user . The called via is a farce that only work with about 4 services out of hundreds and the meta data v2 with its ip level TTL is a marketing gimmick
- aramalipoor 6y agoThanks for sharing this. Looks like a sophisticated solution, although might be too complex and enterprise-y for what I needed as a sleepy developer, but is definitely worth having a look :)
- dabeeeenster 6y agoThis is awesome - Does it work cross-region? +1 to add in: - Elasticache instances - Load balancers associated with ECS clusters
- barkingcat 6y agoI find it price gouging that AWS doesn't already implement this on their own within their pricing structure and that you need a 3rd party tooling to do this.
- cybrix 6y agoThey earn more money by not doing this.
- spockz 6y agoThe downside of shutting down the resources is that there may be none available anymore in that sizing or pricing when you want to start them up again. I suppose they also don’t want to risk being liable for that.
- bufferoverflow 6y agoIf you don't like their prices, why don't you just switch to another provider? Plenty of good clouds. And even more dedicated / VPS servers, if you want truly crazy savings.
- aliswe 6y agoAre you seriously alleging that NOT implementing a "stop resources when I sleep/am not around" feature amounts to price gouging?
- pc86 6y agoThat's not what price gouging is.
- aynsof 6y agoAWS Instance Scheduler is an official AWS Solution: https://aws.amazon.com/solutions/implementations/instance-scheduler/ https://aws.amazon.com/solutions/implementations/instance-sc... It allows you to create automatic start and stop schedules for your Amazon EC2 and Amazon RDS instances. It uses a combination of CloudWatch/Lambda/DynamoDB to check against tagged instances.
- gvenkat 6y agoaws has a instance scheduler as a solution that achieves some of this objectives https://aws.amazon.com/solutions/implementations/instance-scheduler/ https://aws.amazon.com/solutions/implementations/instance-sc...
- aramalipoor 6y agoThat's a good alternative for EC2 / RDS instances but when it comes to other cases like DynamoDB provisioned throughput or Kinesis shards or ElastiCache it wouldn't help much
- 60secz 6y agoBecause my favorite place for AWS credentials is some random NPM module?
- debaserab2 6y agoThis works like every other AWS CLI tool - it uses your shells AWS credentials dir or ENV vars
- fareesh 6y agoIs it common to have a dev environment on AWS? I just use it for production and maybe staging. Staging is scheduled to run during office hours.
- carnitas 6y agoMy company tasked me to mad dash their dev environment to AWS. Why? Because “cloud”, and perceived less risky than moving QA and Prod first.
- vitalysh 6y agoI guess depends on the size of the company. How else would you constantly test integrations between different services? Especially if some services can't be easily ran on the laptop. Also if you add terraform/ansible and all that jazz to the table. Where else? Infra team needs dev/stage as well.
- wuunderbar 6y agoLocalstack can get you pretty far.
- fareesh 6y agoYes but in those cases would you really be looking to be Frugal like this? I figured the underlying assumption is that this is a tip for regular folks
- ldoughty 6y agoOps people need a Dev environment to test, especially for Infrastructure as code. With an increased connectivity between developer code and Infrastructure, I prefer my team finds out everything in a development environment that's exactly like staging and prod. That way deployment to staging is proof of the IAC change. There's less risk. This also lets developers try out new services in the cloud without impacting the staging... (This is on top of a sandbox for research, but integration of research takes time and an environment too)
- mcintyre1994 6y ago
- weeefun 6y agoIt looks like a nice tool doing it's job well :-) I'd also consider services like provisioned DynamoDB tables or Kinesis streams in this context because there you can also waste lots of money depending on your setup. For example, you could decrease the provisioned read/write capacity for a DynamoDB table or decrease the shards of a Kinesis stream over night. I've discussed these topics in a blog post, in the context of a CloudFormation stack, if anyone's interested: https://www.sebastianhesse.de/2018/04/22/shut-down-cloudformation-stack-resources-over-night-using-aws-lambda/ https://www.sebastianhesse.de/2018/04/22/shut-down-cloudform...
- aramalipoor 6y agoThanks for sharing your experience. Both of them sounds like great ideas, will add them to ToDo right away :)
- chromedev 6y agoThat is what Kind is for.
- stekern 6y agoIf you're using IaC and have it set up in a CI/CD pipeline, you could also achieve the same by having a cronjob set a flag outside of work hours, and use conditionals in your IaC based on the value of that flag (e.g., for Terraform `count = var.scale_down ? 0 : 1`)
- aramalipoor 6y agoThere were a few reasons why I couldn't use IaC: 1) Using conditionals to achieve this usually makes terraform unmanageable specially if your terraform is complex. 2) Sometimes you'd need to do more complex steps to save some money, for example in case of ElastiCache you'd need to snapshot/delete/create. 3) Using terraform in a Lambda to schedule this was not straightforward (in my current company) so I gave up making it work :D
- iflowfor8hours 6y agoGreat addition to pile of scripts to take with you on problem solving missions. I have something similar, but written in terraform. WRT AWS cost visibility, I find it is usually faster to spin up and grab most of the relevant information using mlabouardy's [komiser](https://github.com/mlabouardy/komiser https://github.com/mlabouardy/komiser) than to try and standardize yourself. Not shilling, just a tool I like.
- kevin_nisbet 6y agoWhile this only works with instances, I usually just cron the shutdown command with a future time. Nice thing is, if working late I give myself a couple hour window to cancel shutdown on any nodes I'm still using. echo "0 22,0,2,4,6,8,10 * * * /sbin/shutdown +115" > /etc/cron.d/autoshutdown I'm all for a better tool, just wanted to point out a builtin alternative.
- ngcc_hk 6y agoAny similar for other cloud services like azure, gcp and especially digital ocean
- gbolo 6y agolooks cool. If anyone is interested, I made something similar a while back (web app instead of cli): https://github.com/gbolo/aws-power-toggle https://github.com/gbolo/aws-power-toggle