8 ms·
AWS Nuke – delete all resources associated with AWS account
- Centigonal 4y agoThis is a fantastic utility to have in my back pocket, and the attention to safety is commendable. nice!
- pcthrowaway 4y agoNeeds a flag to run non-interactively
- danw1979 4y agoI nearly took this bait
- pcthrowaway 4y ago`aws-nuke --no-dry-run --non-interactive --i-really-mean-it --yes-i-agree-to-the-second-prompt-also`
- code_duck 4y agoI can barely imagine what is in my AWS account. The last time I used it was 2011.
- lvh 4y agoI've used aws-nuke a bunch but the use case seems significantly diminished now that AWS Organizations has the ability to delete entire accounts.
- wielebny 4y agoLast time I've checked it was a lengthy process involving attaching a credit card, leaving the organization and then deleting the account. Has it been changed?
- bonobocop 4y agohttps://docs.aws.amazon.com/organizations/latest/APIReference/API_CloseAccount.html https://docs.aws.amazon.com/organizations/latest/APIReferenc... Now has an API (with some caveats)
- dhess 4y agoI'm in the process of doing this now. You can close accounts from Control Tower without needing to log in as root to each separate account, adding a credit card, removing it from the org, and then closing it manually. However, you can only close them from Control Tower at the rate of 2 to 3 per month, due to a hard limit quota which cannot be changed, even if you request it. Needless to say, this sucks when you've followed AWS's own best practices and created lots of accounts using Control Tower's "vending machine." AWS's archaic account model is one reason we've switched to GCP.
- Aeolun 4y agoIt seems to me this cannot possibly be a hard limit. If it’s a hard limit it’s only because AWS wants to milk you dry.
- janstice 4y agoI suspect it’s a hard limit to prevent disgruntled (former) admin blast radius.
- wongarsu 4y agoIf that was the concern then surely they could enable support to restore accounts during some grace period. The machinery for that already exists for people who completely close their AWS accounts down. Maybe it was quicker to implement with a hard limit, or there is some internal service that can't easily handle large volumes of account removals. But if it was Amazon losing money from this limitation instead of the customer I imagine it would be fixed pretty quickly.
- liftm 4y ago
- jmainguy 4y ago
- OJFord 4y agoI would love a `terraform <plan|apply> --nuke`. Not `destroy` - the opposite - destroy/nuke what I don't have in my config.
- deleted 4y ago[deleted]
- iasay 4y agoOh that would be glorious. Not sure how it'd be possible though.
- mrg2k8 4y agoMay be possible using driftctl and some API calls.
- OJFord 4y agoI've hitherto assumed it didn't because it wasn't, short of calling every single `*:Describe*` API anyway. But the existence of aws-nuke makes me think (I haven't looked into what it's doing yet) there must be a better way of discovering used services/resources. Through billing perhaps?
- scapecast 4y agoWe've built an (open source) asset inventory product just for that use case, to continuously discover all inventory, not just what's provisioned by Terraform (or Pulumi, etc.). Link to the repo in my HN profile.
- thayne 4y agoThat's one of my big complaints with terraform, and it's state-based system. It doesn't inform you if there are resources you don't know about.
- inkyoto 4y agoIt is a feature and a good one (with a caveat though). terraform is best used on a per solution basis: one solution, one dedicated terraform project that will manage its state and its own state only. Multiple projects and people work carry out work in the same cloud account in parallel, and their terraform projects are not meant to interfere with each other. It works best for solutions that make use of fully managed cloud services. Then there are also platform or connectivity level cloud resources (e.g. Transit Gateway and subnets that are mapped into the internal organisational network address space in AWS) that a random terraform project ought not to manage. Lastly, if there is an actual need, a resource that terraform does not know about can be manually imported into the terraform's project state. This works best when infrastructure level resources were manually created a while ago and now have to be refactored into and managed by a terraform project. It is a tedious process that has to proceed with a lot of caution. The caveat. It gets somewhat tricky when a non-serverless cloud resource requires an explicit subnet range allocation within an existing and managed CIDR or similar. There is no one solution to fit it all but containing such projects to their own dedicated VPC and setting up the VPC peering between a solution specific VPC and the main account VPC usually works satisfactory. That is, for example, how Kafka (AWS MSK) can be introduced into an AWS account without affecting existing CIDR mapping.
- evanfarrar 4y agohttps://github.com/genevieve/leftovers https://github.com/genevieve/leftovers A co-worker made this when we worked together on a project that ran a large number of terraform configurations in CI against real IaaSes. Each account was a sandbox, so we would run this at the end of a pipeline to clean up any failures or faulty teardowns.
- bilekas 4y agoWow! This is actually some great work!
- crazysim 4y agoUnlike aws-nuke, there seem to be actual automated tests here for this tool. That's really neat! Though, I do feel the project reaches out to quite a lot of IAAS's for its level of current maintenance, which seems to be somewhat zero. aws-nukes gets contributions here and there and maybe that's because it's quite focused. That's a real shame, the implementations for each provider look to be great, standalone by themselves.
- jftuga 4y agoThis fork is 36 commits ahead. It aims to stabilize deletion (so far for AWS) and add extended regex filtering support. https://github.com/notrepo05/leftovers https://github.com/notrepo05/leftovers
- hk1337 4y agoI would think they should probably change the name so it doesn’t come across as an official AWS product but seems nice.
- orf 4y agoShout out to AWS batch, where if you delete the role assigned to a compute cluster the cluster itself becomes impossible to delete. Found this out after using AWS nuke
- OJFord 4y agoThat's a weird one, but surely an aws-nuke bug? It must already use a deliberate order - there's plenty of resources that need anything linked/constituent deleted first - so that order is/was just not correct for those?
- bilekas 4y agoNah I've seen this too. It is considered an expected behavior. In a way it makes sense. But it's not well documented.
- OJFord 4y agoBut that's what I mean, if that's how it works, nuke needs to account for it. I assume it already does for other dependencies, because there are a lot of them - including that do make more (obvious) sense. You're not the only one to complain about the documentation though, so it's easy to see how aws-nuke apparently missed it!
- Marazan 4y agoNo, Batch's interaction with IAM roles and permission is super weird and not at all documented. It is easy to screw it up.
- dekhn 4y agoAws batch is a terrible product
- aaaaaaaaata 4y agoAWS is a terrible product, unless you happen to be in specific corners they've bothered to document/make their UI architecture not a horror show.
- yieldcrv 4y agothis will shave 20% off of Amazon's earnings!
- jrockway 4y agoI sure wish cloud providers had a "fake" implementation of their APIs that I could use in tests, for cases like the one that this program was written for. There are so many cross-dependencies that can be checked ("sorry can't create that VM, you need a service account first" or "VM names can only be 69 characters and can't contain '!'") without creating actual resources. You can test all the edge cases that resolve to failure in a unit test, and then only need one integration test to get that end-to-end "when I run 800 Terraform files, can I talk to my webserver at test239847abcf123.domain-name.com?". I've found that the alternative to this is either a CI run that takes hours, or simply not testing the edge cases and hoping for the best. The former is a nightmare for velocity, the latter is a nightmare for having more than one person working on the team ever. Lately, I've been fortunate to not be working in the "general cloud provider" space (where everything is a black box that can change at any moment, and documentation is an afterthought of afterthoughts) and have only focused on things going on in Kubernetes. To facilitate fast tests, I forked Kubernetes, made some internal testing infrastructure public, and implemented a kubelet-alike that runs "containers" in the same process as the test. For the application I work on at work, we are basically a data-driven job management system that runs on K8s. People have already written the easy tests; build the code into a container, create a Kubernetes cluster, start the app running, poke at it over the API. These take for-fucking-ever to run. (They do give you good confidence that Linux's sleep system call works well, though! Boy can it sleep.) With my in-process Kubernetes cluster, most of that time goes away, and you can still test a lot of stuff. If you want to test "what happens when a very restrictive AppArmor policy is applied to all pods in the cluster", yeah, the "fake" doesn't work. If you want to test "when my worker starts up, will it start processing work", it works great. (And, since the 100% legit "api machinery" is up and running, you can still test things like "if my user's spec contains an invalid pod patch, will they get a good error message?") Most bugs (and mistakes that people make while adding new features) are in that second category, and so you end up spending milliseconds instead of minutes testing the parts of your application that are most hurtful to users when they break. (And, nobody is saying not to run SOME live integration tests. You should always start with, and keep, integration tests against real environments running real workloads. When they pass, you get some confidence that there are no major showstoppers. When they fail, you want the lighter-weight tests to point you with precision to the faulty assumption or bug.) Anyway... it makes me sad that things like aws-nuke are the kind of tooling you need to produce reliable software focused on deployment on cloud providers. I'd certainly pay $0.01 more per VM hour to be able to delete 99% of my slow tests. But I think I'm the only person in the world that thinks tests should be thorough and fast, so I'm on my own here. Sad.
- time0ut 4y agoI’ve contributed to and used this tool extensively. I have accounts where I run this thing out of a CodeBuild job on a cron schedule to dejunk after a day of terraform development. Fantastic tool.
- thedougd 4y agoAre Azure resource groups still pretty comprehensive? You use to be able to create all your stuff in a resource group. Delete the group and it deletes all the stuff with it. Super convenient.
- bbkane 4y agoYeah they seem to be. Except Azure AD stuff, which kinda makes sense (still wish that part were more easily automated)
- qwertyuiop_ 4y agoProject sponsored by Azure ?
- svenwltr 4y agoNo, we use it to clean up after our Terraform integration-tests. It is not always possible to use `terraform destroy`, because it might happen that the state does not allow it (at least that was our experience a couple of years ago).
- ezekiel11 4y agoso i have this weird monthly charge I cannot for the love of god find out what is causing it. I cancelled my credit card and now my credit score has taken a hit. All because some unknown AWS service that is in some zone that I cannot find at all and neither can AWS support.
- ___8___ 4y agoIs the charge the same every month?
- yownie 4y agoWhat does the account statement say?
- GeorgeWBasic 4y agoThat seems like the sort of thing chargebacks are intended to solve. Your credit shouldn't be taking a hit for that.
- aaaaaaaaata 4y agoI love how everyone wants to help you/figure out the mystery bug, because, after all, you just need knowledge and experience to maybe avoid getting screwed around by AWS.
- arkadiyt 4y ago99% of our AWS resources are terraformed, but developers constantly push back / want to use the console to create stuff to test or play around with. So we setup a separate "hack" AWS account and give them admin access in there, and have an automated job using AWS Nuke to delete everything in there once a quarter.
- Frajedo 4y agoWe give each one of our developers their very own aws account managed through AWS organizations service. They are full administrators and responsible for resources and cost. So far we haven’t had any issues or bad surprises, although we have setup some aws billing alerts just in case. Feel free to make them responsible for cost and resources and you’ll be surprised how well they can manage their own account.
- vageli 4y ago> We give each one of our developers their very own aws account managed through AWS organizations service. They are full administrators and responsible for resources and cost. How many developers work at your organization?
- Frajedo 4y ago25 at the moment :)
- cure 4y ago> although we have setup some aws billing alerts just in case. My experience with these has been decidedly mixed. As in, you define them and never, ever see an alert.
- deleted 4y ago[deleted]
- Frajedo 4y agoHmmm, weird. We always get the alerts in time with thresholds set to 70% of the wished value.
- AtomicOrbital 4y agoto me its obvious amazon managers intentionally avoid giving us such a delete all resources button simply to maximize profits
- adamw2k 4y agoFunny enough, on the completely opposite end of the spectrum, I was once surprised that after you close an account (post M&A in this case), you can typically restore the resources for ~90 days if you decide you want it back and don't nuke before you request the closure. Can be useful or scary depending on the contents of the account...
- master_crab 4y agoYeah it’s actually a selling point in case a disgruntled employee or hacker gets credentials and asks for the account to be shutdown.
- fideloper 4y agoThis is also a useful tool just to find resources hiding within an account. It defaults to performing a dry run (it’s a PITA to actually delete stuff, on purpose).
- tpmx 4y agoI noticed that this GCP article about Terraform best practices linked to a similar tool: cloud-nuke (https://github.com/gruntwork-io/cloud-nuke https://github.com/gruntwork-io/cloud-nuke) https://cloud.google.com/docs/terraform/best-practices-for-terraform https://cloud.google.com/docs/terraform/best-practices-for-t... > After you run the terraform destroy command, also run additional clean-up procedures to remove any resources that Terraform failed to destroy. Do this by deleting any projects used for test execution or by using a tool like cloud-nuke. > Warning: Don't use such tools in a production environment. It seems like this tool only supports AWS though, at least nowadays. I kind of feel bad for the GCP people here. Damned if you do (link to an external project, which might change), damned if you don't. Besides this thing, I really enjoyed reading this article though! AWS is missing this kind of content.
- thematrixturtle 4y agoIn GCP, the tool is not necessary, because you can simply delete the containing project to "nuke" any resources attached to it.
- deleted 4y ago[deleted]
- tpmx 4y agoThere is a 30 day "marked for deletion period" after you do this. The resources related to the project aren't immediately nuked. https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects https://cloud.google.com/resource-manager/docs/creating-mana...
- thematrixturtle 4y agoDeleting a project does immediately turn off (stop billing) the vast majority of associated resources, including VMs. The main exception is storage, but that cost is typically fairly marginal.
- 4y ago
- emersonrsantos 4y agoNamespaces could solve all of that. Why AWS haven't added that yet? I understand that they rarely change the service, but c'mon, we're in 2022.
- aaaaaaaaata 4y agoHow does that help them make more money?
- thayne 4y agoIt astounds me that aws doesn't have a way to do this built in. Or even just a way to list all the resources you have, across all services in an account, without having to make thousands of API calls and hope you didn't miss a resource type.
- anothernewdude 4y agoShout out to my Azure homies enjoying their Resource Groups.
- brundolf 4y agoThe fact that this is hard enough to do it requires a dedicated third-party tool
- 44gg44gg 4y agoI deleted my AWS account because SES refused to take me out of sandbox due to a security violation that could not be disclosed to me. Interesting that almost all forms of SMTP are blocked once you are on some secret black list.
- aporetics 4y agoWhat’s the fallout?
- svenwltr 4y agoHello. I am one of the maintainers. Sorry that we did not spend much time for aws-nuke lately. Feel free to ask any questions.