I sure wish cloud providers had a "fake" implementation of their APIs that I could use in tests, for cases like the one that this program was written for. Ther

I sure wish cloud providers had a "fake" implementation of their APIs that I could use in tests, for cases like the one that this program was written for. There are so many cross-dependencies that can be checked ("sorry can't create that VM, you need a service account first" or "VM names can only be 69 characters and can't contain '!'") without creating actual resources. You can test all the edge cases that resolve to failure in a unit test, and then only need one integration test to get that end-to-end "when I run 800 Terraform files, can I talk to my webserver at test239847abcf123.domain-name.com?". I've found that the alternative to this is either a CI run that takes hours, or simply not testing the edge cases and hoping for the best. The former is a nightmare for velocity, the latter is a nightmare for having more than one person working on the team ever. Lately, I've been fortunate to not be working in the "general cloud provider" space (where everything is a black box that can change at any moment, and documentation is an afterthought of afterthoughts) and have only focused on things going on in Kubernetes. To facilitate fast tests, I forked Kubernetes, made some internal testing infrastructure public, and implemented a kubelet-alike that runs "containers" in the same process as the test. For the application I work on at work, we are basically a data-driven job management system that runs on K8s. People have already written the easy tests; build the code into a container, create a Kubernetes cluster, start the app running, poke at it over the API. These take for-fucking-ever to run. (They do give you good confidence that Linux's sleep system call works well, though! Boy can it sleep.) With my in-process Kubernetes cluster, most of that time goes away, and you can still test a lot of stuff. If you want to test "what happens when a very restrictive AppArmor policy is applied to all pods in the cluster", yeah, the "fake" doesn't work. If you want to test "when my worker starts up, will it start processing work", it works great. (And, since the 100% legit "api machinery" is up and running, you can still test things like "if my user's spec contains an invalid pod patch, will they get a good error message?") Most bugs (and mistakes that people make while adding new features) are in that second category, and so you end up spending milliseconds instead of minutes testing the parts of your application that are most hurtful to users when they break. (And, nobody is saying not to run SOME live integration tests. You should always start with, and keep, integration tests against real environments running real workloads. When they pass, you get some confidence that there are no major showstoppers. When they fail, you want the lighter-weight tests to point you with precision to the faulty assumption or bug.) Anyway... it makes me sad that things like aws-nuke are the kind of tooling you need to produce reliable software focused on deployment on cloud providers. I'd certainly pay $0.01 more per VM hour to be able to delete 99% of my slow tests. But I think I'm the only person in the world that thinks tests should be thorough and fast, so I'm on my own here. Sad.