The most interesting thing here is that this took down the bastion boxes that allowed PROD SSH access. I would argue that was the biggest red flag by far. As th

The most interesting thing here is that this took down the bastion boxes that allowed PROD SSH access. I would argue that was the biggest red flag by far. As the author points out, if you can't access the machines somehow, you are pretty screwed. Great job that you managed to get back in anyway, but really the production access mechanism should be as dumb, simple, and standalone as you can make it. We use multiple bastion hosts with user accounts provisioned by Ansible. There is one in each data center, plus a "shell" box at AWS that lives outside the firewall and also has SSH access to everything we need. All the bastions can access every server. Plus, some of the mission-critical boxes actually have SSH exposed to the public internet. I would much rather take the security risk with (key-only) OpenSSH than the risk that I get locked out of PROD when I need to fix things after a 2am page. It is entirely possible to make a box so secure that you yourself cannot access it, while not actually doing much to defend against more "reasonable" security risks, like typo-squatting and spear-phishing. It's all about the threat model.