Beyond backing up to an offline machine and hoping for the best, it's tricky. I'm not sure I'd be able to cook up something that sits well even thinking on it

Beyond backing up to an offline machine and hoping for the best, it's tricky. I'm not sure I'd be able to cook up something that sits well even thinking on it for some time. My mind goes to immutability (eg: read-only filesystems), being able to repeatably build the hosts/generate as much state as possible, and relying on basic availability/eventual stability -- basically have so much of 'it' that they may not take it all down. To practically do what I envision would require at least a decent amount of hardware, if not engineering time/workflow changes. I think the main benefit from this is that it opens the door to more effective administration. You'd be able to see a bunch of audit policy denials for servers trying to write where they shouldn't. You could also routinely reload the hosts to ensure they're clean. Ideally no one system controls everything, so they have to break many barriers to truly deny you access. By the way, this is why SELinux is so highly recommended. It captures countless different ways these kinds of things can happen. Web servers opening unexpected sockets [for reverse shells], users reading devices they have no business with, etc. Enable it, tune the policies as needed, and then monitor it. It does a good job at protecting you. Too good, too many people disable it in my opinion. Most don't realize it makes a fantastic snitch for nefarious actors.