There should be a backup device or system offsite somewhere that automatically grabs incrementals of everything (and yells at you if it can't for some reason),

There should be a backup device or system offsite somewhere that automatically grabs incrementals of everything (and yells at you if it can't for some reason), essentially nobody should have any kind of remote access to it, and the fewest possible people should have physical access to it and gaining physical access should be such a hassle that you don't do it unless it's critical. That's how I do it, anyway. We can grant that this can be logistically difficult at certain scales, but it doesn't fall into the "engineering-impossible" bucket until you reach Facebook's size.