I firmly believe No-SSH is a goal you should always strive for but never actually achieve. There are always cases where you need to do really detailed troublesh

I firmly believe No-SSH is a goal you should always strive for but never actually achieve. There are always cases where you need to do really detailed troubleshooting that requires things like tcpdump, a debugger, or even running lsof (or other expensive command that you can't afford to run regularly and log). Some random user with some edge case will put one of your nodes for your service into a bad state. Throwing away the machine will just lose the state and push the user somewhere else. Also centralized log and metrics stores get real big and expensive really fast. Sometimes you simply can't afford to ship everything to it. So you'll find yourself putting detailed debug logs on the "free" ephemeral/local disks while info logs go to your central store.