I use an SSH CA and it solves this problem. It's simple to setup. You first create a CA, and then you sign the servers public key and the client's public key,

I use an SSH CA and it solves this problem. It's simple to setup. You first create a CA, and then you sign the servers public key and the client's public key, and you take the resulting certs (alongside the CA's pubkey) and move them over to the machines. And I think it's important to note that you can do this process on any machine-- perhaps an air-gapped $30 Raspberry Pi, or if you're less paranoid you can do it on your laptop, but the point is that you can generate keys in bulk, offline, and move them over to dozens of machines in one fell swoop. You still need to maintain an authorized_keys file in your homedir with your client public key, which means if someone was able to get the CA key they still wouldn't be able to get into the server without first getting the key that corresponds to the entry in your authorized_keys file. And likewise, if someone was able to get access to your user key, they would be able to login with it but wouldn't be able to add a backdoor key to the authorized_keys file unless they also had the CA key to sign it with, because both the client and server check the other's certificate against the CA pubkey. Of course, if you see ssh keys as a panacea and subsequently use passwordless sudo or a weak enough password, they have root and can just edit your sshd_config and this was all for naught. But it does solve the problem of an unchecked authorized_keys file.