I use ssh on a non-standard port, not for security, but to avoid all the annoying attempts in the logs. I use ssh keys only. I've disabled the use of passwords

I use ssh on a non-standard port, not for security, but to avoid all the annoying attempts in the logs. I use ssh keys only. I've disabled the use of passwords for any account other than root on the console, and for that, I use a very long random password which is stored very carefully, encrypted with a long passphrase. This is only needed in emergencies. Be careful to protect your ssh keys (long passphrases and ssh-agent), and it's probably a good idea to use hashes in your known_hosts file rather than addresses so malware can't find all the hosts you can access.