There is another side to this risk. Perhaps you do lower the risk of compromise by changing your port. But you also increase the risk of locking yourself out so

There is another side to this risk. Perhaps you do lower the risk of compromise by changing your port. But you also increase the risk of locking yourself out somehow (especially if you use other tricks such as limiting IP ranges). This may seem unlikely to you, but so is being compromised if you're security conscious, which you obviously are if you're thinking about changing the port (you'll stay up to date, disable password authentication, etc). Most security problems occur when you overlook something. For example, you're far more likely to be compromised through a vulnerability in a web app you don't use a package manager for and so don't receive automatic updates for. Or through a user who set a weak password or compromised another machine with his key on it. And so on. I think that on balance it makes little difference when you consider the real reasons why people get compromised.