It's true that just changing the port doesn't add much to protect against brute-force attacks, but it's not the full story. Many automatic scanners try port 22

It's true that just changing the port doesn't add much to protect against brute-force attacks, but it's not the full story. Many automatic scanners try port 22 and then give up for that host. I've found that changing the port means that I'm hit but far fewer scans. One could also use port knocking. Using just a single port it would add the 16 bits of randomness making it 32 bits of extra randomness which is significant. Another advantage of using a different port is that it's often quite easy to detect port scans against a host so an automated tool can easily just block the scanning IP. Slow scans exist but if someone is willing to take that amount of time then there's not much you can do to stop them from eventually finding your service.