My guess is just that my sites fly under the radar because they are unimportant, have little traffic etc.. But if people finds you random (not <1024) ssh port,

My guess is just that my sites fly under the radar because they are unimportant, have little traffic etc.. But if people finds you random (not <1024) ssh port, they probably scanned all of you ports? Which seems drastic. Last time I checked, a few years ago, it was not normal to scan all the ports on a large block of IPs. Maybe that changed, but I have never seen that on the few machines and networks I can check. Anyway, my point is that targeted attacks is a another ballgame, but that does not make it pointless to protect against the dumb, automated attacks.