3 ms·

IMHO there's no need to worry (but you should disable password access), but if you really want to, port knocking is an option.