The other really common case with delayed connections is that the sshd is waiting on a timeout while it tries to look up the rdns for your connecting IP. U

The other really common case with delayed connections is that the sshd is waiting on a timeout while it tries to look up the rdns for your connecting IP. UseDNS no # disable rdns lookups in sshd logs. See man sshd_config There's a possible security benefit to having it enabled, in that it'll try to resolve your rdns, and then confirm your forward dns matches it, but I'm not sure how much of a benefit that is, in practice.