That's what file system permissions are for, on the private key. (I might be wrong about this, but...) I've seen quite a few instances where SSH refuses to let

That's what file system permissions are for, on the private key. (I might be wrong about this, but...) I've seen quite a few instances where SSH refuses to let people even try to log in if there's a chance that a private key has the wrong permissions applied, so as to minimize the risk of someone accidentally letting others read it -- as you suggested. You'll find the default SSH config on the more noteworthy Linux distributions to do this 90% of the time time without even asking :)