Yes, this is why having a passphrase on your key is recommended. The idea is not to use keys instead of passwords, but in addition to: the password protects the

Yes, this is why having a passphrase on your key is recommended. The idea is not to use keys instead of passwords, but in addition to: the password protects the private key locally, which is used to authenticate to the server. However that seems to be uncommon practice.