If you password protect the keys it's essentially the same at your end as if you were using a password. The only difference being that they need to copy the key

If you password protect the keys it's essentially the same at your end as if you were using a password. The only difference being that they need to copy the keys if after compromising the password they want to log on elsewhere.