it absolutely does not defeat the purpose: if you somehow manage to get my password/ssh key _remotely_, there is still one more obstacle you cannot get as it's

it absolutely does not defeat the purpose: if you somehow manage to get my password/ssh key _remotely_, there is still one more obstacle you cannot get as it's _local only_. that is why it's a second factor. something you know with something you have. I'd say this covers the 99% of how credentials are being misappropriated.