You give your public key (typically into ~/.ssh/authorized_keys) and then prove you have access to the matching private key as the essential part of the challen

You give your public key (typically into ~/.ssh/authorized_keys) and then prove you have access to the matching private key as the essential part of the challenge. You always keep the private key.