You still need the public key to validate the signature. If the attacker can change the public key, he can change the signature without you knowing - unless you

You still need the public key to validate the signature. If the attacker can change the public key, he can change the signature without you knowing - unless you explicitly want to trust each and every key for every gem you install.