Maybe I'm missing something, but where does fail2ban come in on the "keys vs. passwords" debate? Do I become more vulnerable if I run fail2ban on a keys-only ss

Maybe I'm missing something, but where does fail2ban come in on the "keys vs. passwords" debate? Do I become more vulnerable if I run fail2ban on a keys-only ssh setup? If not, then why is the argument "don't use passwords or fail2ban" rather than just "fail2ban doesn't provide adequate protection for allowing passwords on ssh?"