Then you lose one factor which is the username/password. If someone managed to get your hardware token and was able to brute force the PIN (if there was a PIN

Then you lose one factor which is the username/password. If someone managed to get your hardware token and was able to brute force the PIN (if there was a PIN set), then they could log into any of your accounts. By requiring both, an adversary would not be able to log in unless they had the certificate and the username/password. Using one of the two would not be sufficient.