in the first case, you're merely abstaining. In the case you describe, you're passing judgement on the other person. People who pass judgement on other people

in the first case, you're merely abstaining. In the case you describe, you're passing judgement on the other person. People who pass judgement on other people are unlikely to be much fun to talk to at dinner parties.