There is no ad hominem fallacy in that comment. You are probably misusing the term to refer to any negative comment about a person. The ad hominem fallacy is a

There is no ad hominem fallacy in that comment. You are probably misusing the term to refer to any negative comment about a person. The ad hominem fallacy is a fallacy of irrelevance, of the form "the arguer's argument is invalid because the arguer has this negative quality," when the negative quality is irrelevant to the validity of the argument.