Is there a reason someone shouldn't make such a clarification if they choose? "I may not agree with what you say, but I'll defend your right to say it" gets bro

Is there a reason someone shouldn't make such a clarification if they choose? "I may not agree with what you say, but I'll defend your right to say it" gets brought up an awful lot in free speech discussions on HN, and that's fine, but we need to emember that it definitionally includes defending the right to say, implicitly or explicitly, "That guy over there is saying really stupid/dangerous things and this is why he should stop and why you shouldn't listen to him."