There is nothing special about speech here. In general I have a legal right to go the the park and wave my arms about. But punching people doesn't fall under

There is nothing special about speech here. In general I have a legal right to go the the park and wave my arms about. But punching people doesn't fall under that right, and wouldn't even if "the right to wave your arms about" were explicitly in the constitution. The thing that makes people so keen to find limitations on free speech in particular is that they are often keen to supress what someone else is saying and are casting about for an excuse that they can stretch in their favour.