I'm ambivalent towards this issue and I can't say I have a definite answer (and I'm also someone who thinks threats of violence should carry far far harsher pen

I'm ambivalent towards this issue and I can't say I have a definite answer (and I'm also someone who thinks threats of violence should carry far far harsher penalties, just think of DV for example), however: that logic doesn't hold. You can certainly define when threats or other speech constitute violence. Just as brandishing a gun and threatening to shoot is not violence in the strictest sense of the word but it is violence for all intents and purposes. Just as burning crosses in front of the house of a black family. Of course these are the clear cut examples and leave no margin for error, the majority of situations are much more ambiguous, and there I agree we should tend to err on the side of caution.