I think the really key aspect is that we can recognize when someone is using tags improperly like that. Downvotes are generic, you can't really say that someone

I think the really key aspect is that we can recognize when someone is using tags improperly like that. Downvotes are generic, you can't really say that someone used one "wrong". So we can look at systems like the "meta-moderation" that Slashdot had, where you can ask other (trusted) users whether they think someone else is abusing their tools. If they are, you can start trusting their tags less, or eventually even take away their ability to use them. And if they also have to build up to being able to use tags in the first place, that makes it way more difficult to keep abusing something since they can't just create a new account to immediately circumvent any punishment.