The problem with making it cost is no one would down vote the obvious spam and one liners. I think making it public or having meta-moderation where users are as

The problem with making it cost is no one would down vote the obvious spam and one liners. I think making it public or having meta-moderation where users are asked to review down-votes, where if you do to many down votes that meta-moderators flag as incorrect, then you get down-vote privileges removed for x amount of time with 3 strikes and your out and you then have down-vote privileges permanently removed.