I have noticed that a single downvote sometimes counts for -2 rather than -1. Sometimes I'll undo a downvote because I think a comment wasn't bad enough to cost

I have noticed that a single downvote sometimes counts for -2 rather than -1. Sometimes I'll undo a downvote because I think a comment wasn't bad enough to cost 2 points. I have been trying to spot the pattern behind it for over a year with no luck. Can you explain this or is it secret?