I've noticed that sometimes interesting answers get downvoted, and irrelevant/badly thought-out answers get upvoted. So I don't take downvotes so seriously now.

I've noticed that sometimes interesting answers get downvoted, and irrelevant/badly thought-out answers get upvoted. So I don't take downvotes so seriously now. However, there's a backlash: if I get a bunch of upvotes, I'll now think that it's probably just a bunch of those idiot voters. Which in a way is a good thing, because it means I have to fall back on my own judgment, instead of being guided by karma. (I do listen to intelligent rebuttals and counter view-points, as they have actual content, unlike a downvote). For a long time, I didn't down-vote anyone. Now I'll sometimes downvote, to put comments in the order of insightfulness; and very occasionally for irrelevance/childishness. For really obnoxious comments, I'll downvote to grey them, and also flag them. But my usual response is to reply "please elaborate" (for a content-free comment), which offers the human behind the comment an opportunity to grow into the community; or an impartial recital of the facts (for a mistaken, inappropriate or irrelevant comment).