The direct parent commenter (i.e. the person the comment responds to) cannot downvote. It just does not show a downvote button for them, only an upvote button.

The direct parent commenter (i.e. the person the comment responds to) cannot downvote. It just does not show a downvote button for them, only an upvote button. So the downvotes have to come from everyone else.