There isn’t “really no reason” why it’s annoying; it’s exactly as you said: the site is falsely representing your upvote as a deliberate action. And your action

There isn’t “really no reason” why it’s annoying; it’s exactly as you said: the site is falsely representing your upvote as a deliberate action. And your action changes a number, labeled “kudos”, seen by all future visitors to the page, so your action ends up implying endorsement of the page to future visitors. That’s why the kudos feature annoys me – it falsely represents the page to be popular, but rather than just completely making up a number of kudos like a normal liar, it ties the number of kudos to the actions of unwitting page visitors. That gives the author plausible deniability that it’s possible that that many people really did like the page, though in fact that is very unlikely.