Votes go down and up and that's normal. You comment seemed skeptically dismissive, so it was downvoted. Once someone responded and you interacted in a way that

Votes go down and up and that's normal. You comment seemed skeptically dismissive, so it was downvoted. Once someone responded and you interacted in a way that showed your true interest, subsequent readers no longer perceived it as skeptical. As a result, your votes increased.