Please honour rel="nofollow" attributes on anchor tags. Every comment on my blog articles contains an anchor tag which looks like this: <a href="/articlename

Please honour rel="nofollow" attributes on anchor tags. Every comment on my blog articles contains an anchor tag which looks like this: <a href="/articlename?reply_to=comment_id" rel="nofollow">Reply</a> Clicking that link reloads the page, but with a textarea underneath the comment available for replying. So your crawler is currently loading the same blog post over and over with a different reply_to attribute each time atm.