didn't know about it. https://github.com/HackerNews/API https://github.com/HackerNews/API . Checked it out, the api wouldn't have helped me: the API reponse for

didn't know about it. https://github.com/HackerNews/API https://github.com/HackerNews/API . Checked it out, the api wouldn't have helped me: the API reponse for this flagged story doesn't indicate, that it has been flagged. https://hacker-news.firebaseio.com/v0/item/29103056.json?print=pretty https://hacker-news.firebaseio.com/v0/item/29103056.json?pri... On the other hand: they gave me an idea. I can structure the crawler differently. Right now, I am now following the next page (apparently there is a limit on the number of 'next' pages). Instead, it is possible to get the maximum item id, then just decrement the item id down and down again.