4 ms·
It's an ASP.net core website. I have a 3 droplets powering it $5 for web, $5 for utilities, and a $25 droplet for the HN Api crawler. My API crawler is an C# c
by jermaustin1 7y ago
It's an ASP.net core website. I have a 3 droplets powering it $5 for web, $5 for utilities, and a $25 droplet for the HN Api crawler.
My API crawler is an C# console app that just increments the last crawled item until it gets a 404 then sleeps for 60 seconds. I currently have just over 3GB of data from the API.
- rapfaria 7y agoAre you crawling firebase or algolia?
- jermaustin1 7y agoFirebase
- rapfaria 7y agoThan you could crawl up to the maxitem instead of relying on 404 https://hacker-news.firebaseio.com/v0/maxitem.json?print=pretty https://hacker-news.firebaseio.com/v0/maxitem.json?print=pre...