2 ms·
Well, ideally you'd be doing HEAD requests and HN is serving (almost) everything from cache, so it should really be no more onerous to their system than ~12M lo
by smt88 10y ago
Well, ideally you'd be doing HEAD requests and HN is serving (almost) everything from cache, so it should really be no more onerous to their system than ~12M lookups in an in-memory index (and a few MB of transfer). As for you... it could definitely be time-consuming if you did it serially. If you did it in parallel, it wouldn't take long but would be "bad citizen" behavior toward HN's servers. There might be a happy medium, but only HN knows what that would be.
I think that if no one speaks up with a database, you should approach HN or Algolia directly. If they wanted you to have this info, it would be trivial for them to dump it (and less strenuous on their resources).
In fact, I'd suggest scraping Algolia (sorted by date[1]) instead of the method I proposed earlier, since it would require far, far less overhead.
1. https://hn.algolia.com/?query=&sort=byDate&prefix&page=0&dateRange=all&type=all https://hn.algolia.com/?query=&sort=byDate&prefix&page=0&dat...
- jacquesm 10y agoGood points. I've found something I can use to start with, older dumps of comments and stories that allow me to at least figure out which ones are the dead ones (because they're not in either set, so they are worth crawling individually, dead items are only a very small number compared to the rest). That should get a base of about 7.5M items cataloged to test with, once I have that working I can use algolia per your suggestion to fill in the blanks. Thank you very much for the suggestions!