3 ms·
They provide database dumps already, and those dumps have the diff information. Crawlers are ignoring the dumps and scraping the websites anyway.
by joshuaissac 1y ago
They provide database dumps already, and those dumps have the diff information. Crawlers are ignoring the dumps and scraping the websites anyway.
- PeterStuer 1y agoHave they ever asked the customers why they prefer scraping over the data deltas?
- ks2048 1y agoI would bet the answer is it is easier to write a script the simply downloads everything it can (foreach <a href=>: download and recurse), rather than looking into which sites provide data dumps and how to use them.
- PeterStuer 1y agoSo the solution would be an edgecached site exactly like the full site with just the deltas since a periodic timepoint? The crawler still crawls but can confidently rest assured it still has all the info with the base+delta as if it had recrawled everyting?