3 ms·
Have they ever asked the customers why they prefer scraping over the data deltas?
by PeterStuer 2y ago
Have they ever asked the customers why they prefer scraping over the data deltas?
- ks2048 2y agoI would bet the answer is it is easier to write a script the simply downloads everything it can (foreach <a href=>: download and recurse), rather than looking into which sites provide data dumps and how to use them.
- PeterStuer 2y agoSo the solution would be an edgecached site exactly like the full site with just the deltas since a periodic timepoint? The crawler still crawls but can confidently rest assured it still has all the info with the base+delta as if it had recrawled everyting?