3 ms·
Also, what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible. You're just not always in a place to
by always_good 9y ago
Also, what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible.
You're just not always in a place to scale to the abuse or build something more complex than some simple heuristic filters.
- josteink 9y ago> what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible. Often? Based on what data? I find it much more likely you only often notice aggressive scrapers. That however tells you nothing about the behavior of the average web scraper or web scrapers in general.
- tyingq 9y agoThe system encourages it. Ingress data is cheap, and so many scrapers just default to high frequency.