3 ms·
They work because they offer ways to opt out, they honor crawl delay, setting ideal scraping times, IndexNow, etc. And they give you real, valuable traffic in
by spiderfarmer 7mo ago
They work because they offer ways to opt out, they honor crawl delay, setting ideal scraping times, IndexNow, etc.
And they give you real, valuable traffic in return.
- avazhi 7mo agoMost offer ways to opt out, some don’t. Scraping somebody’s website might be annoying or problematic traffic-wise but that’s a far (very far) step removed from saying scrapers should be criminalised. The latter statement is outright laughable.