3 ms·
Allowing a site to only be scraped/crawled by professionals (i.e. Google) ensures that those engaging such activity do so in a respectable manner. It would be
by spynxic 9y ago
Allowing a site to only be scraped/crawled by professionals (i.e. Google) ensures that those engaging such activity do so in a respectable manner.
It would be ideal to define the terms for scraping (similar to how API calls work) as opposed to the acceptable authors of scrapers.
- wslh 9y agoI am not sure you can define those terms in a formal way. You can crawl the data in some way but use it for many different purposes that are difficult to control.
- ellimilial 9y agoFancy extension to robots.txt?
- madshiva 9y agoThey are professionals now, but at the beginning it was not at all.
- spynxic 9y agoI would imagine even in the beginning that they, as developers, practiced good procedures while scouring the internet. Given the difference in the size of the internet now and then, the backlash faced for poor practices may not be comparable.. so perhaps not.