4 ms·
How would one know he is disallowed without reading each site?
by mminer237 3y ago
How would one know he is disallowed without reading each site?
- swatcoder 3y agoThe convention is that crawlers first read /robots.txt to see what they're encouraged to scrape and what they're not meant to, and then hopefully honor those directions. In this case, as in many, the disallow rules are intentionally meant to protect the signal quality and efficiency of the crawler.