2 ms·
I'm not sure any publisher means for their robots.txt to be read as: "You're disallowed, but go head and slurp the content anyway so you can look for external
by swatcoder 3y ago
I'm not sure any publisher means for their robots.txt to be read as:
"You're disallowed, but go head and slurp the content anyway so you can look for external links or any indication that maybe you are allowed to digest this material anyway, and then interpret that how you'd like. I trust you to know what's best and I'm sure you kind of get the gist of what I mean here."
- mminer237 3y agoHow would one know he is disallowed without reading each site?
- swatcoder 3y agoThe convention is that crawlers first read /robots.txt to see what they're encouraged to scrape and what they're not meant to, and then hopefully honor those directions. In this case, as in many, the disallow rules are intentionally meant to protect the signal quality and efficiency of the crawler.