3 ms·
> Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers It's plenty logical. That doesn't make it correct. > if
by Dylan16807 2y ago
> Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers
It's plenty logical. That doesn't make it correct.
> if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?
Ha. Ask HTTP Referer.
A million standards have quirks in them that we're stuck with.