28 ms·
> My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product Not really. Th
by JohnFen 3y ago
> My understanding is that the philosophy behind robots.txt is owners not wanting their content automatically included in someone else's product
Not really. That use case is done, of course, but the primary purpose of robots.txt is to help crawlers by indicating what parts of the website are appropriate to be searched and what parts aren't.
Robots.txt is not intended primarily as a means to defend a site against crawlers. That's why it relies on the goodwill of crawlers to work.