4 ms·

> web crawlers were never under any obligation to follow robots.txt. other than the fact that you could get successful sued if you do not follow them i.e.: eB

by franze 3y ago

> web crawlers were never under any obligation to follow robots.txt. other than the fact that you could get successful sued if you do not follow them i.e.: eBay v. Bidder's Edge in 2000

nadermx 3y ago
That was overturned, https://en.wikipedia.org/wiki/Intel_Corp._v._Hamidi https://en.wikipedia.org/wiki/Intel_Corp._v._Hamidi and White Buffalo Ventures LLC v. University of Texas at Austin
geor9e 3y ago
The Internet Archive never follows robots.txt and they're still around