3 ms·
What about using those employees to "crawl" the web for you then?
by Raphmedia 10y ago
What about using those employees to "crawl" the web for you then?
- greglindahl 10y agoI suspect it's roughly the same as a crawer -- same issues of fair use, TOS/CFAA, etc -- but likely there's no expectation that humans will read and follow robots.txt.