2 ms·
I don't know if it's yet been tested in court, but the robots.txt conventions have over a decade of widespread custom behind them. It would be hard for anyone c
by gojomo 16y ago
I don't know if it's yet been tested in court, but the robots.txt conventions have over a decade of widespread custom behind them. It would be hard for anyone competent enough to do sophisticated crawling to plead ignorance of them. Custom counts in law:
http://en.wikipedia.org/wiki/Custom_%28law%29 http://en.wikipedia.org/wiki/Custom_%28law%29