Interesting question. Anecdotally, I believe Facebook recently took the position that robots.txt has no legal standing: "Their contention was robots.txt had no

Interesting question. Anecdotally, I believe Facebook recently took the position that robots.txt has no legal standing: "Their contention was robots.txt had no legal force and they could sue anyone for accessing their site even if they scrupulously obeyed the instructions it contained. The only legal way to access any web site with a crawler was to obtain prior written permission." http://petewarden.typepad.com/searchbrowser/2010/04/how-i-got-sued-by-facebook.html http://petewarden.typepad.com/searchbrowser/2010/04/how-i-go...