3 ms·

To be clear, robots.txt is not legally binding, Google is not bound to follow it, and in fact I believe that Google doesn't follow it and hasn't for a very long

To be clear, robots.txt is not legally binding, Google is not bound to follow it, and in fact I believe that Google doesn't follow it and hasn't for a very long time, for the simple reason that many sites' robots.txt file is wrong. The intent of robots.txt is to help crawlers, for example, to keep crawlers from getting stuck in a recursive loop of dynamic pages, or from crawling pages with no value. robots.txt is not for banning, restricting, or hindering crawlers.

superkuh 3y ago
That's just because google is a corporate person who is more equal than a human person. Human persons, at least in the USA, get charged under the CFAA 1030 law if they're using non-browser tools to access the public website of someone with power and/if they happen to rock the boat (like weev w/wget). That's not to say that I disagree. In most cases robots.txt is not legally binding. It only becomes a legal danger to not follow it when the person running the site has power and can buy a DA to indict you.
- rafark 3y ago
  If a tool can access a url, does that not make it a browser?
  - TeMPOraL 3y ago
    Not under any but most narrow of meanings, i.e. "can follow URLs / can talk HTTP". By itself, it's not a browser to users, it's not a browser to software developers, and it's definitely not a browser to lawyers and judges.
    - rafark 3y ago
      Is there a legal definition of a web browser though? I think it’s an interesting topic.
- TeMPOraL 3y ago
  > like weev w/wget Speaking of this and other cases of trying to punish someone for every iteration of a for loop - I wonder if the result would be the same if the accused drove actual browser to click stuff in a for loop, vs. using curl directly. I imagine the same, but then... ... what if they paid N people some token amount of money, to have each of those people do one step of the loop and send them the result? Does executing a for loop entirely on in part on the human substrate instead of in silico is seen as abuse under CFAA? (I have a feeling that it might not be - there's lots of jobs online and offline that involve one company paying lots of people some money for gathering information from their competitors, in a way the latter very much don't like.)
lisasays 3y ago
for the simple reason that many sites' robots.txt file is wrong. Which is of course not the real reason. The reason Google doesn't follow the robots.txt protocol is (1) they don't want to (2) they can get away with it.
floomk 3y ago
They are in the EU. If something was not meant to be accessible you may not scrape it.

superkuh 3y ago

That's just because google is a corporate person who is more equal than a human person. Human persons, at least in the USA, get charged under the CFAA 1030 law if they're using non-browser tools to access the public website of someone with power and/if they happen to rock the boat (like weev w/wget). That's not to say that I disagree. In most cases robots.txt is not legally binding. It only becomes a legal danger to not follow it when the person running the site has power and can buy a DA to indict you.

rafark 3y ago
If a tool can access a url, does that not make it a browser?
- TeMPOraL 3y ago
  Not under any but most narrow of meanings, i.e. "can follow URLs / can talk HTTP". By itself, it's not a browser to users, it's not a browser to software developers, and it's definitely not a browser to lawyers and judges.
  - rafark 3y ago
    Is there a legal definition of a web browser though? I think it’s an interesting topic.
TeMPOraL 3y ago
> like weev w/wget Speaking of this and other cases of trying to punish someone for every iteration of a for loop - I wonder if the result would be the same if the accused drove actual browser to click stuff in a for loop, vs. using curl directly. I imagine the same, but then... ... what if they paid N people some token amount of money, to have each of those people do one step of the loop and send them the result? Does executing a for loop entirely on in part on the human substrate instead of in silico is seen as abuse under CFAA? (I have a feeling that it might not be - there's lots of jobs online and offline that involve one company paying lots of people some money for gathering information from their competitors, in a way the latter very much don't like.)

lisasays 3y ago

for the simple reason that many sites' robots.txt file is wrong. Which is of course not the real reason. The reason Google doesn't follow the robots.txt protocol is (1) they don't want to (2) they can get away with it.

floomk 3y ago

They are in the EU. If something was not meant to be accessible you may not scrape it.