5 ms·

Respecting robots.txt isn't a rule, it's a courtesy.

sschueller 2y ago
It can be a legal thing and hold up in court in some countries. Is respecting a "no trespassing" sign also just courtesy?
- dylan604 2y ago
  What countr[y|ies] has codified respecting robots.txt file in a legal manner?
- throwup238 2y ago
  > It can be a legal thing and hold up in court in some countries. Which countries? (I'm genuinely curious) > Is respecting a "no trespassing" sign also just courtesy? In countries with right to roam, the answer is often yes. In states like California with right to access waterways, many "no trespassing" signs are unenforceable too if they block access to rivers or beaches.
  - Ukv 2y ago
    > Which countries? (I'm genuinely curious) The EU's AI act points to the DSM directive's text and data mining exemption, allowing for commercial data mining so long as machine-readable opt-outs are respected - robots.txt is typically taken as the established standard for this. It has generally been respected as far as I'm aware (though, this article asserts otherwise). In the US, respecting robots.txt/<NoAI>/... could potentially be a fallback if the Fair Use defense falls through, by implied license like for website caching in Field v. Google Inc ("Google reasonably interpreted absence of meta-tags as permission to present 'Cached' links to the pages of Field's site").
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- tedivm 2y ago
  In the US, where all these companies are based, it is legal to scrape any site that doesn't require a login regardless of the robots.txt file. However, what you do with that data is still subject to copyright law.
- OutOfHere 2y ago
  It is akin to a religious convention that some believe in. If you force your religion on others, expect others to force theirs on you, so don't. It is not trespassing because the site is already on the public web.
skybrian 2y ago
Courtesy is a practical way of avoiding legal disputes. If you ignore robots.txt, that means you don’t have an agreement, so the question is what do you have a right to do, and do you really want to argue about that?
- lagniappe 2y ago
  You're free to bring anything to court, that's your right.

It can be a legal thing and hold up in court in some countries. Is respecting a "no trespassing" sign also just courtesy?

dylan604 2y ago
What countr[y|ies] has codified respecting robots.txt file in a legal manner?
throwup238 2y ago
> It can be a legal thing and hold up in court in some countries. Which countries? (I'm genuinely curious) > Is respecting a "no trespassing" sign also just courtesy? In countries with right to roam, the answer is often yes. In states like California with right to access waterways, many "no trespassing" signs are unenforceable too if they block access to rivers or beaches.
- Ukv 2y ago
  > Which countries? (I'm genuinely curious) The EU's AI act points to the DSM directive's text and data mining exemption, allowing for commercial data mining so long as machine-readable opt-outs are respected - robots.txt is typically taken as the established standard for this. It has generally been respected as far as I'm aware (though, this article asserts otherwise). In the US, respecting robots.txt/<NoAI>/... could potentially be a fallback if the Fair Use defense falls through, by implied license like for website caching in Field v. Google Inc ("Google reasonably interpreted absence of meta-tags as permission to present 'Cached' links to the pages of Field's site").
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
tedivm 2y ago
In the US, where all these companies are based, it is legal to scrape any site that doesn't require a login regardless of the robots.txt file. However, what you do with that data is still subject to copyright law.
OutOfHere 2y ago
It is akin to a religious convention that some believe in. If you force your religion on others, expect others to force theirs on you, so don't. It is not trespassing because the site is already on the public web.

skybrian 2y ago

Courtesy is a practical way of avoiding legal disputes. If you ignore robots.txt, that means you don’t have an agreement, so the question is what do you have a right to do, and do you really want to argue about that?

lagniappe 2y ago
You're free to bring anything to court, that's your right.