3 ms·

You can put up a "no trespassing sign" on your property (although there's some debate as to how much that actually counts for - a quick search pulls up https://

by ikeboy 9y ago

You can put up a "no trespassing sign" on your property (although there's some debate as to how much that actually counts for - a quick search pulls up https://www.washingtonpost.com/news/volokh-conspiracy/wp/2017/04/12/the-fourth-amendment-and-no-trespassing-signs/?utm_term=.4e4998979dce https://www.washingtonpost.com/news/volokh-conspiracy/wp/201...)

jstarfish 9y ago
Robots.txt is not a 'no-trespassing' sign. Robots.txt is a 'whites-only' sign. The information is available to the public, just not for certain classes. This is and should be legally unenforceable. If something is truly meant to be private it should not be referenced from a public-facing page or it should have access control enabled.
- ikeboy 9y ago
  Except race /skin color is a legally protected class, and robots aren't (and why should they be? They can't enter into contracts, conduct business, etc. So it's perfectly legitimate to exclude them from a site where they cannot use it in the intended manner). "If you truly didn't want trespassers you should've put up a gate."
  - jstarfish 9y ago
    Bots being legally protected as a class or not, using robots.txt as the ultimate test of what distinguishes normal traffic from CFAA violations is a very flawed mechanism. It turns your website into a minefield. As a property owner, a no-trespassing sign won't protect you from the lawsuits that result when a toddler drowns in your pool. You're expected to do more (like putting up that gate). Equifax's systems are peppered with "no-trespassing" motds at login. They also have a robots.txt file. We expected them to do more. Same for leaving keys in your ignition, guns unlocked on your nightstand, etc. "Don't touch" signs won't absolve you of responsibility when either gets stolen and used in a spree killing. So yes, as the owner of any sort of asset, in most contexts it is your responsibility to implement access controls to keep unauthorized traffic out.
    - grkvlt 9y ago
      > won't protect you from the lawsuits that result when a toddler drowns in your pool Good analogy. I wonder i operating fa poorly secured website that leaks private information could be seen as an 'Attractive Nuisance' [0] and the owners could be prosecuted for that, rather than the hackers! 0. https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine
- 33W 9y ago
  Maybe it's closer to a No Trespassing sign, written in a language that only certain classes will understand.
- RcouF1uZ4gsC 9y ago
  Robots.txt is more like a "No trucks allowed on street" sign. It allows uses that are typically associated with individuals (viewing a web page, being in a car), while disallowing things that are normally associated with business (web scraping, driving a truck).
- Frqy3 9y ago
  In this case it is a "white bots only" sign, as it allows some bots but wants to block the rest. Even for those who think that robots.txt should be enforceable, allowing some bots but not others makes it difficult for a new player to have the same equitable access to information as the big players.

by ikeboy 9y ago

jstarfish 9y ago

Robots.txt is not a 'no-trespassing' sign. Robots.txt is a 'whites-only' sign. The information is available to the public, just not for certain classes. This is and should be legally unenforceable. If something is truly meant to be private it should not be referenced from a public-facing page or it should have access control enabled.

ikeboy 9y ago
Except race /skin color is a legally protected class, and robots aren't (and why should they be? They can't enter into contracts, conduct business, etc. So it's perfectly legitimate to exclude them from a site where they cannot use it in the intended manner). "If you truly didn't want trespassers you should've put up a gate."
- jstarfish 9y ago
  Bots being legally protected as a class or not, using robots.txt as the ultimate test of what distinguishes normal traffic from CFAA violations is a very flawed mechanism. It turns your website into a minefield. As a property owner, a no-trespassing sign won't protect you from the lawsuits that result when a toddler drowns in your pool. You're expected to do more (like putting up that gate). Equifax's systems are peppered with "no-trespassing" motds at login. They also have a robots.txt file. We expected them to do more. Same for leaving keys in your ignition, guns unlocked on your nightstand, etc. "Don't touch" signs won't absolve you of responsibility when either gets stolen and used in a spree killing. So yes, as the owner of any sort of asset, in most contexts it is your responsibility to implement access controls to keep unauthorized traffic out.
  - grkvlt 9y ago
    > won't protect you from the lawsuits that result when a toddler drowns in your pool Good analogy. I wonder i operating fa poorly secured website that leaks private information could be seen as an 'Attractive Nuisance' [0] and the owners could be prosecuted for that, rather than the hackers! 0. https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine
33W 9y ago
Maybe it's closer to a No Trespassing sign, written in a language that only certain classes will understand.
RcouF1uZ4gsC 9y ago
Robots.txt is more like a "No trucks allowed on street" sign. It allows uses that are typically associated with individuals (viewing a web page, being in a car), while disallowing things that are normally associated with business (web scraping, driving a truck).
Frqy3 9y ago
In this case it is a "white bots only" sign, as it allows some bots but wants to block the rest. Even for those who think that robots.txt should be enforceable, allowing some bots but not others makes it difficult for a new player to have the same equitable access to information as the big players.