4 ms·

> Hardly legally enforceable, no? Up to the lawyers and the courts. And I can definitely see how it could be interpreted to be a legitimate "bots are not autho

> Hardly legally enforceable, no? Up to the lawyers and the courts. And I can definitely see how it could be interpreted to be a legitimate "bots are not authorized to view this site" directive since robots.txt is a standard which is respected across the industry.

jraph 3y ago
I could write that people wearing yellow pants should not view my website on my website prominently. I sure would hope it would not cut it in courts.
- doctorpangloss 3y ago
  Why not? If you wrote that on your cake shop’s banner, you’d prevail with more egregious policies.
- notjoemama 3y ago
  If you want to completely ignore context, then sure. I got a chuckle. But I think ad absurdism didn’t work here. Also, you deleted your comment conveying disdain over the Twitter to X rebrand. Considering rebranding is very common in mergers and acquisitions, would you say you feel strongly about it, and if so, why do you feel that way?
  - jraph 3y ago
    It was not disdain, more like childish mockery. I deleted it because I found this comment was crap. It did not bring anything useful, and was not funny neither. And would have risked bringing sterile discussion. It was just not at the level of what I expect from myself. Sorry if you tried to reply and failed as a consequence. > would you say you feel strongly about it No. There are so many things that are more meaningful and enjoyable to focus on in life that spending emotion on some rebranding is not worth it. I'm not even a user of the thing. The rename is stupid though, even more so than my deleted comment, no doubt about it. ;-)
- falcolas 3y ago
  So, a clickwrap User Agreement? Those are generally accepted as legal and enforceable. Though enforcing a pants color might be hard.
jncfhnb 3y ago
LinkedIn cases have set precedents otherwise
- Klonoar 3y ago
  https://news.ycombinator.com/item?id=37264676#37266853 https://news.ycombinator.com/item?id=37264676#37266853 For an interesting breakdown including from a lawyer on this very site.
- dahfizz 3y ago
  https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn Originally, the ninth circuit ruled that web scraping was allowed. This was overturned by a Supreme Court decision. It was ultimately found that HiQ was in violation of the User Agreement and they settled with LinkedIn.
  - jncfhnb 3y ago
    > LinkedIn petitioned the Supreme Court to review the Ninth Circuit's decision.[9] In an order on June 14, 2021,[10] the Supreme Court vacated the Ninth Circuit's decision on the basis of their ruling on CFAA the week prior in Van Buren v. United States, which had ruled that the "exceeds authorized access" of CFAA only applies when an individual has valid access to a system but accesses parts of a system they are not intended to access.[4] The case was remanded to the Ninth Circuit for further review under Van Buren.[11] > In a second ruling in April 2022 the Ninth Circuit affirmed its decision.[5][6] Is this not saying the opposite?
    - dahfizz 3y ago
      The final ruling on this case was in November 2022[1]. The April 2022 decision was overturned in August 2022. It came back up in November 2022, where it was decided that HiQ has violated the User Agreement and had to pay damaged to LinkedIn. HiQ no longer exists, so I think the case is definitely done. It doesn't seem to have set any firm precedent about whether web scraping violates the CFAA, but considering HiQ ultimately had to pay damages, I don't think its a resounding win for web scrapers. [1] https://www.natlawreview.com/article/court-finds-hiq-breached-linkedin-s-terms-prohibiting-scraping-mixed-ruling-declines https://www.natlawreview.com/article/court-finds-hiq-breache...
cma 3y ago
HiQ v LinkedIn
- falcolas 3y ago
  Seems like this was ultimately judged to be a breach of the user agreement, and they settled. At least, that's the last ruling noted in the wiki article. The SC also basically ruled that that HiQ did not have the right to scrape LinkedIn's website.
  - cma 3y ago
    It says they remanded it and the court it was remanded to affirmed its decision. "Its" is ambiguous, but it refers to the court it was remanded to, this is the source they reference: > In its second ruling on Monday, the Ninth Circuit reaffirmed its original decision and found that scraping data that is publicly accessible on the internet is not a violation of the Computer Fraud and Abuse Act, or CFAA, which governs what constitutes computer hacking under U.S. law. https://techcrunch.com/2022/04/18/web-scraping-legal-court/ https://techcrunch.com/2022/04/18/web-scraping-legal-court/
dylan604 3y ago
Can you compare a robots.txt file on a website to a sign on a physical door that says "authorized admittance only"? Both the website and the building are open to the public, but there are designated areas for authorized visitors only.
- falcolas 3y ago
  IMO, yes. And you can also trespass users - prevent them from ever coming back - even as you provide a business generally open to the public.
  - dylan604 3y ago
    much like a physical business having the right to refuse service, a website should also be able to refuse serving data to users that have abused the service. a bartender says you've had too many, so no more service. websites can rate limit as well. we recently saw Musk test this concept.
    - jraph 3y ago
      Where I live, you can't really legally refuse serving an individual without a strong reason. And if you did, you'd better be able to prove that you didn't do it for discriminatory reasons. Here, bartenders can say you've had too many, they are supposed to stop selling you alcohol if they think it could be dangerous for you or other people. This is one of the cases where not only the business can refuse to serve you, it also has to. Now, I'm not a lawyer but I wouldn't expect robots.txt to be legally enforceable. Of course, in any case, the website can totally refuse to fulfill a request, rate limit, etc. I'd say this is not comparable to the bartender or a business refusing to serve you though. Responding to an HTTP request is not offering a commercial service. And if it is, whatever legal document that promises you the service can totally specify that they won't promise to answer your robot.
      - dylan604 3y ago
        If you're being an nuisance and causing the other patrons to have a less than ideal experience, absolutely, you should be asked to leave. If your site is constantly getting scraped and your service is affected due to it which causes other users to have slowed responses, you should absolutely be able to refuse requests from known knuckleheads. To this day, bars have a list of "refuse entry". Sporting teams have life time bans to hooligans. Sure, the ban hammer should be slow to swing, but it shouldn't a never allowed to swing situation.

jraph 3y ago

I could write that people wearing yellow pants should not view my website on my website prominently. I sure would hope it would not cut it in courts.

doctorpangloss 3y ago
Why not? If you wrote that on your cake shop’s banner, you’d prevail with more egregious policies.
notjoemama 3y ago
If you want to completely ignore context, then sure. I got a chuckle. But I think ad absurdism didn’t work here. Also, you deleted your comment conveying disdain over the Twitter to X rebrand. Considering rebranding is very common in mergers and acquisitions, would you say you feel strongly about it, and if so, why do you feel that way?
- jraph 3y ago
  It was not disdain, more like childish mockery. I deleted it because I found this comment was crap. It did not bring anything useful, and was not funny neither. And would have risked bringing sterile discussion. It was just not at the level of what I expect from myself. Sorry if you tried to reply and failed as a consequence. > would you say you feel strongly about it No. There are so many things that are more meaningful and enjoyable to focus on in life that spending emotion on some rebranding is not worth it. I'm not even a user of the thing. The rename is stupid though, even more so than my deleted comment, no doubt about it. ;-)
falcolas 3y ago
So, a clickwrap User Agreement? Those are generally accepted as legal and enforceable. Though enforcing a pants color might be hard.

jncfhnb 3y ago

LinkedIn cases have set precedents otherwise

Klonoar 3y ago
https://news.ycombinator.com/item?id=37264676#37266853 https://news.ycombinator.com/item?id=37264676#37266853 For an interesting breakdown including from a lawyer on this very site.
dahfizz 3y ago
https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn Originally, the ninth circuit ruled that web scraping was allowed. This was overturned by a Supreme Court decision. It was ultimately found that HiQ was in violation of the User Agreement and they settled with LinkedIn.
- jncfhnb 3y ago
  > LinkedIn petitioned the Supreme Court to review the Ninth Circuit's decision.[9] In an order on June 14, 2021,[10] the Supreme Court vacated the Ninth Circuit's decision on the basis of their ruling on CFAA the week prior in Van Buren v. United States, which had ruled that the "exceeds authorized access" of CFAA only applies when an individual has valid access to a system but accesses parts of a system they are not intended to access.[4] The case was remanded to the Ninth Circuit for further review under Van Buren.[11] > In a second ruling in April 2022 the Ninth Circuit affirmed its decision.[5][6] Is this not saying the opposite?
  - dahfizz 3y ago
    The final ruling on this case was in November 2022[1]. The April 2022 decision was overturned in August 2022. It came back up in November 2022, where it was decided that HiQ has violated the User Agreement and had to pay damaged to LinkedIn. HiQ no longer exists, so I think the case is definitely done. It doesn't seem to have set any firm precedent about whether web scraping violates the CFAA, but considering HiQ ultimately had to pay damages, I don't think its a resounding win for web scrapers. [1] https://www.natlawreview.com/article/court-finds-hiq-breached-linkedin-s-terms-prohibiting-scraping-mixed-ruling-declines https://www.natlawreview.com/article/court-finds-hiq-breache...

cma 3y ago

HiQ v LinkedIn

falcolas 3y ago
Seems like this was ultimately judged to be a breach of the user agreement, and they settled. At least, that's the last ruling noted in the wiki article. The SC also basically ruled that that HiQ did not have the right to scrape LinkedIn's website.
- cma 3y ago
  It says they remanded it and the court it was remanded to affirmed its decision. "Its" is ambiguous, but it refers to the court it was remanded to, this is the source they reference: > In its second ruling on Monday, the Ninth Circuit reaffirmed its original decision and found that scraping data that is publicly accessible on the internet is not a violation of the Computer Fraud and Abuse Act, or CFAA, which governs what constitutes computer hacking under U.S. law. https://techcrunch.com/2022/04/18/web-scraping-legal-court/ https://techcrunch.com/2022/04/18/web-scraping-legal-court/

dylan604 3y ago

Can you compare a robots.txt file on a website to a sign on a physical door that says "authorized admittance only"? Both the website and the building are open to the public, but there are designated areas for authorized visitors only.

falcolas 3y ago
IMO, yes. And you can also trespass users - prevent them from ever coming back - even as you provide a business generally open to the public.
- dylan604 3y ago
  much like a physical business having the right to refuse service, a website should also be able to refuse serving data to users that have abused the service. a bartender says you've had too many, so no more service. websites can rate limit as well. we recently saw Musk test this concept.
  - jraph 3y ago
    Where I live, you can't really legally refuse serving an individual without a strong reason. And if you did, you'd better be able to prove that you didn't do it for discriminatory reasons. Here, bartenders can say you've had too many, they are supposed to stop selling you alcohol if they think it could be dangerous for you or other people. This is one of the cases where not only the business can refuse to serve you, it also has to. Now, I'm not a lawyer but I wouldn't expect robots.txt to be legally enforceable. Of course, in any case, the website can totally refuse to fulfill a request, rate limit, etc. I'd say this is not comparable to the bartender or a business refusing to serve you though. Responding to an HTTP request is not offering a commercial service. And if it is, whatever legal document that promises you the service can totally specify that they won't promise to answer your robot.
    - dylan604 3y ago
      If you're being an nuisance and causing the other patrons to have a less than ideal experience, absolutely, you should be asked to leave. If your site is constantly getting scraped and your service is affected due to it which causes other users to have slowed responses, you should absolutely be able to refuse requests from known knuckleheads. To this day, bars have a list of "refuse entry". Sporting teams have life time bans to hooligans. Sure, the ban hammer should be slow to swing, but it shouldn't a never allowed to swing situation.