5 ms·

If it is publicly available information it is legal to scrape it, regardless of what robots.txt says. See: https://www.webspidermount.com/is-web-scraping-legal

If it is publicly available information it is legal to scrape it, regardless of what robots.txt says. See: https://www.webspidermount.com/is-web-scraping-legal-yes/ https://www.webspidermount.com/is-web-scraping-legal-yes/

otterley 1y ago
As an attorney (and this is not legal advice), I don't think it's quite that simple. The court held that the CFAA does not proscribe scraping of pages to which the user already has access and in a way that doesn't harm the service, and thus it's not a crime. But there are other mechanisms that might impact a scraper, such as civil liability, that have not been addressed uniformly by the courts yet. And if you scrape in such a way that does harm the operator (e.g. by denying service), it might still be unlawful, even criminal. There's a relevant footnote in the cited HiQ Labs v. LinkedIn case: "LinkedIn’s cease-and-desist letter also asserted a state common law claim of trespass to chattels. Although we do not decide the question, it may be that web scraping exceeding the scope of the website owner’s consent gives rise to a common law tort claim for trespass to chattels, at least when it causes demonstrable harm." They also said: "Internet companies and the public do have a substantial interest in thwarting denial-of-service attacks and blocking abusive users, identity thieves, and other ill-intentioned actors." It's a good idea to take legal conclusions from media sites with a grain of salt. Same goes for any legal discussion on social media, including HN. If you want a thorough analysis of legal risk--either for your business or for personal matters--hire a good lawyer.
- hgaddipa001 1y ago
  Smart
  - nhod 1y ago
    Or run your legal questions through a frontier model and then have a lawyer verify the answers. You can save a lot of money and time. Yes, all LLM caveats apply. Due your diligence. But they are quite good at this now.
    - hgaddipa001 1y ago
      Hmm this is a good idea too
    - otterley 1y ago
      Have you actually tried this approach? I’m curious as to the result, especially when you took it to your lawyer. Not a contract review but a business practice risk evaluation.
      - Terretta 1y ago
        Some context from coverage of GPT 5: https://legaltechnology.com/2025/08/08/openai-launches-gpt-5-some-initial-legal-benchmarking-results-thoughts/ https://legaltechnology.com/2025/08/08/openai-launches-gpt-5... https://www.artificiallawyer.com/2025/08/08/gpt-5-tops-harveys-biglaw-bench-eval/ https://www.artificiallawyer.com/2025/08/08/gpt-5-tops-harve... Remember when "asking for a friend" was a thing? Today's expression is "I asked a friend". You can try that when talking to your lawyer about your latest ChatGPT — they might still believe you.
- nikolayasdf123 1y ago
  what a nonsense. they explicitly say "do not scrape us, unless we approve". they put paywalls and captchas. their service is literally selling access to users data. now you scraping it. this is direct violation and direct harm to their business, despite their explicit statements for you to stop. you loose the case, it is clear as day.
nikolayasdf123 1y ago
what a nonsense. this is equivalent of "sovereign citizens" online. go and try it, and get yourself into jail.
- breadwinner 1y ago
  Do not confuse strong language with strong argument. Yours is the former not the latter.

otterley 1y ago

As an attorney (and this is not legal advice), I don't think it's quite that simple. The court held that the CFAA does not proscribe scraping of pages to which the user already has access and in a way that doesn't harm the service, and thus it's not a crime. But there are other mechanisms that might impact a scraper, such as civil liability, that have not been addressed uniformly by the courts yet. And if you scrape in such a way that does harm the operator (e.g. by denying service), it might still be unlawful, even criminal. There's a relevant footnote in the cited HiQ Labs v. LinkedIn case: "LinkedIn’s cease-and-desist letter also asserted a state common law claim of trespass to chattels. Although we do not decide the question, it may be that web scraping exceeding the scope of the website owner’s consent gives rise to a common law tort claim for trespass to chattels, at least when it causes demonstrable harm." They also said: "Internet companies and the public do have a substantial interest in thwarting denial-of-service attacks and blocking abusive users, identity thieves, and other ill-intentioned actors." It's a good idea to take legal conclusions from media sites with a grain of salt. Same goes for any legal discussion on social media, including HN. If you want a thorough analysis of legal risk--either for your business or for personal matters--hire a good lawyer.

hgaddipa001 1y ago
Smart
- nhod 1y ago
  Or run your legal questions through a frontier model and then have a lawyer verify the answers. You can save a lot of money and time. Yes, all LLM caveats apply. Due your diligence. But they are quite good at this now.
  - hgaddipa001 1y ago
    Hmm this is a good idea too
  - otterley 1y ago
    Have you actually tried this approach? I’m curious as to the result, especially when you took it to your lawyer. Not a contract review but a business practice risk evaluation.
    - Terretta 1y ago
      Some context from coverage of GPT 5: https://legaltechnology.com/2025/08/08/openai-launches-gpt-5-some-initial-legal-benchmarking-results-thoughts/ https://legaltechnology.com/2025/08/08/openai-launches-gpt-5... https://www.artificiallawyer.com/2025/08/08/gpt-5-tops-harveys-biglaw-bench-eval/ https://www.artificiallawyer.com/2025/08/08/gpt-5-tops-harve... Remember when "asking for a friend" was a thing? Today's expression is "I asked a friend". You can try that when talking to your lawyer about your latest ChatGPT — they might still believe you.
nikolayasdf123 1y ago
what a nonsense. they explicitly say "do not scrape us, unless we approve". they put paywalls and captchas. their service is literally selling access to users data. now you scraping it. this is direct violation and direct harm to their business, despite their explicit statements for you to stop. you loose the case, it is clear as day.

nikolayasdf123 1y ago

what a nonsense. this is equivalent of "sovereign citizens" online. go and try it, and get yourself into jail.

breadwinner 1y ago
Do not confuse strong language with strong argument. Yours is the former not the latter.