4 ms·

On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.

dragonwriter 5y ago
> On the contrary, there are no laws that say you can't scrape a site. You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.
- EMIRELADERO 5y ago
  Depends on what exactly is being scraped. If it's something like price data or exact values then it isn't protected by copyright at all.
  - kingcharles 5y ago
    Price data can be protected by copyright as a compliation of data. https://www.bitlaw.com/copyright/database.html https://www.bitlaw.com/copyright/database.html
    - EMIRELADERO 5y ago
      In that case, the data compilation itself would be protected, not the individual data points. If I used a scraper to copy everything verbatim, then yes, it would be a violation.
marginalia_nu 5y ago
There are laws against unauthorized computer access. This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.
- Aeolun 5y ago
  The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP. Maybe we need a status code that means ‘lay off all the requests made from this entire system’?
  - marginalia_nu 5y ago
    How do you interpret a 401 Unauthorized to mean you are authorized to access the resource?
    - jhgb 5y ago
      https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli... > Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response. So it would seem that it actually doesn't positively imply that you're NOT authorized. Which kind of makes sense; machines can't detect legality of things, just that certain procedural niceties haven't been observed.
      - marginalia_nu 5y ago
        Fine, send a 403 then. > The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"
        hattmall 5y ago
        403 is per request, not requester. I get random 403s when just browsing some websites. Does that mean I should close the browser and not hit refresh for fear of breaking some wire fraud unauthorized access law?

dragonwriter 5y ago

> On the contrary, there are no laws that say you can't scrape a site. You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.

EMIRELADERO 5y ago
Depends on what exactly is being scraped. If it's something like price data or exact values then it isn't protected by copyright at all.
- kingcharles 5y ago
  Price data can be protected by copyright as a compliation of data. https://www.bitlaw.com/copyright/database.html https://www.bitlaw.com/copyright/database.html
  - EMIRELADERO 5y ago
    In that case, the data compilation itself would be protected, not the individual data points. If I used a scraper to copy everything verbatim, then yes, it would be a violation.

marginalia_nu 5y ago

There are laws against unauthorized computer access. This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.

Aeolun 5y ago
The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP. Maybe we need a status code that means ‘lay off all the requests made from this entire system’?
- marginalia_nu 5y ago
  How do you interpret a 401 Unauthorized to mean you are authorized to access the resource?
  - jhgb 5y ago
    https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli... > Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response. So it would seem that it actually doesn't positively imply that you're NOT authorized. Which kind of makes sense; machines can't detect legality of things, just that certain procedural niceties haven't been observed.
    - marginalia_nu 5y ago
      Fine, send a 403 then. > The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"
      - hattmall 5y ago
        403 is per request, not requester. I get random 403s when just browsing some websites. Does that mean I should close the browser and not hit refresh for fear of breaking some wire fraud unauthorized access law?