4 ms·

Legitimate uses of scraping include price comparison, and finding copyright or trademark infringement.

Dah00n 5y ago
>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.
- EMIRELADERO 5y ago
  On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.
  - dragonwriter 5y ago
    > On the contrary, there are no laws that say you can't scrape a site. You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.
    - EMIRELADERO 5y ago
      Depends on what exactly is being scraped. If it's something like price data or exact values then it isn't protected by copyright at all.
      - kingcharles 5y ago
        Price data can be protected by copyright as a compliation of data. https://www.bitlaw.com/copyright/database.html https://www.bitlaw.com/copyright/database.html
        EMIRELADERO 5y ago
        In that case, the data compilation itself would be protected, not the individual data points. If I used a scraper to copy everything verbatim, then yes, it would be a violation.
  - marginalia_nu 5y ago
    There are laws against unauthorized computer access. This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.
    - Aeolun 5y ago
      The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP. Maybe we need a status code that means ‘lay off all the requests made from this entire system’?
      - marginalia_nu 5y ago
        How do you interpret a 401 Unauthorized to mean you are authorized to access the resource?
        jhgb 5y ago
        https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli... > Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response. So it would seem that it actually doesn't positively imply that you're NOT authorized. Which kind of makes sense; machines can't detect legality of things, just that certain procedural niceties haven't been observed.
        marginalia_nu 5y ago
        Fine, send a 403 then. > The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"
- jonatron 5y ago
  If you look at Ryanair, they've had legal battles with scrapers for more than 10 years, so it's not that simple.
  - aembleton 5y ago
    I found Ryanair one of the more friendly ones to scrape, albeit for my own personal project. When you query for flights, they make a GET request with a JSON object in response, complete with flight times and prices. For example: curl "https://www.ryanair.com/api/booking/v4/en-gb/availability?ADT=1&CHD=0&DateIn=&DateOut=2021-11-15&Destination=BER&Disc=0&INF=0&Origin=MAN&TEEN=0&promoCode=&IncludeConnectingFlights=false&FlexDaysBeforeOut=2&FlexDaysOut=2&ToUs=AGREED" | jq
    - jonatron 5y ago
      I haven't tried to scrape Ryanair, you could be right that it's trivial. It's the legal side that has a long and interesting history. Personally I wouldn't scrape them unless it was when working for a company that can afford lawyers.
    - shapefrog 5y ago
      {"code":"TermsOfUseAreNotAccepted","message":"TermsOfUseAreNotAccepted"}
kzrdude 5y ago
I would say legitimate use includes archiving, and nothing more.

>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.

EMIRELADERO 5y ago
On the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.
- dragonwriter 5y ago
  > On the contrary, there are no laws that say you can't scrape a site. You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.
  - EMIRELADERO 5y ago
    Depends on what exactly is being scraped. If it's something like price data or exact values then it isn't protected by copyright at all.
    - kingcharles 5y ago
      Price data can be protected by copyright as a compliation of data. https://www.bitlaw.com/copyright/database.html https://www.bitlaw.com/copyright/database.html
      - EMIRELADERO 5y ago
        In that case, the data compilation itself would be protected, not the individual data points. If I used a scraper to copy everything verbatim, then yes, it would be a violation.
- marginalia_nu 5y ago
  There are laws against unauthorized computer access. This is a scenario where you have a server explicitly saying "Stop! You are not permitted to access this computer!", and yet you persist in circumventing that by hiding your identity and accessing it anyway. Those are some murky waters.
  - Aeolun 5y ago
    The server says nothing of the kind. The response that was previously positive is now broken, and it happens to be fixed if you access it from a different IP. Maybe we need a status code that means ‘lay off all the requests made from this entire system’?
    - marginalia_nu 5y ago
      How do you interpret a 401 Unauthorized to mean you are authorized to access the resource?
      - jhgb 5y ago
        https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli... > Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response. So it would seem that it actually doesn't positively imply that you're NOT authorized. Which kind of makes sense; machines can't detect legality of things, just that certain procedural niceties haven't been observed.
        marginalia_nu 5y ago
        Fine, send a 403 then. > The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"
jonatron 5y ago
If you look at Ryanair, they've had legal battles with scrapers for more than 10 years, so it's not that simple.
- aembleton 5y ago
  I found Ryanair one of the more friendly ones to scrape, albeit for my own personal project. When you query for flights, they make a GET request with a JSON object in response, complete with flight times and prices. For example: curl "https://www.ryanair.com/api/booking/v4/en-gb/availability?ADT=1&CHD=0&DateIn=&DateOut=2021-11-15&Destination=BER&Disc=0&INF=0&Origin=MAN&TEEN=0&promoCode=&IncludeConnectingFlights=false&FlexDaysBeforeOut=2&FlexDaysOut=2&ToUs=AGREED" | jq
  - jonatron 5y ago
    I haven't tried to scrape Ryanair, you could be right that it's trivial. It's the legal side that has a long and interesting history. Personally I wouldn't scrape them unless it was when working for a company that can afford lawyers.
  - shapefrog 5y ago
    {"code":"TermsOfUseAreNotAccepted","message":"TermsOfUseAreNotAccepted"}

kzrdude 5y ago

I would say legitimate use includes archiving, and nothing more.