6 ms·

Is that legal?

ocdtrekkie 5y ago
Why wouldn't it be? Google scrapes the web to populate it's results, why wouldn't other search engines scrape the web as well? Google is a website.
- xpe 5y ago
  This is naive. Web sites have various terms of service.
  - therein 5y ago
    Violating a website's ToS is hardly illegal, though.
    - xpe 5y ago
      > Violating a website's ToS is hardly illegal, though. Can you be more specific as to your claim? Not illegal in what sense(s)? And what is your basis for the claim? I'm not a lawyer, but saying "violating a website's ToS is hardly illegal" is fraught advice. While individuals may get some leeway when it comes to ToS violations (see [1] and [2]), I would expect companies scraping and/or extracting content would be treated differently. [1]: https://www.eff.org/deeplinks/2010/07/court-violating-terms-service-not-crime-bypassing https://www.eff.org/deeplinks/2010/07/court-violating-terms-... [2]: https://arstechnica.com/tech-policy/2020/03/court-violating-a-sites-terms-of-service-isnt-criminal-hacking/ https://arstechnica.com/tech-policy/2020/03/court-violating-... [3]: https://www.octoparse.com/blog/10-myths-about-web-scraping https://www.octoparse.com/blog/10-myths-about-web-scraping [4]: https://www.law.com/newyorklawjournal/almID/1202610687621/?slreturn=20210523014130 https://www.law.com/newyorklawjournal/almID/1202610687621/?s...
      - Kiro 5y ago
        The only way for it to be illegal is if they're breaking a law, but then it would be illegal regardless of what the ToS says.
- lern_too_spel 5y ago
  Because Google's robots.txt disallows it, and those websites allow it.
  - smsm42 5y ago
    robots.txt is not a legal contract. It's just a convention to express the wishes of the site author, but there's no legal obligation to follow these wishes.
    - lern_too_spel 5y ago
      It does indicate that those other sites want Google to scrape them, while Google does not want others to scrape their results, which is an important distinction ocdtrekkie ignored for whether the scrapee will want to take legal action.
      - ocdtrekkie 5y ago
        You may wish to review https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l... Google Search results are definitely "public data" so long as Google provides them to anyone who asks.
        lern_too_spel 5y ago
        Then why does Startpage pay Google and DDG pay Microsoft?
        ocdtrekkie 5y ago
        While scraping search results isn't illegal, by any means, it's also not illegal for Google or Microsoft to block requests they believe are from competing search engines. Presumably the cost of paying them is less than the cost of hiring engineers to constantly try to find new ways to outwit Google and Microsoft engineers. Again, if scraping data from websites without permission, Google simply wouldn't exist. Bear in mind, robots.txt is a feature that Google and Microsoft choose to respect, but the default assumption search engines have made from the beginning, is that they are free to grab whatever they want from the web, unless you ask them otherwise to please not.
- SamBam 5y ago
  It's not scraping static text in order to point you to those sites, it's using the features of the site to perform a service better than you can do yourself. It's completely different. If I made a site that claimed to help you with your math homework and simply sent the queries to WolframAlpha, that would also not just be "scraping."
  - ocdtrekkie 5y ago
    This is basically what Google does to Wikipedia and rebrands as "Knowledge Graph".
    - SamBam 5y ago
      Google has donated many millions to the Wikimedia Foundation, basically in payment for this. (But, again, there's a difference between scraping and echoing a request on another site and waiting for its response. The latter is basically unauthorized use of its API, not scraping.)
      - ocdtrekkie 5y ago
        It'll be very telling if Google switches to using Wikipedia Enterprise: https://www.wired.com/story/wikipedia-finally-asking-big-tech-to-pay-up/ https://www.wired.com/story/wikipedia-finally-asking-big-tec... My guess is Google's "donation" is pennies on the dollar from what they benefit from Wikipedia, and more of a token gesture than anything else.
mthoms 5y ago
It's just a redirect to Google.
- decrypt 5y ago
  There seem to be two different features: 1. Redirection to Google. 2. Piping Google's results back to Brave Search's UI: https://news.ycombinator.com/item?id=27594754 https://news.ycombinator.com/item?id=27594754 The original commenter was asking about the latter.
  - mthoms 5y ago
    Ah, I see, Thanks for the heads up.

Is that legal?

Why wouldn't it be? Google scrapes the web to populate it's results, why wouldn't other search engines scrape the web as well? Google is a website.

xpe 5y ago
This is naive. Web sites have various terms of service.
- therein 5y ago
  Violating a website's ToS is hardly illegal, though.
  - xpe 5y ago
    > Violating a website's ToS is hardly illegal, though. Can you be more specific as to your claim? Not illegal in what sense(s)? And what is your basis for the claim? I'm not a lawyer, but saying "violating a website's ToS is hardly illegal" is fraught advice. While individuals may get some leeway when it comes to ToS violations (see [1] and [2]), I would expect companies scraping and/or extracting content would be treated differently. [1]: https://www.eff.org/deeplinks/2010/07/court-violating-terms-service-not-crime-bypassing https://www.eff.org/deeplinks/2010/07/court-violating-terms-... [2]: https://arstechnica.com/tech-policy/2020/03/court-violating-a-sites-terms-of-service-isnt-criminal-hacking/ https://arstechnica.com/tech-policy/2020/03/court-violating-... [3]: https://www.octoparse.com/blog/10-myths-about-web-scraping https://www.octoparse.com/blog/10-myths-about-web-scraping [4]: https://www.law.com/newyorklawjournal/almID/1202610687621/?slreturn=20210523014130 https://www.law.com/newyorklawjournal/almID/1202610687621/?s...
    - Kiro 5y ago
      The only way for it to be illegal is if they're breaking a law, but then it would be illegal regardless of what the ToS says.
lern_too_spel 5y ago
Because Google's robots.txt disallows it, and those websites allow it.
- smsm42 5y ago
  robots.txt is not a legal contract. It's just a convention to express the wishes of the site author, but there's no legal obligation to follow these wishes.
  - lern_too_spel 5y ago
    It does indicate that those other sites want Google to scrape them, while Google does not want others to scrape their results, which is an important distinction ocdtrekkie ignored for whether the scrapee will want to take legal action.
    - ocdtrekkie 5y ago
      You may wish to review https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l... Google Search results are definitely "public data" so long as Google provides them to anyone who asks.
      - lern_too_spel 5y ago
        Then why does Startpage pay Google and DDG pay Microsoft?
        ocdtrekkie 5y ago
        While scraping search results isn't illegal, by any means, it's also not illegal for Google or Microsoft to block requests they believe are from competing search engines. Presumably the cost of paying them is less than the cost of hiring engineers to constantly try to find new ways to outwit Google and Microsoft engineers. Again, if scraping data from websites without permission, Google simply wouldn't exist. Bear in mind, robots.txt is a feature that Google and Microsoft choose to respect, but the default assumption search engines have made from the beginning, is that they are free to grab whatever they want from the web, unless you ask them otherwise to please not.
SamBam 5y ago
It's not scraping static text in order to point you to those sites, it's using the features of the site to perform a service better than you can do yourself. It's completely different. If I made a site that claimed to help you with your math homework and simply sent the queries to WolframAlpha, that would also not just be "scraping."
- ocdtrekkie 5y ago
  This is basically what Google does to Wikipedia and rebrands as "Knowledge Graph".
  - SamBam 5y ago
    Google has donated many millions to the Wikimedia Foundation, basically in payment for this. (But, again, there's a difference between scraping and echoing a request on another site and waiting for its response. The latter is basically unauthorized use of its API, not scraping.)
    - ocdtrekkie 5y ago
      It'll be very telling if Google switches to using Wikipedia Enterprise: https://www.wired.com/story/wikipedia-finally-asking-big-tech-to-pay-up/ https://www.wired.com/story/wikipedia-finally-asking-big-tec... My guess is Google's "donation" is pennies on the dollar from what they benefit from Wikipedia, and more of a token gesture than anything else.

mthoms 5y ago

It's just a redirect to Google.

decrypt 5y ago
There seem to be two different features: 1. Redirection to Google. 2. Piping Google's results back to Brave Search's UI: https://news.ycombinator.com/item?id=27594754 https://news.ycombinator.com/item?id=27594754 The original commenter was asking about the latter.
- mthoms 5y ago
  Ah, I see, Thanks for the heads up.