5 ms·

Does Google hold the copyright on its search results? Why can't I scrape Google?

tanilama 6y ago
Here is the question: Does Google has a right to block you? I believe they do, it is their API afterall. In Genius's case, does it disallow Google for scraping? From their robots.txt, I can't tell: https://genius.com/robots.txt https://genius.com/robots.txt
- elliekelly 6y ago
  You shouldn’t need a law degree to interpret a robots.txt though. If that’s where we’re at then we as a society need to re-evaluate.
  - tanilama 6y ago
    Why? Genius is given a way out to prevent itself from being scrapped. But it doesn't. Mean they probably value traffic from Google. Which means they are only not OK with scraping when Google uses the scrapped for purpose they deemed reduced their traffic
    - DangitBobby 6y ago
      > Which means they are only not OK with scraping when Google uses the scrapped for purpose they deemed reduced their traffic Obviously. The way the profit model for the internet works right now, for sites to coexist with Google, they must actually receive some of the traffic that is generated from searches matching their content. Who would be okay with having all of their content scraped with the result being that they get none of the traffic and thus the monetary benefit from the work they do?
      - icebraining 6y ago
        It's not Genius' content, though. It's the artists'. If it was their content, they could simply sue for copyright infringement.
- rootsudo 6y ago
  I wonder who that mecha guy is that he has his own rule in robots.txt https://genius.com/mecha https://genius.com/mecha
  - why_only_15 6y ago
    Maybe mecha is intended to be a robot user? Like the deleted user on Stack Overflow? In that case it might have lots of garbage contributions.
adventured 6y ago
Because they'll block you. You can prevent Google from indexing your content using robots.txt (Google has a robots.txt on its site as well). You don't have a right to access their service as many times as you want to, eg by automated means, although you can attempt it. Flip a coin on whether they sue to stop you if you become too annoying. The Genius complaint is essentially that they want to be represented in Google search without having Google take lyrics from their service and use them in their own served-up content snippets (making a sizable part of the value of genius.com void). Genius knows Google can get lyrics elsewhere if they have to, the lawsuit is probably out of spite due to past conflict with Google and their annoyance at Google competing with them in a shady way (Google was de facto using Genius's service to reduce the value of Genius).

Here is the question: Does Google has a right to block you? I believe they do, it is their API afterall. In Genius's case, does it disallow Google for scraping? From their robots.txt, I can't tell: https://genius.com/robots.txt https://genius.com/robots.txt

elliekelly 6y ago
You shouldn’t need a law degree to interpret a robots.txt though. If that’s where we’re at then we as a society need to re-evaluate.
- tanilama 6y ago
  Why? Genius is given a way out to prevent itself from being scrapped. But it doesn't. Mean they probably value traffic from Google. Which means they are only not OK with scraping when Google uses the scrapped for purpose they deemed reduced their traffic
  - DangitBobby 6y ago
    > Which means they are only not OK with scraping when Google uses the scrapped for purpose they deemed reduced their traffic Obviously. The way the profit model for the internet works right now, for sites to coexist with Google, they must actually receive some of the traffic that is generated from searches matching their content. Who would be okay with having all of their content scraped with the result being that they get none of the traffic and thus the monetary benefit from the work they do?
    - icebraining 6y ago
      It's not Genius' content, though. It's the artists'. If it was their content, they could simply sue for copyright infringement.
rootsudo 6y ago
I wonder who that mecha guy is that he has his own rule in robots.txt https://genius.com/mecha https://genius.com/mecha
- why_only_15 6y ago
  Maybe mecha is intended to be a robot user? Like the deleted user on Stack Overflow? In that case it might have lots of garbage contributions.

adventured 6y ago

Because they'll block you. You can prevent Google from indexing your content using robots.txt (Google has a robots.txt on its site as well). You don't have a right to access their service as many times as you want to, eg by automated means, although you can attempt it. Flip a coin on whether they sue to stop you if you become too annoying. The Genius complaint is essentially that they want to be represented in Google search without having Google take lyrics from their service and use them in their own served-up content snippets (making a sizable part of the value of genius.com void). Genius knows Google can get lyrics elsewhere if they have to, the lawsuit is probably out of spite due to past conflict with Google and their annoyance at Google competing with them in a shady way (Google was de facto using Genius's service to reduce the value of Genius).