4 ms·
Here is the question: Does Google has a right to block you? I believe they do, it is their API afterall. In Genius's case, does it disallow Google for scraping
by tanilama 6y ago
Here is the question: Does Google has a right to block you? I believe they do, it is their API afterall.
In Genius's case, does it disallow Google for scraping?
From their robots.txt, I can't tell:
https://genius.com/robots.txt https://genius.com/robots.txt
- elliekelly 6y agoYou shouldn’t need a law degree to interpret a robots.txt though. If that’s where we’re at then we as a society need to re-evaluate.
- tanilama 6y agoWhy? Genius is given a way out to prevent itself from being scrapped. But it doesn't. Mean they probably value traffic from Google. Which means they are only not OK with scraping when Google uses the scrapped for purpose they deemed reduced their traffic
- DangitBobby 6y ago> Which means they are only not OK with scraping when Google uses the scrapped for purpose they deemed reduced their traffic Obviously. The way the profit model for the internet works right now, for sites to coexist with Google, they must actually receive some of the traffic that is generated from searches matching their content. Who would be okay with having all of their content scraped with the result being that they get none of the traffic and thus the monetary benefit from the work they do?
- icebraining 6y agoIt's not Genius' content, though. It's the artists'. If it was their content, they could simply sue for copyright infringement.
- rootsudo 6y agoI wonder who that mecha guy is that he has his own rule in robots.txt https://genius.com/mecha https://genius.com/mecha
- why_only_15 6y agoMaybe mecha is intended to be a robot user? Like the deleted user on Stack Overflow? In that case it might have lots of garbage contributions.