4 ms·

Probably a silly question, but why not just use robots.txt? That was designed for preventing exactly this.

asutekku 6y ago
I’d say most of Genius’ visitors comes from the “song x lyrics” so hiding those with robots would ultimately make them lose almost all of their traffic.
Polylactic_acid 6y ago
robots.txt is designed to keep garbage off search results. It has absolutely no power to prevent a bot to do anything. Also if the site added robots.txt they might as well shut down because their entire userbase comes from people searching lyrics on google.
- mikemotherwell 6y ago
  Other way around. It was invented to stop crawling. Indexing is still technically allowed even when blocked by robots.txt from crawling.
jtxx 6y ago
robots.txt isn’t enforced by anything
dewey 6y ago
Not due to robots.txt but you can see what happens to genius formerly rapgenius when they get removed from the index: https://techcrunch.com/2013/12/25/google-rap-genius/ https://techcrunch.com/2013/12/25/google-rap-genius/
- encom 6y ago
  Wear a condom before clicking techcrunch.com links: https://archive.ph/9eUkv https://archive.ph/9eUkv
  - jackdh 6y ago
    Outline works well for TC and provides a nicer reading experience. https://outline.com/https://techcrunch.com/2013/12/25/google-rap-genius/ https://outline.com/https://techcrunch.com/2013/12/25/google...
  - icebraining 6y ago
    To be fair to TC, if you disable JavaScript you get a pretty good experience - just the full article, legible. Not like those sites that require JS to load the text and/or images.
    - wizzwizz4 6y ago
      But you need JavaScript to get past the "we value your privacy, so give us permission to sell it" banner.
AgloeDreams 6y ago
The problem is that Google is stealing content and placing it on search so the user never goes to the source, By blocking it with robots they block themselves from google results AND Google may already keep scraping the content.

I’d say most of Genius’ visitors comes from the “song x lyrics” so hiding those with robots would ultimately make them lose almost all of their traffic.

Polylactic_acid 6y ago

robots.txt is designed to keep garbage off search results. It has absolutely no power to prevent a bot to do anything. Also if the site added robots.txt they might as well shut down because their entire userbase comes from people searching lyrics on google.

mikemotherwell 6y ago
Other way around. It was invented to stop crawling. Indexing is still technically allowed even when blocked by robots.txt from crawling.

jtxx 6y ago

robots.txt isn’t enforced by anything

dewey 6y ago

Not due to robots.txt but you can see what happens to genius formerly rapgenius when they get removed from the index: https://techcrunch.com/2013/12/25/google-rap-genius/ https://techcrunch.com/2013/12/25/google-rap-genius/

encom 6y ago
Wear a condom before clicking techcrunch.com links: https://archive.ph/9eUkv https://archive.ph/9eUkv
- jackdh 6y ago
  Outline works well for TC and provides a nicer reading experience. https://outline.com/https://techcrunch.com/2013/12/25/google-rap-genius/ https://outline.com/https://techcrunch.com/2013/12/25/google...
- icebraining 6y ago
  To be fair to TC, if you disable JavaScript you get a pretty good experience - just the full article, legible. Not like those sites that require JS to load the text and/or images.
  - wizzwizz4 6y ago
    But you need JavaScript to get past the "we value your privacy, so give us permission to sell it" banner.

AgloeDreams 6y ago

The problem is that Google is stealing content and placing it on search so the user never goes to the source, By blocking it with robots they block themselves from google results AND Google may already keep scraping the content.