2 ms·
In light of the recent surge in scraped content trumping original content on Google search, this is so true... people who scrape your site do not care about you
by tomxor 4y ago
In light of the recent surge in scraped content trumping original content on Google search, this is so true... people who scrape your site do not care about your preferences in a txt file.
- stark98 4y agoYeah, I feel like it only discourages legitimate bots, but for malicious ones it's a big red sign saying "SENSITIVE CONTENT HERE, SCRAPE IT"