13 ms·
Do you also block all google bots? Who's to say your data isn't scraped by the company but buys it from brokers who can? Robots.txt is theater.
by seanthemon 2y ago
Do you also block all google bots? Who's to say your data isn't scraped by the company but buys it from brokers who can? Robots.txt is theater.