2 ms·
Sadly true. That's why I gave up on robots.txt years ago and started blocking crawlers outright in .htaccess Of course, that became unsustainable so now I have
by JohnFen 2y ago
Sadly true. That's why I gave up on robots.txt years ago and started blocking crawlers outright in .htaccess
Of course, that became unsustainable so now I have everything behind a login wall.