3 ms·Which, just like robots.txt, worked decently well until the AI bot crawlers overwhelmed all else.by Quarrel 1y agoWhich, just like robots.txt, worked decently well until the AI bot crawlers overwhelmed all else.