2 ms·
Can we get an AI-powered tarpit for these crawlers? Add a hidden link, put it in robots.txt A crawler hits that link, a light-on-resources language model prod
by theshrike79 1y ago
Can we get an AI-powered tarpit for these crawlers?
Add a hidden link, put it in robots.txt
A crawler hits that link, a light-on-resources language model produces infinite amounts of plausible-looking gibberish for them to crawl with links and everything.