4 ms·
Has anyone worked on anything like this for AI scrapers?
by orliesaurus 2y ago
Has anyone worked on anything like this for AI scrapers?
- dartos 2y agoA robots.txt that asks AI scrapers not to scrape? There’s a couple services that keep updated lists of known scraper user agents. A quick search reveals a handful.
- zorked 2y agohttps://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
- gnaman 2y agohttps://llmstxt.org/ https://llmstxt.org/ https://www.answer.ai/posts/2024-09-03-llmstxt.html https://www.answer.ai/posts/2024-09-03-llmstxt.html