3 ms·
> The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it: https://p
by flir 1y ago
> The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots
So, uh... where's all the extra traffic coming from?