3 ms·
Everyone ignores robots.txt because every robots.txt blocks everyone.
by inigyou 2mo ago
Everyone ignores robots.txt because every robots.txt blocks everyone.
- nubinetwork 2mo agoI allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
- inigyou 2mo agoThink about incentives, what benefit does Anthropic obtain by going to hell?