3 ms·
AI scrapers are pretty widely ignoring robots.txt, and plenty lie about their user agents. https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/
by ceejayoz 2y ago
AI scrapers are pretty widely ignoring robots.txt, and plenty lie about their user agents. https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/ https://rknight.me/blog/perplexity-ai-is-lying-about-its-use...
I'd fully expect OpenAI to do some checks that their bot isn't getting different responses than a seemingly real request.