4 ms·
They use different user-agent strings. The crawlers obfuscate themselves and use residential proxies. The agents call themselves ChatGPT-User. Of course Cloudfl
by tardedmeme 6mo ago
They use different user-agent strings. The crawlers obfuscate themselves and use residential proxies. The agents call themselves ChatGPT-User. Of course Cloudflare wants OpenAI to pay them for not blocking ChatGPT-User by default.
- faangguyindia 6mo agoIt's true, crawlers used for AI training don't say they are crawlers at all.