4 ms·
Why do people want to block LLM crawlers? Everyone wants to be visible in GPT, Claude, and others, don’t they?
by bakigul 11mo ago
Why do people want to block LLM crawlers? Everyone wants to be visible in GPT, Claude, and others, don’t they?
- aiven 11mo ago- If LLM knows about your content, people don't really need to visit your site - LLM crawlers can be pretty aggressive and eat up a lot of traffic - Google will not suggest its misleading "summaries" from its search for your site - Some people just hate LLMs that much ¯\_(ツ)_/¯
- saltysalt 11mo agoExactly! With many people moving to AI-driven search, if you are not in their indexes you traffic will reduce even further. And I say this as someone who built a search engine with no AI: I know my audience for that service is very niche, with the vast majority of people using AI search because it's more convenient.
- suckler 11mo agoYou're a funny guy.
- ThomasMidgley 11mo agoNo
- bo1024 11mo agoOne issue is just the sheer amount of such traffic.
- xgulfie 11mo agoIf someone doesn't want to visit my website then why would I provide my website to them