4 ms·
Does it work though? The big LLM crawlers do not read llms.txt so will they read and follow the same instructions as HTML?
by graemep 5mo ago
Does it work though? The big LLM crawlers do not read llms.txt so will they read and follow the same instructions as HTML?
- prismlfx 5mo agoWhere did you see the big crawlers don't read it? Anthropic does.. they're pretty big.
- graemep 5mo agoIf I look at the logs of sites that Anthropic crawls (and in some cases references in answers) and there are no requests for llms.txt
- mapcars 5mo agoI have no idea, in theory it might catch some miss-configured agents off-guard
- Aboutplants 5mo agoSomeone has to have done or is doing an experiment with this right? I also think that if it was an actual profitable thing then we would know about it pretty quickly. It would pop up everywhere
- iamacyborg 5mo agoApparently new checks in Chrome Lighthouse are checking for the existence of the file. https://searchengineland.com/google-llms-txt-chrome-lighthouse-478246 https://searchengineland.com/google-llms-txt-chrome-lighthou...
- patwards 5mo agoYeah I want to know how many donations they get
- flexagoon 5mo agoThis is not for LLM data crawlers, it's for LLM agents the end users are running on their machines