3 ms·
Remember robots.txt? Now we have llms.txt
- deleted 2y ago[deleted]
- p_ing 2y agoThis is not a 'web standard', but rather a proposal (to who?). This does not operate like robots.txt, which would allow site administrators to tell the LLM to f-off. Instead, it "guides" LLMs through your site for better contextual information (i.e., documentation). The official site is at https://llmstxt.org/ https://llmstxt.org/.
- mdhb 2y agoAnd here I am writing custom middleware to fuck with LLMs the moment they try to engage with my websites content.
- LinuxBender 2y agoI do remember robots.txt and have always considered it a dumb idea. Bots will do whatever they can do or whatever their shady operators tell them to do. Anything I want hidden from bots requires authentication. The more friction the better. Even the "well behaved" bots will ignore robots.txt depending on circumstances such as scanning for malware in email links. LLM's will just be getting goatse and tubgirl (Don't google that).