4 ms·
It's telling that nearly every site listed is directly involved with AI in some way, unsurprisingly the broader internet isn't particularly interested in going
by jsheard 2y ago
It's telling that nearly every site listed is directly involved with AI in some way, unsurprisingly the broader internet isn't particularly interested in going out of its way to make its content easier for AI companies to scrape.
Deliberately putting garbage data in your llms.txt could be funny though.
- KTibow 2y agoI've seen many people joke about intentionally poisoning training data but has that ever worked?
- jsheard 2y agoIt's hard to gauge the effectiveness of poisoning huge training sets since anything you do is a figurative drop in the ocean, but if you can poison the small amount of data that an AI agent requests on-the-fly to use with RAG then I would guess it's much easier to derail it.
- nyrikki 2y agoThis study shows that controlling 0.1% may be enough. https://arxiv.org/abs/2410.13722v1 https://arxiv.org/abs/2410.13722v1 I have noticed some popular copied but incorrect leetcode examples leaking into the dataset. I suspect it depends on domain specificity, but that seems within the ability of an SEO spammer or decentralized group of individuals.
- ALittleLight 2y agoSeems silly to put garbage data there. Like intentionally doing bad SEO so Google doesn't link you. I think you should think about it as: I want the LLM to recognize my site as a high quality resource and direct traffic to me. Imagine user asks ChatGPT a question. LLM has scrapped your website and answers the question. User wants some kind of follow up - read more, what's the source, how can I buy this, whatever - so the LLM links the page it got the data from. LLMs seem like they're supplanting search. Being early to work with them is an advantage. Working to make your pages look low quality seems like an odd choice.
- deleted 2y ago[deleted]
- vouaobrasil 2y agoI'd prefer not to play that game. I'd rather lose a bit of money and traffic and not help LLMs as far as humanly possible.
- shreddit 2y agoThat sounds like those “react youtubers” taking your content without permission and telling you that you should be grateful for the exposure.
- ALittleLight 2y agoIs it a good response to the react YouTubers to make your content terrible? Or to provide something in your content not available on their? Whether you like it or not LLMs are going to be how people explore the web. They simply work better than search engines - not least because they can quickly scan numerous sites simultaneously, consume and synthesize the content. You can choose to sabotage your own content in a likely futile effort to make things worse for LLM users if you want - my point is just that it serves no purpose and misses out on the opportunities in front of you.
- freeone3000 2y agoAre you kidding? The follow-through on attribution links, where present, is nearly zero. There’s no gains to be had here, only losses.
- spencerchubb 2y agoYou seem to be misunderstanding why a website would make llms.txt Obviously, they would not make it just for an AI company to scrape Here's an example. Let's say I run a dev tools company, and I want users to be able to find info about me as easily as possible. Maybe a user's preferred way of searching the web is through a chatbot. If that chatbot also uses llms.txt, it's easy for me to deliver the info, and easy for them to consume. Win-win Of course adoption is not very widespread, but such is the case for every new standard.
- ppqqrr 2y agoThe point of LLMs is they are able to make sense of the web the same way humans can (roughly speaking); so why do they get the special treatment of having direct, ad-free, plain text version of the actual info they’re looking for, while humans aren’t allowed to scroll through a salad recipe without being bombarded with 20 ads?
- spencerchubb 2y agoA human could read the llms.txt if they want to. And a developer could put ads in llms.txt if they wanted to!