3 ms·
There are two common misconceptions in this post. The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when t
by simonw 1y ago
There are two common misconceptions in this post.
The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots
Since LLM skeptics frequently characterize all LLM vendors as dishonest mustache-twirling cartoon villains there's little point trying to convince them that companies sometimes actually do what they say they are doing.
The bigger misconception though is the idea that LLM training involves indiscriminately hoovering up every inch of text that the lab can get hold of, quality be damned. As far as I can tell that hasn't been true since the GPT-3 era.
Building a great LLM is entirely about building a high quality training set. That's the whole game! Filtering out garbage articles full of spelling mistakes is one of many steps a vendor will take in curating that training data.
- charles_f 1y agoI somewhat agree with your viewpoint on copyright, but what terrifies me is VCs like a16z or Sequoia simultaneously backing up large LLMs profiting from ignoring copyright and media firms where they'll use whatever power and lobby they have to protect copyright. I don't think the content I produce is worth that much, I'm glad if it can serve anyone, but I find amusing the idea to poison the well
- vintermann 1y agoThere are definitively scrapers that ignore your robots.txt file. Whether they're some "Enemy State" LLM outfit, an "Allied State" corporation outsourcing their dirty work a step or two, or just some data hoarder worried that the web as we know it is going away soon, everyone is saying they're a problem lately, I don't think everyone is lying. But it's certainly also true that anyone feeding the scrapings to an LLM will filter it first. It's very naive of this author to think that his adlib-spun prose won't get detected and filtered out long before it's used for training. Even the pre-LLM internet had endless pages of this sort of thing, from aspiring SEO spammers. Yes, you're wasting a bit of the scraper's resources, but you can bet they're already calculating in that waste.
- simonw 1y agoThere are definitively scrapers that ignore your robots.txt file Of course. But those aren't the ones that explicitly say "here is how to block us in robots.txt" The exact quote from the article that I'm pushing back on here is: "If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality." Which appears directly below this: User-agent: GPTBot Disallow: /
- spacebuffer 1y agoYour initial comment made sense after reading through the openai docpage. so I opened up my site to add those to robots.txt, turns out I had added all 3 of those user-agents to my robots file [0], out of curiosity I asked chatgpt about my site and it did scrape it, it even mentioned articles that have been published after adding the robots file [0]: https://yusuf.fyi/robots.txt https://yusuf.fyi/robots.txt
- simonw 1y agoCan you share that ChatGPT transcript? One guess: ChatGPT uses Bing for search queries, and your robots.txt doesn't block Bing. If that's what is happening here I agree that this is really confusing and should be clarified by the OpenAI bots page.
- spacebuffer 1y agoHere you go: https://chatgpt.com/share/68bc125a-9e9c-8005-9c9f-298dbd541d3f https://chatgpt.com/share/68bc125a-9e9c-8005-9c9f-298dbd541d...
- simonw 1y agoYeah, wow that's a lot of information for a site that's supposedly blocked using robots.txt! My best guess is that this is a Bing thing - ChatGPT uses Bing as their search partner (though they don't make that very obvious at all), and BingBot isnt't blocked by your site. I think OpenAI need to be a whole lot more transparent about this. It's very misleading to block their crawlers and have it not make any difference at all to the search results returns within ChatGPT.
- Retric 1y ago> The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it: That’s testable and you can find content “protected” by robots.txt regurgitated by LLM’s. In practice it doesn’t matter if that’s through companies lying or some 3rd party scraping your content and then getting scraped.
- simonw 1y agoDo you have an example that demonstrates that?
- whilenot-dev 1y agoUser Agent "Perplexity‑User"[0]: > Since a user requested the fetch, this fetcher generally ignores robots.txt rules. [0]: https://docs.perplexity.ai/guides/bots https://docs.perplexity.ai/guides/bots
- nine_k 1y agoBut this is more like `curl https://some/url/...` ignoring robots.txt. Crawlers are the thing that should honor robots.txt, "nofollow", etc.
- whilenot-dev 1y agoThe title of this site is "Perplexity Crawlers"... And it clearly voids simonw's stance > it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it
- simonw 1y agoWhere did Perplexity say they would obey robots.txt? They explicitly document that they do not obey robots.txt for that form of crawling (user-triggered, data is not gathered for training.) Their documentation is very clear: https://docs.perplexity.ai/guides/bots https://docs.perplexity.ai/guides/bots
- CrossVR 1y ago> Since LLM skeptics frequently characterize all LLM vendors as dishonest mustache-twirling cartoon villains there's little point trying to convince them that companies sometimes actually do what they say they are doing. Even if the large LLM vendors respect it, there's enough venture capital going around that plenty of smaller vendors are attempting to train their own LLMs and they'll take every edge they can get, robots.txt be damned.
- simonw 1y agoYeah this is definitely true.
- flir 1y ago> The first isn't worth arguing against: it's the idea that LLM vendors ignore your robots.txt file even when they clearly state that they'll obey it: https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots So, uh... where's all the extra traffic coming from?
- hooloovoo_zoo 1y agoYour link does not say they will obey it.
- simonw 1y agoDirect quote from https://platform.openai.com/docs/bots https://platform.openai.com/docs/bots "OpenAI uses the following robots.txt tags to enable webmasters to manage how their sites and content work with AI." Then for GPT it says: "GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models." What are you seeing here that I'm missing?
- hooloovoo_zoo 1y agoMy read is that they are describing functionality for site owners to provide input about what the site owner thinks should happen. OpenAI is not promising that is what WILL happen, even in the narrow context of that specific bot.
- rozab 1y agoAfter I set up a self hosted git forge a little while ago, I found that within minutes it immediately got hammered by OpenAI, Anthropic, etc. They were extremely aggressive, grabbing every individual file from every individual commit, one at a time. I hadn't backlinked the site anywhere and was just testing, so I hadn't thought to put up a robots.txt. They must have found me through my cert registration. After I put up my robots.txt (with explicit UA blocks instead of wildcards, I heard some ignore them), I found after a day or so the scraping stopped completely. The only ones I get now are vulnerability scanners, or random spiders taking just the homepage. I know my site is of no consequence, but for those claiming OpenAI et al ignore robots.txt I would really like to see some evidence. They are evil and disrespectful and I'm gutted they stole my code for profit, but I'm still sceptical of these claims. Cloudflare have done lots of work here and have never mentioned crawlers ignoring robots.txt: https://blog.cloudflare.com/control-content-use-for-ai-training/ https://blog.cloudflare.com/control-content-use-for-ai-train...