33 ms·
I wonder if it will actually respect the robots.txt this time.
by NBJack 2y ago
I wonder if it will actually respect the robots.txt this time.
- micromacrofoot 2y agoalmost no one does, robots.txt is practically a joke at this point — right up there with autocomplete=off
- Demiurge 2y agoIn what circles is it a joke? Google bots seem to respect it on my sites according to logs.
- otikik 2y agoApparently, the regular search crawler does it, but the ai thingie doesn't.
- supriyo-biswas 2y agoHuh? You can add Google-Extended[1] to opt out from Generative AI summaries. [1] https://blog.google/technology/ai/an-update-on-web-publisher-controls/ https://blog.google/technology/ai/an-update-on-web-publisher...
- micromacrofoot 2y agoGoogle will still scrape it for training data either way, this only impacts search results.
- supriyo-biswas 2y ago> Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help *improve Bard and Vertex AI generative APIs*, including future generations of models that power those products.
- micromacrofoot 2y agohttps://www.theverge.com/news/630079/openai-google-copyright-fair-use-exception https://www.theverge.com/news/630079/openai-google-copyright... they're literally asking to break laws to train AI for national security. A sentence in a press release from 2 years ago is worthless... look at what they're actually doing
- lucgagan 2y agoCan confirm. My website is flooded with AI bots despite attempts to block crawlers to certain parts of it.
- mediumsmart 2y agoI know an artist that had noindex turned on by mistake in robots.txt for the last 5 years - google, kagi and duckduckgo find tons of links relevant to the artist and the artwork but not a single one from the website. so not seem to or apparently but matter of fact like. robots.txt works for the intended audience
- joecool1029 2y agoIt's in a small circle of those that do. Blame the internet archive for starting this trend: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- hirako2000 2y agoGiven websites do disappear or worse, get their content adultered. Also given the long history of the internet archive as a non profit - and the commons service it has served so far, the joke would be to see that bot honor it.
- nikisweeting 2y agolol IA did not start that, if anything they were late to the game. only the top handful of US-based search engines ever bothered respecting it in the first place
- wuming2 2y agoSorry to intrude with something unrelated. But YC closed the earlier discussion. Saw your comment about Kannel WAP of few months back and wanted to ask if do you know of any WAP Push full service provider still in operation.
- joecool1029 2y agoNah I just know about that public gateway I linked. I can't use it anymore as 2G was shut down on my local towers back in January.
- micromacrofoot 2y agoA small number of search engines respect it, no one else does. Just about every content scraping bot ignores it, including a number of Google's.
- geekrax 2y agoI have replaced all robots.txt rules with simple WAF rules, which are cheaper to maintain than dealing with offending bots.
- Joe_Cool 2y agoFor my personal stuff I also included a Nepenthes tarpit. Works great and slows the bots down while feeding them garbage. Not my fault when they consume stuff robots.txt says they shouldn't. I'm just not sure if legal would love me doing that on our corporate servers...
- rustc 2y agoThe WAF rule matches based on the user agent header? Perplexity is known to use generic browser user agents to bypass that.
- claudiulodro 2y agoI do essentially both: robots.txt backed by actual server-level enforcement of the rules in robots.txt. You'd think there would be zero hits on the server-level blocking since crawlers are supposed to read and respect robots.txt, but unsurprisingly they don't always. I don't know why this isn't a standard feature in web hosting.
- explain 2y agorobots.txt is meant for automated crawlers, not human-driven actions.
- zupa-hu 2y agoEvery automated crawler follows human-driven actions.
- gopher_space 2y agoWelcome to "Context".
- josh-sematic 2y agoConversely, every browser is a program that automatically executes HTTP requests.
- bayindirh 2y agoYet they respect a lot of things meant for machine to machine interaction. Like server return codes, cookie negotiations, and CAPTCHAs if they behave a certain way. So they sometimes hit bollards and turnstiles made for other types of code which executes HTTP requests. So they're bots basically, but better (or suitably) behaving ones.
- soulofmischief 2y agoBrowsers let you visit websites without regard for robots.txt.
- nicce 2y agoIt must form the search index somehow. That is prior the human action. Simply it would not find the page at all if it respects.
- pests 2y agoI remember in late 90s/early 2000 as a teen going to robots.txt to specifically see what they were trying to hide and exploring those urls. What is the difference if I use a browser or a LLM tool (or curl, or wget, etc) to make those requests?
- _2d30 2y agoI don't think it should. If a user asks the AI to read the web for them, it should read the web for them. This isn't a vacuum charged with crawling the web, it's an adhoc GET request.
- deleted 2y ago[deleted]
- internetter 2y agoYou could make this justification for a lot of unapproved bot activity.
- taskforcegemini 2y agoyou could, but this article is about claude.
- bayindirh 2y agoHow can you be so sure? Processors love locality, so they fetch the data around the requested address. Intel even used to give names to that. So, similarly, LLM companies can see this as a signal to crawl to whole site to add to their training sets and learn from it, if the same URL is hit for a couple of times in a relatively short time period.
- mvdtnz 2y agoNo thank you, when I define a robots.txt file I expect all automated systems to respect it.
- beeflet 2y agoSomeone should call the robots.txt police then, there's a bandit on the loose!
- TheDudeMan 2y agoBut this isn't automated. This is user-driven.
- mtkd 2y agoDo really think LLM vendors that download 80TB+ of data over torrents are going to be labeling their crawler agents correctly and running them out of known datacenters?
- teh_infallible 2y agoApparently they use smart appliances to scrape websites from residential accounts.
- Arnt 2y agoThe ones I noticed in my logfiles behave impeccably: retrieve robots.txt every week or so and act on it. (I noticed Claude, OpenAI and a couple of others whose names were less familiar to me.)
- tylersmith 2y agoIt's a user agent not a robot.
- Y_Y 2y agoWhy not both?
- SoftTalker 2y agoMaybe we need a new "ai.txt" that says "yes I mean you, ChatGPT et. al."
- jsheard 2y agoIf they don't comply with robots.txt, why would they comply with anything else?
- JadoJodo 2y agohttps://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
- verdverm 2y agoBluesky / ATProto has a proposal for User Intents for data. More semantics than robots.txt, but equally unenforceable. Usage with AI is one of the intents to be signaled by users https://github.com/bluesky-social/proposals/tree/main/0008-user-intents https://github.com/bluesky-social/proposals/tree/main/0008-u...
- whoami_nr 2y agoSmall difference. Its called llms.txt https://llmstxt.org/ https://llmstxt.org/
- noddleah 2y agohttps://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ https://thelibre.news/foss-infrastructure-is-under-attack-by...
- NewJazz 2y agoWhy wonder. You can test for yourself.
- furyofantares 2y agoPresumably the crawler that produces whatever index it uses does, which is how it knows what sites to read. Unless you provide it a URL yourself I guess, in which case, it shouldn't.
- JimDabell 2y agoThe LLM shouldn’t. robots.txt is intended to control recursive fetches. It is not intended to block any and all access. You can test this out using wget. Fetch a URL with wget. You will see that it only fetches that URL. Now pass it the --recursive flag. It will now fetch that URL, parse the links, fetch robots.txt, then fetch the permitted links. And so on. wget respects robots.txt. But it doesn’t even bother looking at it if it’s only fetching a single URL because it isn’t acting recursively, so robots.txt does not apply. The same applies to Claude. Whatever search index they are using, the crawler for that search index needs to respect robots.txt because it’s acting recursively. But when the user asks the LLM to look at web results, it’s just getting a single set of URLs from that index and fetching them – assuming it’s even doing that and not using a cached version. It’s not acting recursively, so robots.txt does not apply. I know a lot of people want to block any and all AI fetches from their sites, but robots.txt is the wrong mechanism if you want to do that. It’s simply not designed to do that. It is only designed for crawlers, i.e. software that automatically fetches links recursively.
- manquer 2y agoWhile robots.txt is not there to directly prevent automated requests, it does prevent crawling which is needed for search indices. Without recursive crawling, it will not possible for a engine to know what are valid urls[1]. They will otherwise either have to brute-force say HEAD calls for all/common string combinations and see if they return 404s or more realistically have to crawl the site to "discover" pages. The issue of summarizing specific a URL on demand is a different problem[2] and not related to issue at hand of search tools doing crawling at scale and depriving all traffic Robots.txt does absolutely apply to LLMs engines and search engines equally. All types of engines create indices of some nature (RAG, Inverted Index whatever) by crawling, sometimes LLM enginers have been very aggressive without respecting robots.txt limits, as many webmasters have reported over the last couple of years. --- [1] Unless published in sitemap.xml of course. [2] You need to have the unique URL to ask the llm to summarize in the first place, which means you likely visited the page already, while someone sharing a link with you and a tool automatically summarizing the page deprives the webmaster of impressions and thus ad revenue or sales. This is common usage pattern in messaging apps from Slack to iMessages and been so for a decade or more, also in news aggregators to social media sites, and webmasters have managed to live with this one way or another already.