23 ms·

Robots.txt is a suicide note (2011)

88 points by rafram 1y ago

1vuio0pswjnm7 1y ago
Every website operator should be required to learn _rate-limiting_ as well as how to create a robots.txt file What matters is rate not "user-agent" User-agent can be fake but rate is real
rafram 1y ago
I think this is kind of misguided - it ignores the main reason sites use robots.txt, which is to exclude irrelevant/old/non-human-readable pages that nevertheless need to remain online from being indexed by search engines - but it's an interesting look at Archive Team's rationale.
- xp84 1y ago
  Yes, and I'd add to that dynamically-generated URLs of infinite variability which have two separate but equally-important reasons for automated traffic to avoid: 1. You (bot) are wasting your bandwidth, CPU, storage on a literally unbounded set of pages 2. This may or may not cause resource problems for the owner of the site (e.g. Suppose they use Algolia to power search and you search for 10,000,000 different search terms... and Algolia charges them by volume of searches.) The author of this angry rant really seems specifically ticked at some perceived 'bad actor' who is using robots.txt as an attempt to "block people from getting at stuff" but it's super misguided in that it ignores an entire purpose of robots.txt that is not even necessarily adversarial to the "robot." This whole thing could have been a single sentence: "Robots.txt has a few competing vague interpretations and is voluntary; not all bots obey it, so if you're fully relying on it to prevent a site from being archived, that won't work."
  - deleted 1y ago
    [deleted]
- paulddraper 1y ago
  Correct. That has been one of the biggest uses -- improve SEO by preventing web crawlers from getting lost/confused in a maze of irrelevant content.
rolph 1y ago
the archiveteam statements in the article are sure to win special attention, i think this could be footgunning, and .IF archiveteam .THEN script.exe pleasantries.
marbletiles 1y ago
Not sure the emotive language is warranted. Message appears to be “if you use robots.txt AND archive sites honor it AND you are dumb enough to delete your data without a backup THEN you won’t have a way to recover and you’ll be sorry”. It also presumes that dealing with automated traffic is a solved problem, which with the volumes of LLM scraping going on, is simply not true for more hobbyist setups.
- bigbuppo 1y ago
  Or major web properties for that matter.
- paulddraper 1y ago
  > volumes of LLM scraping FWIW I have not seen a reputable report on the % of web scraping in the past 3 years. (Wikipedia being a notable exception...but I would guess Wikipedia to see a far larger increase than anything else.)
  - esseph 1y ago
    It's hard because of attribution, but it absolutely is happening at very high volume. I actually got an alert this morning when I woke up from our monitoring tools that some external sites were being scraped. Happens multiple times a day. A lot of it is coming through compromised residential endpoint botnets.
    - paulddraper 1y ago
      Even without attribution…seeing bot traffic or general traffic increase
  - tolmasky 1y ago
    Wikipedia says their traffic increased roughly 50% [1] from AI bots, which is a lot, sure, but nowhere near the amount where you'd have to rearchitect your site or something. And this checks out, if it was actually debilitating, you'd notice Wikipedia's performance degrade. It hasn't. You'd see them taking some additional steps to combat this. They haven't. Their CDN handles it just fine. They don't even both telling AI bots to just download the tarballs they specifically make available for this exact use case. More importantly, Wikipedia almost certainly represents the ceiling of traffic increase. But luckily, we don't have to work with such coarse estimation, because according to Cloudflare, the total increase from combined search and AI bots in the last year (May 2024 - May 2025), has just been... 18% [2]. The way you hear people talk about it though, you'd think that servers are now receiving DDOS-levels of traffic or something. For the life of me I have not been able to find a single verifiable case of this. Which if you think about it makes sense... It's hard to generate that sort of traffic, that's one of the reasons people pay for botnets. You don't bring a site to its knees merely by accidentally "not making your scraper efficient". So the only other possible explanation would be such a larger number of scrapers simultaneously but independently hitting sites. But this also doesn't check out. There aren't thousands of different AI scrapers out there that in aggregate are resulting in huge traffic spikes [2]. Again, the total combined increase is 18%. The more you look into this accepted idea that we are in some sort of AI scraping traffic apocalypse, the less anything makes sense. You then look at this Anubis "AI scraping mitigator" and... I dunno. The author contends that one if its tricks is that it not only uses JavaScript, but "modern JavaScript like ES6 modules," and that this is one of the ways it detects/prevents AI scrapers [3]. No one is rolling their own JS engine for a scraper such that they are being blocked from their inability to keep up with the latest ECMAScript spec. You are just using an existing JS engine, all of which support all these features. It would actually be a challenge to find an old JS engine these days. The entire things seems to be built on the misconception that the "common" way to build a scraper is doing something curl-esque. This idea is entirely based on the google scraper which itself doesn't even work that way anymore, and only ever did because it was written in the 90s. Everyone that rolls their own scraper these days just uses Puppeteer. It is completely unrealistic to make a scraper that doesn't run JavaScript and wait for the page to "settle down" because so many pages, even blogs, are just entirely client-side rendered SPAs. If I were to write a quick and dirty scraper today I would trivially make it through Anubis' protections... by doing literally nothing and without even realizing Anubis exists. Just using standard scraping practices with Puppeteer. Meanwhile Anubis is absolutely blocking plenty of real humans, with the author for example telling people to turn on cookies so that Anubis can do its job [4]. I don't think Anubis is blocking anything other than humans and Message's link preview generator. I'm investigating further, but I think this entire thing may have started due to some confusion, but want to see if I can actually confirm this before speculating further. 1. https://www.techspot.com/news/107407-wikipedia-servers-struggling-under-pressure-ai-scraping-bots.html https://www.techspot.com/news/107407-wikipedia-servers-strug... (notice the clickbait title vs. the actual contents) 2. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/?utm_source=chatgpt.com/#general-ai-and-search-crawling-growth-18 https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-cr... 3. https://codeberg.org/forgejo/discussions/issues/319#issuecomment-3486873 https://codeberg.org/forgejo/discussions/issues/319#issuecom... 4. https://github.com/TecharoHQ/anubis/issues/964#issuecomment-3172689053 https://github.com/TecharoHQ/anubis/issues/964#issuecomment-...
SCdF 1y ago
This wiki page was created in 2011, in case you're wondering how long they've held this position
tracerbulletx 1y ago
This is a screed that does not address a single point of the actual philosophical issue. The issue is a debate between what the expectations are for content posted on the public internet. There is the viewpoint that it should be totally machine operable and programmatic and if you want it to be private you should gate it behind authentication, that the semantic web is an important concept and violating it is a breach of protocol. There's also the argument that it's your content, no one has a right to it, and you should be able to license its use anyway you want. There is a trade off between the implications of the two.
procaryote 1y ago
Not having things archived because you explicitly opted out of crawling is a feature, not a bug Otherwise you can whitelist a specific crawler in robots.txt
- rzzzt 1y ago
  (I understand it is a different entity) archive .org at one point started to honor the robots.txt settings of the website's current owner, hiding archived copies you could browse in the past. I don't know whether they still do this.
hosh 1y ago
I absolutely will use a robots.txt on my personal sites, which will include a tarpit. This has nothing to do with keeping my webserver from crashing, and has more to do with crawlers using content to train AI. Anything I actually want to keep as a legacy, I’ll store with permanent.org
soiltype 1y ago
I have more complaints about this shitty article than it is worth. At least it's clearly a human screed, not LLM generated. Just say you won't honor it and move on.
_Algernon_ 1y ago
I mean the main reason is that robots.txt is pointless these days. When it was introduced, the web was largely collaborative project within the academic realm. A system based on the honor system worked for the most part. These days the web is adversarial through and through. A robots.txt file seems like an anachronistic, almost quaint museum piece, reminding us of what once was, while we stoop head first into tech feudalism.
- RajT88 1y ago
  In fact the problem of the "never ending September" has evolved into, "the never ending barrage of Septemberbots and AI vacuum bots". The horrors of the 1990's internet is quaint by comparison to the society level problems we now have.
- rolph 1y ago
  its not a request anymore, its often a warning not to go any farther, lest ye be zipbombed or tarpitted into wasting bandwidth and time.
jawns 1y ago
Is a person not allowed to put up a "no trespassing" sign on their land unless they have a reason that makes sense to would-be trespassers? I know that ignoring a robots.txt file doesn't carry the same legal consequences as trespassing on physical land, but it's still going against the expressed wishes of the site owner. Sure, you can argue that the site owner should restrict access using other gates, just as you might argue a land owner should put up a fence. But isn't this a weird version of Chesterton's Fence, where a person decides that they will trespass beyond the fenced area because they can see no reason why the area should be fenced?
- deleted 1y ago
  [deleted]
- stickfigure 1y ago
  End User License Agreement. To view the contents of this website, you must have purchased one or more 12oz or larger Diet Sprite(TM) products within the last 24 hours. Acknowledge that violations may be referred to local law enforcement for criminal prosecution. Accept [ ] Exit Site [ ] Share and enjoy.
layer8 1y ago
(2011)
- rafram 1y ago
  Thanks, added to title.
Sanzig 1y ago
Ugh. Yeah, this misses the point: not everyone wants their content archived. Of course, there are no feasible technical means to prevent this from happening, so robots.txt is a friendly way of saying "hey, don't save this stuff." Just because theres no technical reason you can't archive doesn't mean that you shouldn't respect someone's wishes. It's a bit like going to a clothing optional beach with a big camera and taking a bunch of photos. Is what you're doing legal? In most countries, yes. Are you an asshole for doing it? Also yes.
xg15 1y ago
Set up a tarpit, put it in the robots.txt as a global exclusion, watch hilarity ensue for all the crawlers that ignore the exclusion.
madamelic 1y ago
robots.txt is the digital equivalent of "one piece per person" on an unwatched Halloween bowl. The people who wouldn't don't need the sign, the people who want to do it anyway. If you don't want crawling, there are other ways to prevent / slow down crawling than asking nicely.
- blipvert 1y ago
  Alternatively, it’s the equivalent of having a sign saying “Caution, Tarpit” and having a tarpit. You’re welcome to ride if you obey the rules of carriage. Don’t make me tap the sign.
  - notatoad 1y ago
    >Alternatively, it’s the equivalent of having a sign saying “Caution, Tarpit”. yeah, the fact that it is actually useful for blocking crawlers is kind of a misleading thing. it's called "robots.txt", it's there to help the robots, not to block them. you use it to help a robot crawl your site more efficiently, and tell them what not to bother looking at so they don't waste their time. people seem to have forgotten really quickly that making your website as accessible as possible to crawlers was actually considered a good thing, and there was a whole industry around optimizing websites for search engines crawlers.
knome 1y ago
given this is from a group determined to copy and archive your data with or without your permission, their opinions on the usefulness of ROBOTS.TXT seem kind of irrelevant. of course they aren't going to respect it. they see themselves as 'rogue digital archivists', and being edgy and legally rather grey is part of their self-image. they're going to back it up, regardless of who says they can't. for the rest of the net, ROBOTS.TXT is still often used for limiting the blast radius of search engines and bot crawl-delays and other "we know you're going to download this, please respect these provisions" type situations, as a sort of gentlemen's agreement. the site operator won't blackhole your net-ranges if you abide their terms. that's a reasonably useful thing to have.
kazinator 1y ago
If you don't obey someone's robots.txt, your bot will end up in their honeypot: be prepared for zip bombs, or generated infinite recursions and whatnot. You better have good countercountermeasures. robots.txt is helping you identify which parts of the website the author believes are of interest for search indexing or AI training or whatever. fetching robots.txt and behaving in a conforming manner can open doors for you. If I spot a bot like that in my logs, I might whitelist them, and feed them a different robots.txt.
- paulddraper 1y ago
  tbf most bots do that nowadays.
giancarlostoro 1y ago
Its mostly for search engines to figure out how to crawl your website. Use it sparingly.
dang 1y ago
Related: ROBOTS.TXT is a suicide note - https://news.ycombinator.com/item?id=13376870 https://news.ycombinator.com/item?id=13376870 - Jan 2017 (30 comments) Robots.txt is a suicide note - https://news.ycombinator.com/item?id=2531219 https://news.ycombinator.com/item?id=2531219 - May 2011 (91 comments)
rglover 1y ago
I see old stuff like this and it starts to become clear why the web is in tatters today. It may not be respected, but unless you have a really silly config (I'm hard-pressed to even guess what you could do short of a weird redirect loop), it won't be doing any harm. > What this situation does, in fact, is cause many more problems than it solves - catastrophic failures on a website are ensured total destruction with the addition of ROBOTS.TXT. Of course an archival pedant [1] will tell you it's a bad idea (because it makes their archival process less effective)—but this is one of those "maybe you should think for yourself and not just implement what some rando says on the internet" moments. If you're using version control, running backups, and not treating your production env like a home computer (i.e., you're aware of the ephemeral nature of a disk on a VPS), you're fine. [1] Archivists are great (and should be supported), but when you turn it into a crusade, you get foolish, generalized takes like this wiki.
- bigstrat2003 1y ago
  I really lost a lot of respect for the team when I read this page. No matter how good their intentions are, by deliberately ignoring robots.txt they are behaving just as badly as the various AI companies (and other similar entities) that scrape data against the wishes of the site owner. They are, in other words, directly contributing to the destruction of the commons by abusing trust and ensuring that everyone has to treat each other as a potential bad actor. Dick move, Archive Team.
  - akk0 1y ago
    Mind you're reading a 14 year old page. I honestly don't see any value in this being posted on HN.
    - rafram 1y ago
      It's unambiguously still the official position of the organization. Linked from their FAQ: https://wiki.archiveteam.org/index.php/Frequently_Asked_Questions https://wiki.archiveteam.org/index.php/Frequently_Asked_Ques...
- 1y ago
snowwrestler 1y ago
Copying my comment from a previous discussion of ignoring robots.txt, below. I actually don’t care if someone ignores my robots.txt, as long as their crawler is well run. But the smug attitude is annoying when so many crawlers are not. ———— We have a faceted search that creates billions of unique URLs by combinations of the facets. As such, we block all crawlers from it in robots.txt, which saves us AND them from a bunch of pointless indexing load. But a stealth bot has been crawling all these URLs for weeks. Thus wasting a shitload of our resources AND a shitload of their resources too. Whoever it is, they thought they were being so clever by ignoring our robots.txt. Instead they have been wasting money for weeks. Our block was there for a reason.
- paulddraper 1y ago
  Yes, this has been the traditional reason for robots.txt -- protects the bot as much as it does the site.
- nonethewiser 1y ago
  Wasting your money too right? I guess another angle on this is putting trust in people to comply with ROBOTS.txt. There is no guarantee so we should probably design with the assumption that our sites will be crawled however people want. Also im curious about your use case. >We have a faceted search that creates billions of unique URLs by combinations of the facets. Are we talking about a search that has filters like (to use ecommerce as an example), brand, price range, color, etc. And then all these combinations make up a URL (hence bilions)? How does a crawler discover these? They are just designed to detect all these filters and try all combinations? That doesn't really jive with my understanding with crawlers but otherwise IDK how it would be generating billions of unique URLs. I guess maybe they could also be included in sitemaps but I doubt that.
  - xigoi 1y ago
    > How does a crawler discover these? They are just designed to detect all these filters and try all combinations? Presumably each of the facets is a link that adds it to the current query, so if you recursively follow links, you will end up with all combinations.
spaceport 1y ago
Renaming my robots.txt to reeeebots.txt and writing a justification line by line on why XYZ shouldn't be archived is now on my todo. Along with adding a tarpit.
gmuslera 1y ago
robots.txt takes as assumptions that are well-meant, and respectful to the site intentions, major players, that try to index/mirror sites while avoiding overwhelming it and accessing only what is supposed to be freely accessed. Using a visible user-agent, having a clearly defined IP block for doing those scans, and a method of scanning goes in the same direction of cooperating with the site owner to both get visibility while not affecting (a lot) functionality. But that doesn't mean that there aren't bad players, that ignore the robots.txt, give random user agent strings, or connects from IPs from all the world to avoid being blocked. LLMs has changed a bit the landscape, mostly because far more players want to get everything or have automated tools to search your information on specific requests. But that doesn't rule out that still exist well-behaved players.
btilly 1y ago
Whatever we think of archive.org's position, modern AI companies have clearly taken the same basic position. And are willing to devote a lot more resources to vacuuming up the internet than crawlers did back in 2011. See https://news.ycombinator.com/item?id=43476337 https://news.ycombinator.com/item?id=43476337 for a random example of a discussion about this. My personal position is that robots.txt is useless when faced with companies who have no sense of shame about abusing the resources of others. And if it is useless, there isn't much of a point in having it. Just make sure that nothing public facing is going to be too expensive for your server. But that's like saying that the solution to thieves is to not carry money around. Yes, it is a reasonable precaution. But it doesn't leave me feeling any better about the thieves.
Bender 1y ago
Any time I think about robots.txt, I think about a quote from Pirates of the Caribbean. [1] "The only rules that really matter are these: what a man can do and what a man can't do." except that I replace man with bot. Everything should be designed to handle pirates, given the hostile nature of the internet. To me, robots.txt is a friendly way to say, "Hey bots, this is what I allow. Stay in these lanes including crawl-delay [2] and I won't block you." Step outside and I can put you on an exercise wheel. I know very few support crawl-delay since it is not part of the standard but that is not my problem. Blocking bots or making them waste a lot of cycles or get dummy data or wildly reordering packets or adding random packet loss or slowing them to 2KB/s is more fun for me than playing Doom. [1] - https://www.youtube.com/watch?v=B4zwh26kP8o https://www.youtube.com/watch?v=B4zwh26kP8o [video][2 mins] [2] - https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directi...
karaterobot 1y ago
> Precisely one reason comes to mind to have ROBOTS.TXT, and it is, incidentally, stupid - to prevent robots from triggering processes on the website that should not be run automatically Counter-point: I have a blog I don't want to appear on search engines because it has private stuff on it. 25 years ago I added two lines to robots.txt file, and I've never seen it show up on any search engine ever since. I'm not pretending nobody has indexed my blog and kept a copy of the results. I'm just saying the blog I started in college doesn't show up when you search for my name on Google, which is all I care about.
- mnw21cam 1y ago
  Admittedly, the whole stupidity of the article aside, they do make a valid point with their "triggering processes" bit. GET should not perform side-effects. That's what POST is for. Also, for your blog, have you considered password-protecting it? As in, http passwords, which are still (surprisingly) a thing. You can even have a password-free landing page handing the password out to every human that visits, with an onward link to the password-protected site. That should stop the bots but keep letting the humans in.
  - zzo38computer 1y ago
    > GET should not perform side-effects. That's what POST is for. I agree, although not all triggered processes are things that cause changes; some might be calculations that are unnecessary for crawling, and other stuff like that. > Also, for your blog, have you considered password-protecting it? As in, http passwords, which are still (surprisingly) a thing. You can even have a password-free landing page handing the password out to every human that visits, with an onward link to the password-protected site. That should stop the bots but keep letting the humans in. I think it is a reasonable idea, although you can also add text into the password prompt that you can use to figure out the password (as the "realm" text). This also means that you will not need to use cookies.
josefritzishere 1y ago
It's a shame that most companies that ignore robots.txt also operate horribly behaved crawlers. I feel like they forfeit the moral high ground to render judgement.
zzo38computer 1y ago
I think it is not so right. It can be useful to exclude some kinds of dynamic files, files with redundant pieces in URLs (e.g. a query string for files that do not require it; if you could reasonably do it (which in many cases you unfortunately can't), you might make it never crawl URLs with a query string), to set crawl delays, etc. (You might also want to e.g. use other ways of mirroring some files; e.g. for a version control repository, you do not need to crawl the web pages and you can just clone the repository instead. This way, you will only need to clone the new and changed files and not all of them.) Robots.txt should not be used for preventing automated access in general or for disallowing mirrors to be made. Someone else wrote "I actually don't care if someone ignores my robots.txt, as long as their crawler is well run." I mostly agree with this, although whoever wrote the crawler does not know everything (but neither does the server operator). In writing the specification for the crawling policy file for Scorpion protocol, I had tried to make some things more clear and avoid some problems, although it is not perfect.
Permik 1y ago
The real question is that why haven't we moved robots.txt to DNS TXT records? This way the consent to scrape would be crystal clear even before they connect to the server.
motbus3 1y ago
Robots.txt was adopted on the trust that people working would respect others like themselves and follow the directives. Robots.txt worked for a while,
floydnoel 1y ago
Interestingly, not having a robots.txt pointing to a sitemap is considered an SEO error by tools like SEM Rush.

88 points by rafram 1y ago

Every website operator should be required to learn _rate-limiting_ as well as how to create a robots.txt file What matters is rate not "user-agent" User-agent can be fake but rate is real

rafram 1y ago

I think this is kind of misguided - it ignores the main reason sites use robots.txt, which is to exclude irrelevant/old/non-human-readable pages that nevertheless need to remain online from being indexed by search engines - but it's an interesting look at Archive Team's rationale.

xp84 1y ago
Yes, and I'd add to that dynamically-generated URLs of infinite variability which have two separate but equally-important reasons for automated traffic to avoid: 1. You (bot) are wasting your bandwidth, CPU, storage on a literally unbounded set of pages 2. This may or may not cause resource problems for the owner of the site (e.g. Suppose they use Algolia to power search and you search for 10,000,000 different search terms... and Algolia charges them by volume of searches.) The author of this angry rant really seems specifically ticked at some perceived 'bad actor' who is using robots.txt as an attempt to "block people from getting at stuff" but it's super misguided in that it ignores an entire purpose of robots.txt that is not even necessarily adversarial to the "robot." This whole thing could have been a single sentence: "Robots.txt has a few competing vague interpretations and is voluntary; not all bots obey it, so if you're fully relying on it to prevent a site from being archived, that won't work."
- deleted 1y ago
  [deleted]
paulddraper 1y ago
Correct. That has been one of the biggest uses -- improve SEO by preventing web crawlers from getting lost/confused in a maze of irrelevant content.

rolph 1y ago

the archiveteam statements in the article are sure to win special attention, i think this could be footgunning, and .IF archiveteam .THEN script.exe pleasantries.

marbletiles 1y ago

Not sure the emotive language is warranted. Message appears to be “if you use robots.txt AND archive sites honor it AND you are dumb enough to delete your data without a backup THEN you won’t have a way to recover and you’ll be sorry”. It also presumes that dealing with automated traffic is a solved problem, which with the volumes of LLM scraping going on, is simply not true for more hobbyist setups.

bigbuppo 1y ago
Or major web properties for that matter.
paulddraper 1y ago
> volumes of LLM scraping FWIW I have not seen a reputable report on the % of web scraping in the past 3 years. (Wikipedia being a notable exception...but I would guess Wikipedia to see a far larger increase than anything else.)
- esseph 1y ago
  It's hard because of attribution, but it absolutely is happening at very high volume. I actually got an alert this morning when I woke up from our monitoring tools that some external sites were being scraped. Happens multiple times a day. A lot of it is coming through compromised residential endpoint botnets.
  - paulddraper 1y ago
    Even without attribution…seeing bot traffic or general traffic increase
- tolmasky 1y ago
  Wikipedia says their traffic increased roughly 50% [1] from AI bots, which is a lot, sure, but nowhere near the amount where you'd have to rearchitect your site or something. And this checks out, if it was actually debilitating, you'd notice Wikipedia's performance degrade. It hasn't. You'd see them taking some additional steps to combat this. They haven't. Their CDN handles it just fine. They don't even both telling AI bots to just download the tarballs they specifically make available for this exact use case. More importantly, Wikipedia almost certainly represents the ceiling of traffic increase. But luckily, we don't have to work with such coarse estimation, because according to Cloudflare, the total increase from combined search and AI bots in the last year (May 2024 - May 2025), has just been... 18% [2]. The way you hear people talk about it though, you'd think that servers are now receiving DDOS-levels of traffic or something. For the life of me I have not been able to find a single verifiable case of this. Which if you think about it makes sense... It's hard to generate that sort of traffic, that's one of the reasons people pay for botnets. You don't bring a site to its knees merely by accidentally "not making your scraper efficient". So the only other possible explanation would be such a larger number of scrapers simultaneously but independently hitting sites. But this also doesn't check out. There aren't thousands of different AI scrapers out there that in aggregate are resulting in huge traffic spikes [2]. Again, the total combined increase is 18%. The more you look into this accepted idea that we are in some sort of AI scraping traffic apocalypse, the less anything makes sense. You then look at this Anubis "AI scraping mitigator" and... I dunno. The author contends that one if its tricks is that it not only uses JavaScript, but "modern JavaScript like ES6 modules," and that this is one of the ways it detects/prevents AI scrapers [3]. No one is rolling their own JS engine for a scraper such that they are being blocked from their inability to keep up with the latest ECMAScript spec. You are just using an existing JS engine, all of which support all these features. It would actually be a challenge to find an old JS engine these days. The entire things seems to be built on the misconception that the "common" way to build a scraper is doing something curl-esque. This idea is entirely based on the google scraper which itself doesn't even work that way anymore, and only ever did because it was written in the 90s. Everyone that rolls their own scraper these days just uses Puppeteer. It is completely unrealistic to make a scraper that doesn't run JavaScript and wait for the page to "settle down" because so many pages, even blogs, are just entirely client-side rendered SPAs. If I were to write a quick and dirty scraper today I would trivially make it through Anubis' protections... by doing literally nothing and without even realizing Anubis exists. Just using standard scraping practices with Puppeteer. Meanwhile Anubis is absolutely blocking plenty of real humans, with the author for example telling people to turn on cookies so that Anubis can do its job [4]. I don't think Anubis is blocking anything other than humans and Message's link preview generator. I'm investigating further, but I think this entire thing may have started due to some confusion, but want to see if I can actually confirm this before speculating further. 1. https://www.techspot.com/news/107407-wikipedia-servers-struggling-under-pressure-ai-scraping-bots.html https://www.techspot.com/news/107407-wikipedia-servers-strug... (notice the clickbait title vs. the actual contents) 2. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/?utm_source=chatgpt.com/#general-ai-and-search-crawling-growth-18 https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-cr... 3. https://codeberg.org/forgejo/discussions/issues/319#issuecomment-3486873 https://codeberg.org/forgejo/discussions/issues/319#issuecom... 4. https://github.com/TecharoHQ/anubis/issues/964#issuecomment-3172689053 https://github.com/TecharoHQ/anubis/issues/964#issuecomment-...

SCdF 1y ago

This wiki page was created in 2011, in case you're wondering how long they've held this position

tracerbulletx 1y ago

This is a screed that does not address a single point of the actual philosophical issue. The issue is a debate between what the expectations are for content posted on the public internet. There is the viewpoint that it should be totally machine operable and programmatic and if you want it to be private you should gate it behind authentication, that the semantic web is an important concept and violating it is a breach of protocol. There's also the argument that it's your content, no one has a right to it, and you should be able to license its use anyway you want. There is a trade off between the implications of the two.

procaryote 1y ago

Not having things archived because you explicitly opted out of crawling is a feature, not a bug Otherwise you can whitelist a specific crawler in robots.txt

rzzzt 1y ago
(I understand it is a different entity) archive .org at one point started to honor the robots.txt settings of the website's current owner, hiding archived copies you could browse in the past. I don't know whether they still do this.

hosh 1y ago

I absolutely will use a robots.txt on my personal sites, which will include a tarpit. This has nothing to do with keeping my webserver from crashing, and has more to do with crawlers using content to train AI. Anything I actually want to keep as a legacy, I’ll store with permanent.org

soiltype 1y ago

I have more complaints about this shitty article than it is worth. At least it's clearly a human screed, not LLM generated. Just say you won't honor it and move on.

_Algernon_ 1y ago

I mean the main reason is that robots.txt is pointless these days. When it was introduced, the web was largely collaborative project within the academic realm. A system based on the honor system worked for the most part. These days the web is adversarial through and through. A robots.txt file seems like an anachronistic, almost quaint museum piece, reminding us of what once was, while we stoop head first into tech feudalism.

RajT88 1y ago
In fact the problem of the "never ending September" has evolved into, "the never ending barrage of Septemberbots and AI vacuum bots". The horrors of the 1990's internet is quaint by comparison to the society level problems we now have.
rolph 1y ago
its not a request anymore, its often a warning not to go any farther, lest ye be zipbombed or tarpitted into wasting bandwidth and time.

jawns 1y ago

Is a person not allowed to put up a "no trespassing" sign on their land unless they have a reason that makes sense to would-be trespassers? I know that ignoring a robots.txt file doesn't carry the same legal consequences as trespassing on physical land, but it's still going against the expressed wishes of the site owner. Sure, you can argue that the site owner should restrict access using other gates, just as you might argue a land owner should put up a fence. But isn't this a weird version of Chesterton's Fence, where a person decides that they will trespass beyond the fenced area because they can see no reason why the area should be fenced?

deleted 1y ago
[deleted]
stickfigure 1y ago
End User License Agreement. To view the contents of this website, you must have purchased one or more 12oz or larger Diet Sprite(TM) products within the last 24 hours. Acknowledge that violations may be referred to local law enforcement for criminal prosecution. Accept [ ] Exit Site [ ] Share and enjoy.

layer8 1y ago

(2011)

rafram 1y ago
Thanks, added to title.

Sanzig 1y ago

Ugh. Yeah, this misses the point: not everyone wants their content archived. Of course, there are no feasible technical means to prevent this from happening, so robots.txt is a friendly way of saying "hey, don't save this stuff." Just because theres no technical reason you can't archive doesn't mean that you shouldn't respect someone's wishes. It's a bit like going to a clothing optional beach with a big camera and taking a bunch of photos. Is what you're doing legal? In most countries, yes. Are you an asshole for doing it? Also yes.

xg15 1y ago

Set up a tarpit, put it in the robots.txt as a global exclusion, watch hilarity ensue for all the crawlers that ignore the exclusion.

madamelic 1y ago

robots.txt is the digital equivalent of "one piece per person" on an unwatched Halloween bowl. The people who wouldn't don't need the sign, the people who want to do it anyway. If you don't want crawling, there are other ways to prevent / slow down crawling than asking nicely.

blipvert 1y ago
Alternatively, it’s the equivalent of having a sign saying “Caution, Tarpit” and having a tarpit. You’re welcome to ride if you obey the rules of carriage. Don’t make me tap the sign.
- notatoad 1y ago
  >Alternatively, it’s the equivalent of having a sign saying “Caution, Tarpit”. yeah, the fact that it is actually useful for blocking crawlers is kind of a misleading thing. it's called "robots.txt", it's there to help the robots, not to block them. you use it to help a robot crawl your site more efficiently, and tell them what not to bother looking at so they don't waste their time. people seem to have forgotten really quickly that making your website as accessible as possible to crawlers was actually considered a good thing, and there was a whole industry around optimizing websites for search engines crawlers.

knome 1y ago

given this is from a group determined to copy and archive your data with or without your permission, their opinions on the usefulness of ROBOTS.TXT seem kind of irrelevant. of course they aren't going to respect it. they see themselves as 'rogue digital archivists', and being edgy and legally rather grey is part of their self-image. they're going to back it up, regardless of who says they can't. for the rest of the net, ROBOTS.TXT is still often used for limiting the blast radius of search engines and bot crawl-delays and other "we know you're going to download this, please respect these provisions" type situations, as a sort of gentlemen's agreement. the site operator won't blackhole your net-ranges if you abide their terms. that's a reasonably useful thing to have.

kazinator 1y ago

If you don't obey someone's robots.txt, your bot will end up in their honeypot: be prepared for zip bombs, or generated infinite recursions and whatnot. You better have good countercountermeasures. robots.txt is helping you identify which parts of the website the author believes are of interest for search indexing or AI training or whatever. fetching robots.txt and behaving in a conforming manner can open doors for you. If I spot a bot like that in my logs, I might whitelist them, and feed them a different robots.txt.

paulddraper 1y ago
tbf most bots do that nowadays.

giancarlostoro 1y ago

Its mostly for search engines to figure out how to crawl your website. Use it sparingly.

dang 1y ago

Related: ROBOTS.TXT is a suicide note - https://news.ycombinator.com/item?id=13376870 https://news.ycombinator.com/item?id=13376870 - Jan 2017 (30 comments) Robots.txt is a suicide note - https://news.ycombinator.com/item?id=2531219 https://news.ycombinator.com/item?id=2531219 - May 2011 (91 comments)

rglover 1y ago

I see old stuff like this and it starts to become clear why the web is in tatters today. It may not be respected, but unless you have a really silly config (I'm hard-pressed to even guess what you could do short of a weird redirect loop), it won't be doing any harm. > What this situation does, in fact, is cause many more problems than it solves - catastrophic failures on a website are ensured total destruction with the addition of ROBOTS.TXT. Of course an archival pedant [1] will tell you it's a bad idea (because it makes their archival process less effective)—but this is one of those "maybe you should think for yourself and not just implement what some rando says on the internet" moments. If you're using version control, running backups, and not treating your production env like a home computer (i.e., you're aware of the ephemeral nature of a disk on a VPS), you're fine. [1] Archivists are great (and should be supported), but when you turn it into a crusade, you get foolish, generalized takes like this wiki.

bigstrat2003 1y ago
I really lost a lot of respect for the team when I read this page. No matter how good their intentions are, by deliberately ignoring robots.txt they are behaving just as badly as the various AI companies (and other similar entities) that scrape data against the wishes of the site owner. They are, in other words, directly contributing to the destruction of the commons by abusing trust and ensuring that everyone has to treat each other as a potential bad actor. Dick move, Archive Team.
- akk0 1y ago
  Mind you're reading a 14 year old page. I honestly don't see any value in this being posted on HN.
  - rafram 1y ago
    It's unambiguously still the official position of the organization. Linked from their FAQ: https://wiki.archiveteam.org/index.php/Frequently_Asked_Questions https://wiki.archiveteam.org/index.php/Frequently_Asked_Ques...
1y ago

snowwrestler 1y ago

Copying my comment from a previous discussion of ignoring robots.txt, below. I actually don’t care if someone ignores my robots.txt, as long as their crawler is well run. But the smug attitude is annoying when so many crawlers are not. ———— We have a faceted search that creates billions of unique URLs by combinations of the facets. As such, we block all crawlers from it in robots.txt, which saves us AND them from a bunch of pointless indexing load. But a stealth bot has been crawling all these URLs for weeks. Thus wasting a shitload of our resources AND a shitload of their resources too. Whoever it is, they thought they were being so clever by ignoring our robots.txt. Instead they have been wasting money for weeks. Our block was there for a reason.

paulddraper 1y ago
Yes, this has been the traditional reason for robots.txt -- protects the bot as much as it does the site.
nonethewiser 1y ago
Wasting your money too right? I guess another angle on this is putting trust in people to comply with ROBOTS.txt. There is no guarantee so we should probably design with the assumption that our sites will be crawled however people want. Also im curious about your use case. >We have a faceted search that creates billions of unique URLs by combinations of the facets. Are we talking about a search that has filters like (to use ecommerce as an example), brand, price range, color, etc. And then all these combinations make up a URL (hence bilions)? How does a crawler discover these? They are just designed to detect all these filters and try all combinations? That doesn't really jive with my understanding with crawlers but otherwise IDK how it would be generating billions of unique URLs. I guess maybe they could also be included in sitemaps but I doubt that.
- xigoi 1y ago
  > How does a crawler discover these? They are just designed to detect all these filters and try all combinations? Presumably each of the facets is a link that adds it to the current query, so if you recursively follow links, you will end up with all combinations.

spaceport 1y ago

Renaming my robots.txt to reeeebots.txt and writing a justification line by line on why XYZ shouldn't be archived is now on my todo. Along with adding a tarpit.

gmuslera 1y ago

robots.txt takes as assumptions that are well-meant, and respectful to the site intentions, major players, that try to index/mirror sites while avoiding overwhelming it and accessing only what is supposed to be freely accessed. Using a visible user-agent, having a clearly defined IP block for doing those scans, and a method of scanning goes in the same direction of cooperating with the site owner to both get visibility while not affecting (a lot) functionality. But that doesn't mean that there aren't bad players, that ignore the robots.txt, give random user agent strings, or connects from IPs from all the world to avoid being blocked. LLMs has changed a bit the landscape, mostly because far more players want to get everything or have automated tools to search your information on specific requests. But that doesn't rule out that still exist well-behaved players.

btilly 1y ago

Whatever we think of archive.org's position, modern AI companies have clearly taken the same basic position. And are willing to devote a lot more resources to vacuuming up the internet than crawlers did back in 2011. See https://news.ycombinator.com/item?id=43476337 https://news.ycombinator.com/item?id=43476337 for a random example of a discussion about this. My personal position is that robots.txt is useless when faced with companies who have no sense of shame about abusing the resources of others. And if it is useless, there isn't much of a point in having it. Just make sure that nothing public facing is going to be too expensive for your server. But that's like saying that the solution to thieves is to not carry money around. Yes, it is a reasonable precaution. But it doesn't leave me feeling any better about the thieves.

Bender 1y ago

Any time I think about robots.txt, I think about a quote from Pirates of the Caribbean. [1] "The only rules that really matter are these: what a man can do and what a man can't do." except that I replace man with bot. Everything should be designed to handle pirates, given the hostile nature of the internet. To me, robots.txt is a friendly way to say, "Hey bots, this is what I allow. Stay in these lanes including crawl-delay [2] and I won't block you." Step outside and I can put you on an exercise wheel. I know very few support crawl-delay since it is not part of the standard but that is not my problem. Blocking bots or making them waste a lot of cycles or get dummy data or wildly reordering packets or adding random packet loss or slowing them to 2KB/s is more fun for me than playing Doom. [1] - https://www.youtube.com/watch?v=B4zwh26kP8o https://www.youtube.com/watch?v=B4zwh26kP8o [video][2 mins] [2] - https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directi...

karaterobot 1y ago

> Precisely one reason comes to mind to have ROBOTS.TXT, and it is, incidentally, stupid - to prevent robots from triggering processes on the website that should not be run automatically Counter-point: I have a blog I don't want to appear on search engines because it has private stuff on it. 25 years ago I added two lines to robots.txt file, and I've never seen it show up on any search engine ever since. I'm not pretending nobody has indexed my blog and kept a copy of the results. I'm just saying the blog I started in college doesn't show up when you search for my name on Google, which is all I care about.

mnw21cam 1y ago
Admittedly, the whole stupidity of the article aside, they do make a valid point with their "triggering processes" bit. GET should not perform side-effects. That's what POST is for. Also, for your blog, have you considered password-protecting it? As in, http passwords, which are still (surprisingly) a thing. You can even have a password-free landing page handing the password out to every human that visits, with an onward link to the password-protected site. That should stop the bots but keep letting the humans in.
- zzo38computer 1y ago
  > GET should not perform side-effects. That's what POST is for. I agree, although not all triggered processes are things that cause changes; some might be calculations that are unnecessary for crawling, and other stuff like that. > Also, for your blog, have you considered password-protecting it? As in, http passwords, which are still (surprisingly) a thing. You can even have a password-free landing page handing the password out to every human that visits, with an onward link to the password-protected site. That should stop the bots but keep letting the humans in. I think it is a reasonable idea, although you can also add text into the password prompt that you can use to figure out the password (as the "realm" text). This also means that you will not need to use cookies.

josefritzishere 1y ago

It's a shame that most companies that ignore robots.txt also operate horribly behaved crawlers. I feel like they forfeit the moral high ground to render judgement.

zzo38computer 1y ago

I think it is not so right. It can be useful to exclude some kinds of dynamic files, files with redundant pieces in URLs (e.g. a query string for files that do not require it; if you could reasonably do it (which in many cases you unfortunately can't), you might make it never crawl URLs with a query string), to set crawl delays, etc. (You might also want to e.g. use other ways of mirroring some files; e.g. for a version control repository, you do not need to crawl the web pages and you can just clone the repository instead. This way, you will only need to clone the new and changed files and not all of them.) Robots.txt should not be used for preventing automated access in general or for disallowing mirrors to be made. Someone else wrote "I actually don't care if someone ignores my robots.txt, as long as their crawler is well run." I mostly agree with this, although whoever wrote the crawler does not know everything (but neither does the server operator). In writing the specification for the crawling policy file for Scorpion protocol, I had tried to make some things more clear and avoid some problems, although it is not perfect.

Permik 1y ago

The real question is that why haven't we moved robots.txt to DNS TXT records? This way the consent to scrape would be crystal clear even before they connect to the server.

motbus3 1y ago

Robots.txt was adopted on the trust that people working would respect others like themselves and follow the directives. Robots.txt worked for a while,

floydnoel 1y ago

Interestingly, not having a robots.txt pointing to a sitemap is considered an SEO error by tools like SEM Rush.