7 ms·
Robots.txt is a suicide note
"Archive Team interprets ROBOTS.TXT as damage and temporary madness, and works around it. Everyone should. If you don't want people to have your data, don't put it online."
- smosher 15y agoThe rationale is weak. Some data is simply not worth indexing, and not worth serving up to bots. The flipside is: your crawler doesn't need to fetch everything on my site, and I'd be happy to ban all non-conforming bots site-wide. It's not just about the functionality, but also a show of good faith and basic respect. If you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all. robots.txt isn't perfect. Ideally a web server would be configured to deny bots access to restricted content via some sort of dnsbl mechanism (or CPAN/whatever module.) Or do both and ban the non-conforming site-wide. The above notwithstanding, I'm voting for this article. It doesn't betray the usual cowardice by hiding the assertion behind the presumptuous Why.
- panza 15y ago"Some data is simply not worth indexing, and not worth serving up to bots." The broader points being made in the article are that the value of information is determined by the visitor, and that the burden of keeping the site up and running should fall on the host. By that reasoning, the presence of a ROBOTS.TXT signifies "damage" or "temporary madness", and should therefore be ignored. Also, keep in mind where this article came from - the Archive Team are bloody-minded about preserving information.
- drivingmenuts 15y agoOK, wait. If the data has value to the user, shouldn't the user be paying the host for the cost of making that data available (or perhaps considerably more, if it has a lot of value to the user)?
- sbierwagen 15y agoExcellent. How much are you paying ycombinator for your account? If the answer is "nothing", then I guess you've just told me that all your comments are useless, and I should ignore them.
- sbierwagen 15y agoI'm committing the classic mistake of talking about the karma system here, but what, precisely, is wrong with my statement? Drivingmenuts says the user should pay for hosting data. HN is hosting his data, but he doesn't pay for it. Is there some error in my logic, here, some flaw in my conclusion?
- smosher 15y agothe value of information is determined by the visitor That may be so in a sense, unless the visitor has no means of making the call -- or does so poorly. I promise you that every robots.txt ignoring bot makes terrible judgments in that regard. (I'm not trying to overstate anything here; those that abide it make, on average, slightly less terrible judgments.) But it's not just the visitor who gets to make judgment calls. The value of serving those pages is something the host can decide. Belligerent drunks who abuse the staff aren't allowed in the coffee shop, and if they do it often enough they're not allowed back in when they sober up either.
- chalst 15y agothe Archive Team are bloody-minded about preserving information They won't do a good job of that if they encourage too many people to make IP blocks of their machines.
- sbierwagen 15y agoAT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.
- chalst 15y agoDo all of them ignore robots.txt?
- sbierwagen 15y agoWell, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to: User-agent: * Disallow: / Does that count? 1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/ http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/
- dalke 15y agoHere's a concrete example of a site not worth archiving - back in the early days of the web (1994?) a friend of mine wrote a 4 function calculator. Each button was a hyperlink, so "34-12" meant going to the pages "?", "?3", "?34", "?34-", "?34-1", and "?34-12". That's an infinite number of web pages which should not be indexed.
- thyrsus 15y agoI remember some anti-spam vigilante code that told legitimate bots to go away with robots.txt, told humans what the game was about, and attacked e-mail address scrapers by presenting an infinite number of fake e-mail addresses. "Click here for 1000 more e-mail addresses that will never bother a human!".
- alanh 15y agoI love it!
- rachelbythebay 15y agoNot all of them were fake. Some of them were spam traps which never appeared anywhere else. Some of those spam traps were also coded so you could track down when they were served to connect the spiders to the spewers. At least, that's the way I used to roll.
- smosher 15y agoAh very good! I was trying to think of a short example that wasn't contrived with this very behaviour.
- chalst 15y agoIf you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all. Quite. My robots.txt reads: User-agent: * Disallow: /cgi-bin/ Disallow: /scratch/ I do not believe, as the archiveteam.org seem to think I do, that I have either stopped bots interfering with CGI scripts or spidering semi-private, temporary content. Indeed, my server logs show clearly enough that bots do these things all the time. Instead, the bot has violated my site's policy and its IP address will be considered for an iptables ban. It is a matter of which bots I consider fair game for rough treatment. This is a common attitude to robots.txt, I think, and it does not reflect well on archiveteam.org's competence that they seem to be unaware of it.
- G_Morgan 15y agoYeah what this really needs is a bot block filter service where bots that do not conform to expectations can be bulk banned. It is too much effort to do this manually but a larger service can work.
- hammock 15y agoI love the tone of this article, I was smiling the whole time. Especially here- the onslaught of some social media hoo-hah edit: just clicked through a few pages- whoever does the writing at Archiveteam is fantastic!
- zzzo 15y agoJason Scott, very distinctive voice. Maker of the BBS Documentary, etc. Engaging public speaker too; lots of good examples on his wikipedia page: http://en.wikipedia.org/wiki/Jason_Scott_Sadofsky http://en.wikipedia.org/wiki/Jason_Scott_Sadofsky
- sbierwagen 15y agoPoint of order: A bunch of us[1] maintain the wiki. Jason just does the big, visible things, and is thus big and visible. http://ascii.textfiles.com/ http://ascii.textfiles.com/ is his personal blog, and is as profanity-ridden as you would expect. 1: http://archiveteam.org/index.php?title=IRC_Channel http://archiveteam.org/index.php?title=IRC_Channel
- deleted 15y ago[deleted]
- forgotusername 15y agoThis is composed from equals parts of insight and daftness, though not entirely for the right reason. The daftness: maybe the claim is true that robots.txt was only a stop-gap measure back when web servers sucked, however the de facto modern use for it goes far beyond that, and ignoring that standard is likely to piss off lots of people. The insight: for crawlers, relying on robots.txt to prevent getting stuck indexing infinite hierarchies of data is a bad idea. It should be able to figure that much out for itself, so it doesn't explode when faced with sites that don't exclude such hierarchies using robots.txt. For servers, relying on a client hint to ensure reliability is daft. It should have some form of rate limiting built in, as that's the only sensible design. This seems the only marginally sensible use of robots.txt from a server standpoint. Using it for any form of security (e.g. preventing DB scraping) is daft, and a more robust mechanism should be employed there too.
- marshray 15y agohowever the de facto modern use for it goes far beyond that, and ignoring that standard is likely to piss off lots of people I think they're saying is that their archive is going to ignore it, and your website should just eliminate it. We'll have to assume that archiveteam.org are by now real experts in guessing which buttons are going to be pissing people off. :-)
- chronomex 15y agoArchiveteam member here. Robots.txt is useful for a site to avoid having search-engine spiders to blindly wander down an infinite hallway. Archiveteam projects are different because we run closely-monitored, highly targeted crawls. If a site has an infinite hallway in it, we'll notice that and exclude it while taking steps to replicate enough of it to retain its valuable attributes. We also do our best to avoid retrieving content more times than necessary. If you operate a database, we try to get enough pages to replicate a good portion of the underlying data.
- derpy 15y agoYou do try to contact the "targets" sometimes though?
- j_baker 15y agoThis may be a dumb move from a legal perspective. Court cases have alluded that robots.txt files may count as technological measures in DMCA cases[1]. Granted, that's far from guaranteed. But I certainly wouldn't want to be the one to go to court over it. [1] http://www.groklaw.net/article.php?story=20070819090725314&query=robots.txt http://www.groklaw.net/article.php?story=20070819090725314&#...
- smosher 15y agoWhile I think the AT stance amounts to arrogance more than anything, enforcing compliance with robots.txt with law is absurd and unjust. To the best of my knowledge it's not a part of the HTTP standards or codified in any law.
- j_baker 15y agoDon't be naive. People get sued over things that aren't codified in any law or part of the HTTP standard every day. This is a bit of a legal grey area. The kind of thing that BigCos with expensive lawyers like to sue over. Would you really be willing to pay lawyers to settle it in court once and for all?
- smosher 15y agoI think you misunderstood. I wasn't commenting on the likelihood, I was commenting on the justice of it. I'm not speaking as a practitioner of law (I am not one), I'm speaking as a subject of the law and it's equally my responsibility to respond to injustice.
- keeperofdakeys 15y agoSuing is not about what is lawful or not; suing is part of the civil court, not court of law. Anyone can sue anybody for any reason and the judge has the responsibility to decide if any wrongs were performed, and if they deserve compensation. So it isn't about what is lawful, it is about what bullshit they can convince the judge of, so getting a clued in judge is what you need.
- zbowling 15y agoI feel like creating a honey pot for bad bots now. Put an exclude line in ROBOTS.TXT and then include that URL in my pages and when a bot hits it anyways, ban the IP.
- catch23 15y agoHow would you know if the robot is autonomous or not? They could just use a typical browser user-agent and you wouldn't know...
- tlrobinson 15y agoHide the link from users?
- pyre 15y agoI seem to remember there being a thread on HN about spam bot scraping for email addresses, and some of them are sophisticated enough that they can even determine if an element has been hidden through rules applied via an external CSS file.
- Aloisius 15y agoLink a 1 pixel image and stick it in the bottle left hand corner? While someone could click on it, it is extremely unlikely. IIRC, technically you can have a 0 pixel image in some formats (gif) though I imagine some browsers won't like that.
- deleted 15y ago[deleted]
- autoreverse 15y agoUsing a 1 pixel transparent GIF works - also worth putting in a warning for screen readers not to follow the link. FWIW I've been using robots.txt honeypots on two sites for the past few years to deter scrapers. Approx 1,600 IP addresses have been banned. Most of these are for data centers.
- yuhong 15y agoBTW, robots.txt disables access to versions of pages already archived on the Wayback Machine. I encountered this when looking for old technotes on developer.apple.com.
- slouch 15y agoThe wayback machine is the only reason I use robots.txt: User-agent: ia_archiver Disallow: / There is no good reason to let old (and possibly embarrassing) website designs distract from the current version of your website.
- gojomo 15y agoFew users prefer the archived versions; only when they need information that has disappeared (or is temporarily unavailable) do they try the Wayback Machine. Many, many web designers are delighted to be able to see their old creations – or even recover past work from the Archive when all other copies are lost to organizational upheaval and system problems. Finally, note that by blocking archiving, you're opting out of history. Those who look back to see the web of today won't see your contributions. Alternatives, competitors, and other projects that are open to archiving will be seen and vividly remembered. Being invisible to the future may be OK for your projects, but many others prefer to part of the shared memory. (FYI: I work on web archiving at the Internet Archive.)
- tszming 15y agoSome parameter are useful, but they are not part of the standard. e.g. Crawl-Delay, prevent DDOS from YAHOO! Slurp
- soult 15y agoIs Yahoo! Slurp still active? I thought they turned it off when they switched to using Bing search results on Yahoo!
- tomjen3 15y agoNo, it is about not being willing to waste bandwidth and server capacity on unworthy projects (no person will ever search for my site through baidu but it still being indexed). Google and archive.org is one thing, I will be happy to support them.
- deleted 15y ago[deleted]
- mmaunder 15y agoI wonder if honeypots that auto-block rogue crawlers have occurred to these yoyos.
- sbierwagen 15y agoWe typically use wget running on consumer machines. This isn't like the spider farm of a typical search engine, with hundreds of machines consuming gigabytes/second of bandwidth. AT hits are specific and targeted.
- djmdjm 15y agoDear "archiveteam", I pay by the MB. How do I opt out of your shit? KTHXBYE
- sbierwagen 15y agoDon't accept user-generated content. Failing that, never go down, and never announce that you're shutting down.
- v21 15y agoOr, if you do, hand over the data in a nicely archivable form, and prevent the need for crawlers.
- jbk 15y agoWell, sure, robots.txt is not the best solution, but it works and helps a lot when you got msnbot or yandexbot that takes more than half of the requests of your mediawiki (differences between revisions), your gitweb (commitdiffs) or your phpBB installation and kills the performance... Bored of having our machine killed by those bots, we use some robots.txt. Sure, there are other solutions (proper blocking), but this one works perfectly fine and avoids having to modify 3rd party applications that we are running for an open-source development team.
- yaix 15y agorobots.txt is simple and effective. I do not want certain bots, especially so-called "archives" to automatically download all my content. And that's what robots.txt is for and works well. The article is just stupid, sorry. There is not one real knowledgable argument.
- chalst 15y agothat's what robots.txt is for and works well Robots.txt doesn't stop anyone doing anything, it is simply a policy. Bots either respect it or they do not. Archiveteam.org have indicated that they wish to join the side of the spammers and incompetent spider authors.
- user24 15y agoare you confusing archive.org and archiveteam?
- chalst 15y agoThanks, fixed. Actually I was not, that was a typo.
- joelmichael 15y agoI agree. archiveteam.org have stated they will ignore robots.txt. This makes me disrespect their organization. Well, that and their ridiculous website.
- gojomo 15y agoThe great things about 'robots.txt' are (1) it's the simplest thing that could possibly work; and (2) the default assumption in the absence of webmaster effort is 'allow'. (2) is immensely valuable. Without it, search engines and the largest archive of web content, the Internet Archive (where I work on web archiving), could not exist at their current scales, as a practical matter. There's a place for ArchiveTeam's style of in-your-face, adversarial archiving... but if it were the dominant approach, the backlash from publishers and the law could result in prevailing conventions that are much worse than robots.txt, such as a default-deny/always-ask-permission-first regime. Search and archiving activities would have to be surreptitious, or limited to those with much deeper pockets for obscuring their actions, requesting/buying permission, or legal defenses. So, Jason, be careful what you wish for.
- adaml_623 15y agoThe Archiveteam.org favicon is a hand making a rude gesture. I think that sums up many peoples opinion to this story. It certainly is an indicator of how seriously you should take this organisation.
- Dylan16807 15y agoAm I missing something? It looks to me like it's making a 'stop', with the hand held up and flat.
- Super_Jambo 15y agoSo a little while ago we had a story which was essentially: "promote your startup / website by causing maximum outrage! Outrage is good! YAY PISSING PEOPLE OFF!" Now we get a non-story which is essentially designed to piss off the people of HN. Looking forward to more of the same given it works.
- ssdsa 15y agoI don't really understand why they compare the Robots.txt file to a suicide note. Could anyone explain this?
- derpy 15y agoArchiveTeam mostly flashmob archives sites that are going to die an rm -rf / death, or atleast kill their users content. Like Geocities, Yahoo! Video, Google Video, Friendster
- kunley 15y agoThis is a childish argument based upon an attitude "don't use robots.txt because it interferes with what we do and what we do is aw3s0m3 l337". This attitude is also prevailing in the archiveteam's comments here. I doubt their actions can be taken seriously. I wonder how this made into 88 votes here on HN..
- perlgeek 15y agoTheir attitude can be summed up as "it's on the internet, it's ours to take". Ok, oversimplified, but that's the essence, no? So, dear archiveteam, please remember that when I put a server on the internet, it's a voluntary and public service, and putting 'Disallow:' lines in the robots.txt means that I set some rules. It's just rude to ignore those rules, whatever you motivations are. You have no right to access my content, just as you have no right to walk into my house. If I invite you, please behave.
- _delirium 15y agoThe analogies here always end up all over the map, but the walking-into-a-house one seems pretty off. The service is set up for anonymous public access, run by an automated process designed to service requests, and faces a public thoroughfare. If we have to make IRL analogies, that sounds closer to a vending machine or other kiosk on the side of a road. In that case, you probably wouldn't have much faith in whether people will follow any instructions you tape to the machine.
- sgentle 15y agoI disagree with this post almost as strongly as I agree with it. Robots.txt is a suicide note. It's utter short-sighted hubris to say "this is MY information and I don't want you spidering it". Are you volunteering to maintain that information forever? Are you promising to never go out of business? Never be ordered to remove it by the government? Never be bought out by Oracle? Right now there seems to be a lot of confusion over the morality of information. People are possessed by the strange idea that you, mister content provider, own that content and have an inalienable right to control it any way you can get away with. But someday you will die, and your company will die, just like Geocities, Google Video, and the Library of Alexandria. Society should have a right to keep that information after you're gone. Of course, the law disagrees. And without the efforts of criminals like geohot, the iPhone DevTeam, The Nomad, Muslix64 and, yes, The Archive Team, people of the future will have no way to access the information we've locked up through our own paranoia. You don't have to cast your mind to a thousand years in the future - it's happening right now. Vast swathes of data are disappearing as DRM servers go dark only a few years after they appear (thanks, MSN Music, Yahoo Music Store). I believe that we owe it to our descendants to give them access to their history. I believe it's not our decision whether the things we make are too valuable or too uncomfortable to be preserved. And I believe that robots.txt is a suicide note, a product of the diseased minds that think our short-term desire for control outweighs our legacy. But I don't know what the fuck the article's talking about. It seems to be making a bunch of points that don't matter. Use robots.txt to prevent technical problems if you like, I don't care. Just don't use it to stop people from crawling your content or you're shitting on the future.
- jamie_ca 15y agoWhile I'm sure the "content provider" would disagree, in many cases it's perfectly legal to make a copy of their content - the laws in place only prevent redistribution for a certain time period. Personally, I'd be ecstatic if there was some organization set up that is manufacturing high-quality archive copies of books, music, film and the like, and storing them with the date of initial publication, as well as the date they will enter public domain (where it is known - not the case for living authors) in various locales. Then have a site live-tracking the release of that content. There's a site that keeps getting linked every year that posts things that would be public domain this year were it not for US copyright extensions, and something like that with accurate PD status and archive copies would be marvelous.
- cullenking 15y agoSorry, but this is terrible advice. Yes, you should make sure your site won't break if it's slammed by a large crawl. Yes you should hide destructive actions behind posts, not gets. But, robots.txt is insanely useful. If I didn't have a robots.txt file, google/bing/yahoo would index countless repetitive non-important files and my site would suffer in search engine ranking. In our case, we host GPX/KML files and textual cuesheets for driving and biking. If that stuff is indexed, our sites' relevant keywords are "left", "right" and GPS timestamp fragments like "0z01". So, use it wisely, but don't abandon its usage altogether.
- GoodIntentions 15y agoWow. What arrogance. A Good reason to honeypot if you aren't already. It's expected. It's polite. Respect the site owner's published policy or expect to get IP banned like any other script kiddie because when a site admin see you ripping content, he isn't thinking "yay! archive team is here to do a free backup!" he thinks you are stealing his shit.
- edu 15y agoThen you should block their crawler :)
- eli 15y agoshrug Things like /search?q= are in my robots.txt because crawling those pages is just a waste of everyone's time and resources.
- jbhelms 15y agoThe only thing I have ever used robots.txt for is to stop from leaking pagerank. I have a folder called /redirect/ and i exclude that folder in my robots.txt. I then link to sites like this /redirect/?l=www.mysite.com Anything I don't want archived, I put behind a login wall.