5 ms·
The rationale is weak. Some data is simply not worth indexing, and not worth serving up to bots. The flipside is: your crawler doesn't need to fetch everything
by smosher 15y ago
The rationale is weak. Some data is simply not worth indexing, and not worth serving up to bots. The flipside is: your crawler doesn't need to fetch everything on my site, and I'd be happy to ban all non-conforming bots site-wide.
It's not just about the functionality, but also a show of good faith and basic respect. If you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all.
robots.txt isn't perfect. Ideally a web server would be configured to deny bots access to restricted content via some sort of dnsbl mechanism (or CPAN/whatever module.) Or do both and ban the non-conforming site-wide.
The above notwithstanding, I'm voting for this article. It doesn't betray the usual cowardice by hiding the assertion behind the presumptuous Why.
- panza 15y ago"Some data is simply not worth indexing, and not worth serving up to bots." The broader points being made in the article are that the value of information is determined by the visitor, and that the burden of keeping the site up and running should fall on the host. By that reasoning, the presence of a ROBOTS.TXT signifies "damage" or "temporary madness", and should therefore be ignored. Also, keep in mind where this article came from - the Archive Team are bloody-minded about preserving information.
- drivingmenuts 15y agoOK, wait. If the data has value to the user, shouldn't the user be paying the host for the cost of making that data available (or perhaps considerably more, if it has a lot of value to the user)?
- sbierwagen 15y agoExcellent. How much are you paying ycombinator for your account? If the answer is "nothing", then I guess you've just told me that all your comments are useless, and I should ignore them.
- sbierwagen 15y agoI'm committing the classic mistake of talking about the karma system here, but what, precisely, is wrong with my statement? Drivingmenuts says the user should pay for hosting data. HN is hosting his data, but he doesn't pay for it. Is there some error in my logic, here, some flaw in my conclusion?
- smosher 15y agothe value of information is determined by the visitor That may be so in a sense, unless the visitor has no means of making the call -- or does so poorly. I promise you that every robots.txt ignoring bot makes terrible judgments in that regard. (I'm not trying to overstate anything here; those that abide it make, on average, slightly less terrible judgments.) But it's not just the visitor who gets to make judgment calls. The value of serving those pages is something the host can decide. Belligerent drunks who abuse the staff aren't allowed in the coffee shop, and if they do it often enough they're not allowed back in when they sober up either.
- chalst 15y agothe Archive Team are bloody-minded about preserving information They won't do a good job of that if they encourage too many people to make IP blocks of their machines.
- sbierwagen 15y agoAT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.
- chalst 15y agoDo all of them ignore robots.txt?
- sbierwagen 15y agoWell, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to: User-agent: * Disallow: / Does that count? 1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/ http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/
- dalke 15y agoHere's a concrete example of a site not worth archiving - back in the early days of the web (1994?) a friend of mine wrote a 4 function calculator. Each button was a hyperlink, so "34-12" meant going to the pages "?", "?3", "?34", "?34-", "?34-1", and "?34-12". That's an infinite number of web pages which should not be indexed.
- thyrsus 15y agoI remember some anti-spam vigilante code that told legitimate bots to go away with robots.txt, told humans what the game was about, and attacked e-mail address scrapers by presenting an infinite number of fake e-mail addresses. "Click here for 1000 more e-mail addresses that will never bother a human!".
- alanh 15y agoI love it!
- rachelbythebay 15y agoNot all of them were fake. Some of them were spam traps which never appeared anywhere else. Some of those spam traps were also coded so you could track down when they were served to connect the spiders to the spewers. At least, that's the way I used to roll.
- smosher 15y agoAh very good! I was trying to think of a short example that wasn't contrived with this very behaviour.
- chalst 15y agoIf you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all. Quite. My robots.txt reads: User-agent: * Disallow: /cgi-bin/ Disallow: /scratch/ I do not believe, as the archiveteam.org seem to think I do, that I have either stopped bots interfering with CGI scripts or spidering semi-private, temporary content. Indeed, my server logs show clearly enough that bots do these things all the time. Instead, the bot has violated my site's policy and its IP address will be considered for an iptables ban. It is a matter of which bots I consider fair game for rough treatment. This is a common attitude to robots.txt, I think, and it does not reflect well on archiveteam.org's competence that they seem to be unaware of it.
- G_Morgan 15y agoYeah what this really needs is a bot block filter service where bots that do not conform to expectations can be bulk banned. It is too much effort to do this manually but a larger service can work.