3 ms·

Archiveteam member here. Robots.txt is useful for a site to avoid having search-engine spiders to blindly wander down an infinite hallway. Archiveteam project

Archiveteam member here. Robots.txt is useful for a site to avoid having search-engine spiders to blindly wander down an infinite hallway. Archiveteam projects are different because we run closely-monitored, highly targeted crawls. If a site has an infinite hallway in it, we'll notice that and exclude it while taking steps to replicate enough of it to retain its valuable attributes. We also do our best to avoid retrieving content more times than necessary. If you operate a database, we try to get enough pages to replicate a good portion of the underlying data.

derpy 15y ago
You do try to contact the "targets" sometimes though?
- sbierwagen 15y ago
  Sure. We ask them to give us database dumps, when they refuse, for some crazy reason, we just download their website.
  - dspillett 15y ago
    I can't say I approve of the "if the crazy fools won't give us what we want, we'll just take it some other way" attitude. As the "some crazy reason" might be that the database contains data that is not publicly addressable for a reason and sanitising the data-set could be non-zero effort operation, letting you crawl the public pages and do some work at your side would be the sensible route from most points of view. Heck, even if there is definitely not anything not already displayed to the public in the DB, letting you go ahead and get the information the hard way is likely to be preferable to lifting a finger to send you a DB dump. Sort of like a cat's attitude to playing fetch: You want it? You go get it. I've got more important things to do.
migrantgeek 15y ago
"Archiveteam projects are different because we run closely-monitored, highly targeted crawls." Maybe, but does that make robots.txt useless? I don't think so. Your app may do the right thing but who's to say another app isn't? Why should someone ignore a tool because Archiveteam claims to do the right thing? The file has more uses than just blocking a particular URL. What about adding a crawler delay for a site on a shared server that gets slammed sometimes? Or what about blocking a particular user-agent that misbehaves? If someone shoots themselves in the foot with their robots.txt, it's their foot and their gun.
bxr 15y ago
Fair enough. I've always seen my robots.txt files as a friendly way of saying "you probably don't need to worry about this stuff" to crawlers. That's not your use case, and I'm happy to have my "Disallowed" content to be crawled, I expect poorly written bots to do so anyway. If I actually want to prevent browsing/crawling of something, I wont ask politely I'm just going to throw up 403s.

derpy 15y ago

You do try to contact the "targets" sometimes though?

sbierwagen 15y ago
Sure. We ask them to give us database dumps, when they refuse, for some crazy reason, we just download their website.
- dspillett 15y ago
  I can't say I approve of the "if the crazy fools won't give us what we want, we'll just take it some other way" attitude. As the "some crazy reason" might be that the database contains data that is not publicly addressable for a reason and sanitising the data-set could be non-zero effort operation, letting you crawl the public pages and do some work at your side would be the sensible route from most points of view. Heck, even if there is definitely not anything not already displayed to the public in the DB, letting you go ahead and get the information the hard way is likely to be preferable to lifting a finger to send you a DB dump. Sort of like a cat's attitude to playing fetch: You want it? You go get it. I've got more important things to do.

migrantgeek 15y ago

"Archiveteam projects are different because we run closely-monitored, highly targeted crawls." Maybe, but does that make robots.txt useless? I don't think so. Your app may do the right thing but who's to say another app isn't? Why should someone ignore a tool because Archiveteam claims to do the right thing? The file has more uses than just blocking a particular URL. What about adding a crawler delay for a site on a shared server that gets slammed sometimes? Or what about blocking a particular user-agent that misbehaves? If someone shoots themselves in the foot with their robots.txt, it's their foot and their gun.

bxr 15y ago

Fair enough. I've always seen my robots.txt files as a friendly way of saying "you probably don't need to worry about this stuff" to crawlers. That's not your use case, and I'm happy to have my "Disallowed" content to be crawled, I expect poorly written bots to do so anyway. If I actually want to prevent browsing/crawling of something, I wont ask politely I'm just going to throw up 403s.