8 ms·

This article forgot the very worst use of robots.txt: User-agent: ia_archiver Disallow: / Those two lines mean that all content hosted on the entire site

This article forgot the very worst use of robots.txt: User-agent: ia_archiver Disallow: / Those two lines mean that all content hosted on the entire site will be blocked from the Internet Archive (archive.org) WayBack Machine, and the public will be unable to look at any previous versions of the website's content. It wipes out a public view of the past. Yeah, I'm looking at you, Washington Post: http://www.washingtonpost.com/robots.txt http://www.washingtonpost.com/robots.txt Banning access to history like that is shameful.

krapp 12y ago
The Internet Archive choosing to honor robots.txt is what's 'banning' the access. Both the request not to be crawled and the decision not to crawl are voluntary, but if the Internet Archive decided it wanted to slurp up the Washington Post tomorrow, there's not much they could do to stop it.
- Asparagirl 12y ago
  Yes, I know, I'm a member of Archive Team, and I use "wget -e robots=off --mirror …" quite a bit, and then I upload those WARC's to the IA. But major content providers like the Washington Post that explicitly choose to block their entire website and its history should be named and shamed. Authors don't get the right to go around removing their novels from public libraries just because they would rather the books be available only for pay in bookstores.
  - akerl_ 12y ago
    It's not really shameworthy to want to regulate access to your own sites, and physical metaphors work about as well here as "you wouldn't steal a car" does for piracy. The Internet Archive does wonderful work, but just because somebody doesn't want you folks crawling their content doesn't make them worthy of "naming and shaming"
    - click170 12y ago
      I was going to reply pointing out that whether or not to name and shame someone is a subjective decision which you and I do not see eye to eye on, and which generally requires quite a few people to agree with you before it becomes a problem for the shamee, but then I rembered the poor way that IA handles changes in ownership with respect to robots. When IA stops wiping out historical content due to a change of domain ownership in the now then I will have more support (and USE) for them.
      - icebraining 12y ago
        How is IA supposed to distinguish a new website from a sincere wish to delete old stuff? A change in domain registration data means nothing; I have a domain that I registered for an association in my name, and which I then sold to them (for a symbolic price), but it was only an administrative issue - the site was the same. IA is on iffy territory w.r.t. copyright as it is; if they stop respecting robots.txt, they could get into a world of hurt.
        ghaff 12y ago
        Your last sentence is key. As I understand it, there's no real legal precedent for IA which basically copies everything out there on an opt-out basis. I personally am glad they do but one of the ways they get off with it is by treading as lightly as possible, including respecting robots.txt even retroactively. They're also non-commercial, broad in scope, arguably serve a valuable scholarly function and have other characteristics that have kept them mostly out of legal hot water. But it's unclear to what degree they're legally different from a site that decided to create an archive of all comics, commercial and otherwise, and slap advertising up.
      - ersii 12y ago
        Internet Archive isn't wiping out historical content. It's just unavailable/hidden for the time being. (As long as there is a restrictive robots.txt available).
    - nitrogen 12y ago
  - chdir 12y ago
    Unfair to name & shame a private entity that doesn't want it's content to be archived.
    - hnha 12y ago
      privately owned but publicly acting!
- DanBC 12y ago
  They have explicitly denied permission to have their content slurped. Why do you think it is legal to then go ahead and slurp it?
  - krapp 12y ago
    I'm not making an argument about the legality or ethics of it one way or another - although as far as i'm aware, it's not actually illegal to ignore robots.txt. I was just pointing out that robots.txt doesn't actually do anything but ask nicely.
    - pasbesoin 12y ago
      When people can go to jail for hitting a publicly available URL, I'd question the "legality" of such activity. (I'm not making a moral argument, but rather question what lawyers and law enforcement may choose to make of a situation.) Politicians keep attempting to write evermore draconian qualifications and punishments into law for what qualifies as a "breach of terms of service". I would expect this to encompass robots.txt at some point if it does not already. Again, I'm not particularly happy about this trend, but I'll try to keep out of its path of destruction.
adventured 12y ago
Quora also disallows the ia_archiver agent. http://www.quora.com/robots.txt http://www.quora.com/robots.txt Here is their explanation (in the robots.txt file) "We opt out of the wayback machine because inclusion would allow people to discover the identity of authors who had written sensitive answers publicly and later had made them anonymous, and because it would prevent authors from being able to remove their content from the internet if they change their mind about publishing it. As far as we can tell, there is no way for sites to selectively programmatically remove content from the archive and so this is the only way for us to protect writers. If they open up an API where we can remove content from the archive when authors remove it from Quora, but leave the rest of the content archived, we would be happy to opt back in. See the page here: https://archive.org/about/exclude.php" https://archive.org/about/exclude.php" "Meanwhile, if you are looking for an older version of any content on Quora, we have full edit history tracked and accessible in product (with the exception of content that has been removed by the author). You can generally access this by clicking on timestamps, or by appending "/log" to the URL of any content page."
pekk 12y ago
No, you do NOT have the right to pound my site with requests and serve data that I decided to pull down.
- angersock 12y ago
  "serve data that I decided to pull down." If it's on their bandwidth and power, why not?
  - deleted 12y ago
    [deleted]
  - bsilvereagle 12y ago
    I think pekk meant that if he deletes a blog post, the IA is still going to serve it. "Pull down" refers to deleting content, not bandwidth usage.
    - blueskin_ 12y ago
      If you don't want it on the internet, don't post it. Assuming anything can ever be made to disappear from the internet is naive, and if people become aware you are, it'll just get Streisanded and become even more widely posted.
      - PavlovsCat 12y ago
        Things disappear from the internet, for good, all the time. Also, if people become aware someone is naive they automatically are dicks to them? Regardless of wether that information is actually of interest to anyone, just because someone wants to take something down, they should not ever be able to? Some people act and think like that that, yeah. But to accept this as the baseline of human behaviour is, well, not for me. This entitledness to watch the lives of others from the the dark may have been bred by reality TV or whatever; but it's more a personality flaw and an addiction, a useless misfiring of synapses become culture, than a cornerstone of an information age.
      - 12y ago
bufordsharkley 12y ago
The thing that really frustrates me about the Internet Archive's treatment of robots.txt: if a domain expires and the domain provider changes the robots.txt to something restrictive, the Wayback Machine will completely clear the history of the site. Even though it's very clearly not the same agent at play-- this is not the creator of the site's content. I've seen it happen, and it breaks my heart every time.
- pestaa 12y ago
  Why wouldn't it consider the archived state of robots.txt?
- blueskin_ 12y ago
  One of the reasons I like archive.today. Obviously, they lack the depth of history, but they don't censor so easily.

krapp 12y ago

The Internet Archive choosing to honor robots.txt is what's 'banning' the access. Both the request not to be crawled and the decision not to crawl are voluntary, but if the Internet Archive decided it wanted to slurp up the Washington Post tomorrow, there's not much they could do to stop it.

Asparagirl 12y ago
Yes, I know, I'm a member of Archive Team, and I use "wget -e robots=off --mirror …" quite a bit, and then I upload those WARC's to the IA. But major content providers like the Washington Post that explicitly choose to block their entire website and its history should be named and shamed. Authors don't get the right to go around removing their novels from public libraries just because they would rather the books be available only for pay in bookstores.
- akerl_ 12y ago
  It's not really shameworthy to want to regulate access to your own sites, and physical metaphors work about as well here as "you wouldn't steal a car" does for piracy. The Internet Archive does wonderful work, but just because somebody doesn't want you folks crawling their content doesn't make them worthy of "naming and shaming"
  - click170 12y ago
    I was going to reply pointing out that whether or not to name and shame someone is a subjective decision which you and I do not see eye to eye on, and which generally requires quite a few people to agree with you before it becomes a problem for the shamee, but then I rembered the poor way that IA handles changes in ownership with respect to robots. When IA stops wiping out historical content due to a change of domain ownership in the now then I will have more support (and USE) for them.
    - icebraining 12y ago
      How is IA supposed to distinguish a new website from a sincere wish to delete old stuff? A change in domain registration data means nothing; I have a domain that I registered for an association in my name, and which I then sold to them (for a symbolic price), but it was only an administrative issue - the site was the same. IA is on iffy territory w.r.t. copyright as it is; if they stop respecting robots.txt, they could get into a world of hurt.
      - ghaff 12y ago
        Your last sentence is key. As I understand it, there's no real legal precedent for IA which basically copies everything out there on an opt-out basis. I personally am glad they do but one of the ways they get off with it is by treading as lightly as possible, including respecting robots.txt even retroactively. They're also non-commercial, broad in scope, arguably serve a valuable scholarly function and have other characteristics that have kept them mostly out of legal hot water. But it's unclear to what degree they're legally different from a site that decided to create an archive of all comics, commercial and otherwise, and slap advertising up.
    - ersii 12y ago
      Internet Archive isn't wiping out historical content. It's just unavailable/hidden for the time being. (As long as there is a restrictive robots.txt available).
  - nitrogen 12y ago
- chdir 12y ago
  Unfair to name & shame a private entity that doesn't want it's content to be archived.
  - hnha 12y ago
    privately owned but publicly acting!
DanBC 12y ago
They have explicitly denied permission to have their content slurped. Why do you think it is legal to then go ahead and slurp it?
- krapp 12y ago
  I'm not making an argument about the legality or ethics of it one way or another - although as far as i'm aware, it's not actually illegal to ignore robots.txt. I was just pointing out that robots.txt doesn't actually do anything but ask nicely.
  - pasbesoin 12y ago
    When people can go to jail for hitting a publicly available URL, I'd question the "legality" of such activity. (I'm not making a moral argument, but rather question what lawyers and law enforcement may choose to make of a situation.) Politicians keep attempting to write evermore draconian qualifications and punishments into law for what qualifies as a "breach of terms of service". I would expect this to encompass robots.txt at some point if it does not already. Again, I'm not particularly happy about this trend, but I'll try to keep out of its path of destruction.

adventured 12y ago

Quora also disallows the ia_archiver agent. http://www.quora.com/robots.txt http://www.quora.com/robots.txt Here is their explanation (in the robots.txt file) "We opt out of the wayback machine because inclusion would allow people to discover the identity of authors who had written sensitive answers publicly and later had made them anonymous, and because it would prevent authors from being able to remove their content from the internet if they change their mind about publishing it. As far as we can tell, there is no way for sites to selectively programmatically remove content from the archive and so this is the only way for us to protect writers. If they open up an API where we can remove content from the archive when authors remove it from Quora, but leave the rest of the content archived, we would be happy to opt back in. See the page here: https://archive.org/about/exclude.php" https://archive.org/about/exclude.php" "Meanwhile, if you are looking for an older version of any content on Quora, we have full edit history tracked and accessible in product (with the exception of content that has been removed by the author). You can generally access this by clicking on timestamps, or by appending "/log" to the URL of any content page."

pekk 12y ago

No, you do NOT have the right to pound my site with requests and serve data that I decided to pull down.

angersock 12y ago
"serve data that I decided to pull down." If it's on their bandwidth and power, why not?
- deleted 12y ago
  [deleted]
- bsilvereagle 12y ago
  I think pekk meant that if he deletes a blog post, the IA is still going to serve it. "Pull down" refers to deleting content, not bandwidth usage.
  - blueskin_ 12y ago
    If you don't want it on the internet, don't post it. Assuming anything can ever be made to disappear from the internet is naive, and if people become aware you are, it'll just get Streisanded and become even more widely posted.
    - PavlovsCat 12y ago
      Things disappear from the internet, for good, all the time. Also, if people become aware someone is naive they automatically are dicks to them? Regardless of wether that information is actually of interest to anyone, just because someone wants to take something down, they should not ever be able to? Some people act and think like that that, yeah. But to accept this as the baseline of human behaviour is, well, not for me. This entitledness to watch the lives of others from the the dark may have been bred by reality TV or whatever; but it's more a personality flaw and an addiction, a useless misfiring of synapses become culture, than a cornerstone of an information age.
    - 12y ago

bufordsharkley 12y ago

The thing that really frustrates me about the Internet Archive's treatment of robots.txt: if a domain expires and the domain provider changes the robots.txt to something restrictive, the Wayback Machine will completely clear the history of the site. Even though it's very clearly not the same agent at play-- this is not the creator of the site's content. I've seen it happen, and it breaks my heart every time.

pestaa 12y ago
Why wouldn't it consider the archived state of robots.txt?
blueskin_ 12y ago
One of the reasons I like archive.today. Obviously, they lack the depth of history, but they don't censor so easily.