7 ms·

The way I read it... the Wayback Machine did not allow archives to be taken down arbitrarily -- but a subsequent targeted robots.txt exclusion of the Wayback Ma

The way I read it... the Wayback Machine did not allow archives to be taken down arbitrarily -- but a subsequent targeted robots.txt exclusion of the Wayback Machine could render prior archives of that website moot? (Because the Wayback Machine starts from scratch each time?)

ry_ry 8y ago
If that's the case you could buy up defunct domains, exclude everything via robots.txt and selectively purge sites from archive.org That seems like a pretty glaring flaw in something designed to create an enduring record.
- Piskvorrr 8y ago
  Yup, that already happened in the past. For some reason, this is apparently a feature, not a bug.
  - TeMPOraL 8y ago
    From what I understand, this is the deal they make to avoid getting sued by everyone for copyright violation.
- proaralyst 8y ago
  I don't think they purge the archives, I think they just don't serve them on the wayback machine.
  - pronoiac 8y ago
    Yes. Instead of deleting anything, I think the Archive tends to mark stuff as "do not show this for a few decades."
    - jake-low 8y ago
      Do you have a source for this? I didn’t know that but it’s very interesting – a good compromise between the interests of current website owners[1] and future historians. [1]: Sure, some people just want to hide embarrassing or incriminating content, but there’s also cases where someone is being stalked or harassed based on things they shared online, and hiding those things from Archive users may mitigate that.
      - aepiepaey 8y ago
        Generally when items are "taken down" from the Internet Archive, they just stop being published, and are not deleted. I don't think it's mentioned in an official document, but it's usually referred to as "darking". It probably safe to assume that the same concept applies to the Wayback Machine as to the rest of IA. Edit: Here's a page that indirectly conveys some information about it: https://archive.org/details/IA_books_QA_codes https://archive.org/details/IA_books_QA_codes
        deleted 8y ago
        [deleted]
      - pronoiac 8y ago
        I thought I'd read it in a blog post by Jason Scott at textfiles.com, but I couldn't find a reference quickly. It could have come from conversation, as I've visited the Internet Archive a few times.
  - pbhjpbhj 8y ago
    Which won't be legal, it seems if serving EU based users, and keep PII on any other EU users.
- klez 8y ago
  They seem to have had this problem in the past and decided to skirt around it by ignoring robots.txt a year ago[0]. Does anybody know what happened to revert this decision? [0]https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
  - icebraining 8y ago
    As per that post, they only ignored robots.txt for .gov and .mil sites.
    - klez 8y ago
      Yes, but it also says > We are now looking to do this more broadly. That's the part I'm asking about.
      - icebraining 8y ago
        Right, but it doesn't mean they reverted it, they are probably still looking into it.
    - aero-002 8y ago
      IA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.
- deleted 8y ago
  [deleted]
- bryanrasmussen 8y ago
  For reference I've submitted a post on the Danish net archives https://news.ycombinator.com/item?id=16919264 https://news.ycombinator.com/item?id=16919264 which by law will archive all of any 'Danish' site and ignore robots.txt exclusions. from the faq: http://netarkivet.dk/in-english/faq/#anchor8 http://netarkivet.dk/in-english/faq/#anchor8 8. Do you respect robots.txt? No, we do not. When we collect the Danish part of the internet we ignore the so-called robots.txt directives. Studies from 2003-2004 showed that many of the truly important web sites (e.g. news media, political parties) had very stringent robots.txt directives. If we follow these directives, very little or nothing at all will be archived from those websites. Therefore, ignoring robots.txt is explicitly mentioned in the commentary to the law as being necessary in order to collect all relevant material I wonder if there are any other national archives of the internet that do the same.
  - Stranger43 8y ago
    The relatively clear mandate for collecting all published materials given to the danish royal libary by the danish constitution is kind of rare in it's scope and status. https://www.bl.uk/collection-guides/uk-web-archive https://www.bl.uk/collection-guides/uk-web-archive describe the much more limited aproach taken by the british library much later in time, but might extend to a similar scope.
    - bryanrasmussen 8y ago
      That web domain dataset they get from the internet archive is interesting in light of the current discussion, in that I am supposing it probably has .uk content that has been removed from the actual internet archive by robots.txt changes. I think if I were running a national or internationally mandated archiving initiative I would basically want to take in content from Internet Archive, and not remove things, and probably it would be less expensive that way than having my own crawler.
- dTal 8y ago
  It's actually very clever. The key is it works both ways. By respecting the live robots.txt, and only the live one, data hiding must be an active process requested on an ongoing basis by a live entity. As soon as the entity goes defunct, any previously scraped data is automatically republished. Thus archive.org is protected from lawsuit by any extant organisaton, yet in the long run still archives everything it reasonably can.
- CM30 8y ago
  It is a glaring flaw. It's meant a lot of sites ended up wiped out of the archive (or at least made inaccessible) simply because their domain has expired and the domain squatters blocked the empty domain from being indexed. The solution (which the Internet Archive really needs to implement) is to look at the domain registration data or something, and then only remove content if the same owner updated the robots.txt file. If not, then just disallow archiving any new content, since the new domain owner usually has no right to decide what happens to the old site content.
downandout 8y ago
Because the Wayback Machine starts from scratch each time? I don’t think that’s it...it’s not a technical thing. Deleting all archives must be a courtesy they extend to anyone that specifically denies access to Wayback Machine in their robots.txt. Does anyone know if this is documented? If so, why didn’t her lawyers just carry out the robots.txt technique and not even bother contacting them? Most importantly, why would they have such a policy? This is all very odd.
- simias 8y ago
  Do we know that the archive was deleted or merely taken offline? The blog post says "excluded" which to me imply that they still have the data, it's just not publicly available. I hang around in emulation circles and there's been some talk in the past few weeks because some Nintendo ROM archives had been taken offline from archive.org but people soon figured out that they could still access them by tinkering with the URL. The situation is a bit different here though.
  - downandout 8y ago
    I assume it isn't deleted...I misspoke. But the net result for the public is the same.
htgb 8y ago
> (Because the Wayback Machine starts from scratch each time?) I don't know what "start from scratch" would mean – the point is that each site is sampled many times throughout history. That said, it is very odd that a current change in robots.txt would prevent looking at old samples. And that's indeed what it looks like [1]: > Page cannot be displayed due to robots.txt. [1] https://web.archive.org/web/*/blog.reidreport.com https://web.archive.org/web/*/blog.reidreport.com
- pbhjpbhj 8y ago
  I'd imagine they're in a dodgy copyright situation and so guard against it by being conservative wrt robots.txt. The robots.txt shows a positive assertion that parts of a site should be excluded from being used by automated systems. In most cases I imagine WBM does not have permission of the owner to keep a duplicate of the site, it's certainly tortuous in UK law. Sites that don't change their robots.txt are probably highly correlated with sites that don't sue for the infringement.
Natsu 8y ago
My understanding of their robots.txt policy is that it only removes the public archive of the site and they still have a private archive. That said, I'm working from memory here and I wasn't able to find anything in their FAQ to confirm this, so it's possible I've misremembered something, though I did find some forum posts saying that formerly-available archives returns a 403 instead of a 404, which might provide some evidence for this.
delinka 8y ago
I don’t read it like that. I think it’s ‘We can no longer crawl and archive the site since robots.txt changed to exclude us. Archives from before that exclusion are still available (though not publicly.)’ Anything otherwise obviates the entire mission of the Wayback Machine.
- skywhopper 8y ago
  Fighting a bunch of court cases to keep perfect purity in their principles would cost too much, when the number of these exclusions is quite small. Better to provide an easy out to a few whiny people like this author, than to spend all their money fighting every edge case on principle and in so doing bail on the entire mission of the organization.
  - rhapsodic 8y ago
    >Fighting a bunch of court cases to keep perfect purity in their principles would cost too much, when the number of these exclusions is quite small. Better to provide an easy out to a few whiny people like this author, than to spend all their money fighting every edge case on principle and in so doing bail on the entire mission of the organization. She's a prominent left-wing television personality. Would you be so accommodating if it were, say, Tucker Carlson trying to scrub embarrassing information about himself from the wayback machine?
    - danso 8y ago
      Yes
jrochkind1 8y ago
Yeah, it doesn't make a lot of sense to me. They say they "declined to take down the archives"-- but they didn't in fact do this at all, they just insisted a request to take down the archives come in the form of a robots.txt, and they automatically and without review comply with all such requests in the form of a robots.txt. They don't in fact ever decline to take down any archives, if the request is properly given as a robots.txt. I don't know why they bothered making statements about "declining to take down the archives" in the first place (to the journalist or to us), making comments about "Reid’s being a journalist (a very high-profile one, at that) and the journalistic nature of the blog archive" -- they did not in fact "decline to take down the archives" at all. The "journalistic nature of the archive" was in fact irrelevant. They took em down. They are down.
- ballenf 8y ago
  There may be one slight (and practically irrelevant) difference: the blog posts are still archived by the wayback machine, but it refused to serve them due to the robots.txt file. If that file were to be removed, presumably the archive would again be served up upon request. The lawyers were asking for the archive store itself to be wiped.
  - flycaliguy 8y ago
    Yeah, I've come to appreciate that their frustrating robot.txt policy is actually a wise way for them to avoid chipping away at the complete archive. Obviously they have a mission that stretches far beyond the lifespan of todays internet users and the relevance of robot.txt files.
- fixed_carbon 8y ago
  My interpretation of what happened is that Ms. Reid's lawyers requested that specific posts within a full archive be removed from the archive. In other words, they weren't asking for removal of the entire archive. They just wanted a "sanitized" version to be accessible to the public. The Internet Archive has a mechanism for doing this, as I understand it. It involves asserting copyright over the material in question and essentially "making a case" for removal. IA decided the case they made didn't pass muster, and denied specific removal on those grounds, which is why they mention "journalistic nature of the archive" and so forth. But that's entirely orthogonal to their policy of treating active maintenance of robots.txt as indicative of positive copyright assertion over the contents of an entire domain -- which Ms. Reid's team appears to have taken as a fallback position. They couldn't get the sanitized archive they wanted, so they just made the whole thing invisible.
mherdeg 8y ago
In fact they used to document this behavior at https://web.archive.org/web/20150411133228/https://archive.org/about/exclude.php https://web.archive.org/web/20150411133228/https://archive.o... but they removed the documentation in mid-2015: ""To remove your site from the Wayback Machine, place a robots.txt file at the top level of your site (e.g. www.yourdomain.com/robots.txt). The robots.txt file will do two things: 1. It will remove documents from your domain from the Wayback Machine. 2. It will tell us not to crawl your site in the future. To exclude the Internet Archive’s crawler (and remove documents from the Wayback Machine) while allowing all other robots to crawl your site, your robots.txt file should say: User-agent: ia_archiver Disallow: /"" Their current documentation no longer says that they stop displaying old archives automatically in the presence of an ia_archiver Disallow directive, but I have not experimented about whether they still actually do this anyway. Just guessing wildly about similar issues, I know that news organizations which have a publicly documented "unpublish" policy tend to get that policy used aggressively by reputation management firms and the like.

ry_ry 8y ago

If that's the case you could buy up defunct domains, exclude everything via robots.txt and selectively purge sites from archive.org That seems like a pretty glaring flaw in something designed to create an enduring record.

Piskvorrr 8y ago
Yup, that already happened in the past. For some reason, this is apparently a feature, not a bug.
- TeMPOraL 8y ago
  From what I understand, this is the deal they make to avoid getting sued by everyone for copyright violation.
proaralyst 8y ago
I don't think they purge the archives, I think they just don't serve them on the wayback machine.
- pronoiac 8y ago
  Yes. Instead of deleting anything, I think the Archive tends to mark stuff as "do not show this for a few decades."
  - jake-low 8y ago
    Do you have a source for this? I didn’t know that but it’s very interesting – a good compromise between the interests of current website owners[1] and future historians. [1]: Sure, some people just want to hide embarrassing or incriminating content, but there’s also cases where someone is being stalked or harassed based on things they shared online, and hiding those things from Archive users may mitigate that.
    - aepiepaey 8y ago
      Generally when items are "taken down" from the Internet Archive, they just stop being published, and are not deleted. I don't think it's mentioned in an official document, but it's usually referred to as "darking". It probably safe to assume that the same concept applies to the Wayback Machine as to the rest of IA. Edit: Here's a page that indirectly conveys some information about it: https://archive.org/details/IA_books_QA_codes https://archive.org/details/IA_books_QA_codes
      - deleted 8y ago
        [deleted]
    - pronoiac 8y ago
      I thought I'd read it in a blog post by Jason Scott at textfiles.com, but I couldn't find a reference quickly. It could have come from conversation, as I've visited the Internet Archive a few times.
- pbhjpbhj 8y ago
  Which won't be legal, it seems if serving EU based users, and keep PII on any other EU users.
klez 8y ago
They seem to have had this problem in the past and decided to skirt around it by ignoring robots.txt a year ago[0]. Does anybody know what happened to revert this decision? [0]https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- icebraining 8y ago
  As per that post, they only ignored robots.txt for .gov and .mil sites.
  - klez 8y ago
    Yes, but it also says > We are now looking to do this more broadly. That's the part I'm asking about.
    - icebraining 8y ago
      Right, but it doesn't mean they reverted it, they are probably still looking into it.
  - aero-002 8y ago
    IA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.
deleted 8y ago
[deleted]
bryanrasmussen 8y ago
For reference I've submitted a post on the Danish net archives https://news.ycombinator.com/item?id=16919264 https://news.ycombinator.com/item?id=16919264 which by law will archive all of any 'Danish' site and ignore robots.txt exclusions. from the faq: http://netarkivet.dk/in-english/faq/#anchor8 http://netarkivet.dk/in-english/faq/#anchor8 8. Do you respect robots.txt? No, we do not. When we collect the Danish part of the internet we ignore the so-called robots.txt directives. Studies from 2003-2004 showed that many of the truly important web sites (e.g. news media, political parties) had very stringent robots.txt directives. If we follow these directives, very little or nothing at all will be archived from those websites. Therefore, ignoring robots.txt is explicitly mentioned in the commentary to the law as being necessary in order to collect all relevant material I wonder if there are any other national archives of the internet that do the same.
- Stranger43 8y ago
  The relatively clear mandate for collecting all published materials given to the danish royal libary by the danish constitution is kind of rare in it's scope and status. https://www.bl.uk/collection-guides/uk-web-archive https://www.bl.uk/collection-guides/uk-web-archive describe the much more limited aproach taken by the british library much later in time, but might extend to a similar scope.
  - bryanrasmussen 8y ago
    That web domain dataset they get from the internet archive is interesting in light of the current discussion, in that I am supposing it probably has .uk content that has been removed from the actual internet archive by robots.txt changes. I think if I were running a national or internationally mandated archiving initiative I would basically want to take in content from Internet Archive, and not remove things, and probably it would be less expensive that way than having my own crawler.
dTal 8y ago
It's actually very clever. The key is it works both ways. By respecting the live robots.txt, and only the live one, data hiding must be an active process requested on an ongoing basis by a live entity. As soon as the entity goes defunct, any previously scraped data is automatically republished. Thus archive.org is protected from lawsuit by any extant organisaton, yet in the long run still archives everything it reasonably can.
CM30 8y ago
It is a glaring flaw. It's meant a lot of sites ended up wiped out of the archive (or at least made inaccessible) simply because their domain has expired and the domain squatters blocked the empty domain from being indexed. The solution (which the Internet Archive really needs to implement) is to look at the domain registration data or something, and then only remove content if the same owner updated the robots.txt file. If not, then just disallow archiving any new content, since the new domain owner usually has no right to decide what happens to the old site content.

downandout 8y ago

Because the Wayback Machine starts from scratch each time? I don’t think that’s it...it’s not a technical thing. Deleting all archives must be a courtesy they extend to anyone that specifically denies access to Wayback Machine in their robots.txt. Does anyone know if this is documented? If so, why didn’t her lawyers just carry out the robots.txt technique and not even bother contacting them? Most importantly, why would they have such a policy? This is all very odd.

simias 8y ago
Do we know that the archive was deleted or merely taken offline? The blog post says "excluded" which to me imply that they still have the data, it's just not publicly available. I hang around in emulation circles and there's been some talk in the past few weeks because some Nintendo ROM archives had been taken offline from archive.org but people soon figured out that they could still access them by tinkering with the URL. The situation is a bit different here though.
- downandout 8y ago
  I assume it isn't deleted...I misspoke. But the net result for the public is the same.

htgb 8y ago

> (Because the Wayback Machine starts from scratch each time?) I don't know what "start from scratch" would mean – the point is that each site is sampled many times throughout history. That said, it is very odd that a current change in robots.txt would prevent looking at old samples. And that's indeed what it looks like [1]: > Page cannot be displayed due to robots.txt. [1] https://web.archive.org/web/*/blog.reidreport.com https://web.archive.org/web/*/blog.reidreport.com

pbhjpbhj 8y ago
I'd imagine they're in a dodgy copyright situation and so guard against it by being conservative wrt robots.txt. The robots.txt shows a positive assertion that parts of a site should be excluded from being used by automated systems. In most cases I imagine WBM does not have permission of the owner to keep a duplicate of the site, it's certainly tortuous in UK law. Sites that don't change their robots.txt are probably highly correlated with sites that don't sue for the infringement.

Natsu 8y ago

My understanding of their robots.txt policy is that it only removes the public archive of the site and they still have a private archive. That said, I'm working from memory here and I wasn't able to find anything in their FAQ to confirm this, so it's possible I've misremembered something, though I did find some forum posts saying that formerly-available archives returns a 403 instead of a 404, which might provide some evidence for this.

delinka 8y ago

I don’t read it like that. I think it’s ‘We can no longer crawl and archive the site since robots.txt changed to exclude us. Archives from before that exclusion are still available (though not publicly.)’ Anything otherwise obviates the entire mission of the Wayback Machine.

skywhopper 8y ago
Fighting a bunch of court cases to keep perfect purity in their principles would cost too much, when the number of these exclusions is quite small. Better to provide an easy out to a few whiny people like this author, than to spend all their money fighting every edge case on principle and in so doing bail on the entire mission of the organization.
- rhapsodic 8y ago
  >Fighting a bunch of court cases to keep perfect purity in their principles would cost too much, when the number of these exclusions is quite small. Better to provide an easy out to a few whiny people like this author, than to spend all their money fighting every edge case on principle and in so doing bail on the entire mission of the organization. She's a prominent left-wing television personality. Would you be so accommodating if it were, say, Tucker Carlson trying to scrub embarrassing information about himself from the wayback machine?
  - danso 8y ago
    Yes

jrochkind1 8y ago

Yeah, it doesn't make a lot of sense to me. They say they "declined to take down the archives"-- but they didn't in fact do this at all, they just insisted a request to take down the archives come in the form of a robots.txt, and they automatically and without review comply with all such requests in the form of a robots.txt. They don't in fact ever decline to take down any archives, if the request is properly given as a robots.txt. I don't know why they bothered making statements about "declining to take down the archives" in the first place (to the journalist or to us), making comments about "Reid’s being a journalist (a very high-profile one, at that) and the journalistic nature of the blog archive" -- they did not in fact "decline to take down the archives" at all. The "journalistic nature of the archive" was in fact irrelevant. They took em down. They are down.

ballenf 8y ago
There may be one slight (and practically irrelevant) difference: the blog posts are still archived by the wayback machine, but it refused to serve them due to the robots.txt file. If that file were to be removed, presumably the archive would again be served up upon request. The lawyers were asking for the archive store itself to be wiped.
- flycaliguy 8y ago
  Yeah, I've come to appreciate that their frustrating robot.txt policy is actually a wise way for them to avoid chipping away at the complete archive. Obviously they have a mission that stretches far beyond the lifespan of todays internet users and the relevance of robot.txt files.
fixed_carbon 8y ago
My interpretation of what happened is that Ms. Reid's lawyers requested that specific posts within a full archive be removed from the archive. In other words, they weren't asking for removal of the entire archive. They just wanted a "sanitized" version to be accessible to the public. The Internet Archive has a mechanism for doing this, as I understand it. It involves asserting copyright over the material in question and essentially "making a case" for removal. IA decided the case they made didn't pass muster, and denied specific removal on those grounds, which is why they mention "journalistic nature of the archive" and so forth. But that's entirely orthogonal to their policy of treating active maintenance of robots.txt as indicative of positive copyright assertion over the contents of an entire domain -- which Ms. Reid's team appears to have taken as a fallback position. They couldn't get the sanitized archive they wanted, so they just made the whole thing invisible.

mherdeg 8y ago

In fact they used to document this behavior at https://web.archive.org/web/20150411133228/https://archive.org/about/exclude.php https://web.archive.org/web/20150411133228/https://archive.o... but they removed the documentation in mid-2015: ""To remove your site from the Wayback Machine, place a robots.txt file at the top level of your site (e.g. www.yourdomain.com/robots.txt). The robots.txt file will do two things: 1. It will remove documents from your domain from the Wayback Machine. 2. It will tell us not to crawl your site in the future. To exclude the Internet Archive’s crawler (and remove documents from the Wayback Machine) while allowing all other robots to crawl your site, your robots.txt file should say: User-agent: ia_archiver Disallow: /"" Their current documentation no longer says that they stop displaying old archives automatically in the presence of an ia_archiver Disallow directive, but I have not experimented about whether they still actually do this anyway. Just guessing wildly about similar issues, I know that news organizations which have a publicly documented "unpublish" policy tend to get that policy used aggressively by reputation management firms and the like.