4 ms·

from the linked page > so no one will ever be able to change the past just because there is no digital record of it. The Web needs a memory, the ability to loo

from the linked page > so no one will ever be able to change the past just because there is no digital record of it. The Web needs a memory, the ability to look back. I thought the issue is/was that site owners (original, or purchasing the domain afterwards) could via a robots.txt remove their site from the archive? or has this changed and now no matter what happens if the archive crawls a site on a date it stays no matter if 10 years down the road somebody buys the domain and decides to retroactively erase everything?

noonshine 10y ago
Ok, but, I'm very thankful I was able to remove the site I made 20 years ago when I was 14.
db48x 10y ago
Adding a robots.txt file to a domain doesn't cause them to delete their archive of it, only to hide it.
- eljimmy 10y ago
  Which is one and the same to the public. IMO, they need to stop applying robots.txt retroactively if they want to be considered a valid archive.
  - Smerity 10y ago
    Key point given the current climate, if the Trump presidency adds a restrictive robots.txt to all .gov domains they will prevent the Internet Archive Wayback Machine from showing history on any of those domains. Not only would this obliterate public access to the Obama, Bush, Clinton era government websites in the archive, it'd prevent the use of the Wayback Machine for keeping track of Trump's shifting agendas, as demonstrated on his web domain recently.
    - ghaff 10y ago
      Government documents including, I assume, web pages are in the public domain.
      - dragonwriter 10y ago
        That's irrelevant; the issue isn't copyright, but the implementation of the Internet Archive, which applies robots.txt retroactively to archived versions of pages as well as respecting it currently.
        jeff_tyrrill 10y ago
        But the reason the Internet Archive applies robots.txt retroactively is copyright, as explained in a sibling to the grandparent.
  - ghaff 10y ago
    The problem is that the Internet Archive exists on legally shaky ground. Neither they nor anyone else has a right to archive copyrighted web content and display it to the public. They manage to continue doing so in part because they're clearly non-commercial. They also manage to continue doing so because they voluntarily respond to robots.txt, even retroactively. Libraries/archives have no special exemption from copyright law, which is actually a good thing, because otherwise libraries would presumably need to be licensed in some way by the government to qualify for special treatment.
    - OJFord 10y ago
      > the Internet Archive exists on legally shaky ground Not least because of the EU's 'right to be forgotten'.
      - db48x 10y ago
        That doesn't seem likely to apply.
    - CM30 10y ago
      Why not look at WHOIS information when getting an update, and then class a site as 'different' based on whether that changes? In most cases, a new domain owner usually means the site isn't the same as the earlier versions. You'd then just have to stop the archive indexing/showing content after the WHOIS information changed, while leaving the stuff before it intact. Maybe you'd then have a nice form to report pages you want removed/hidden (for the edge cases), or even a seperate robots.txt/meta declaration you can make confirming you're the same person that owns the site. After all, most of the reasons why sites go missing aren't deliberate attempts to rewrite history, but domain squatters not wanting holding pages indexed. Feels like it'd be so easy to implement robots.txt in a more logical way on the Internet Archive.
      - db48x 10y ago
        It's been suggested, but there's no way to automatically do it correctly. The whois info might be anonymized, in which case a change means nothing at all. It might just be someone's name and address, with no way of verifying who that someone works for. Also meaningless. Better just to default to something safe, and spend your manpower on something more important.
- deleted 10y ago
  [deleted]
makomk 10y ago
Not just the site owners either. I'm guessing not many people on here saw the early days of Gamergate, but one of the things that stoked the fire was the Internet Archive kindly erasing their archives of other websites' pages discussing the mess on request from a person being discussed after their hosting providers were leaned on to give them the boot, changing the past to fit the correct official narrative. It's one of the reasons why no-one involved with that uses them for archiving anything. (Wasn't doxing or anything like that, just explaining what happened was enough.) In our modern hyperpoliticized era, one has to wonder if the Internet Archive is actually trustworthy for anything contentious anymore.
alexpoulsen 10y ago
So someone* should make a clone of archive.org but built on Tor. Maybe call it Torchive. And with no hiding because of robots.txt and possibly prevent users from looking at/editing the data on their server so a person who wants to clear themselves can't flood the net with nodes wanting data on this person and then wipe them all. *someone who has the skills to make it, because I don't, at least right now

noonshine 10y ago

Ok, but, I'm very thankful I was able to remove the site I made 20 years ago when I was 14.

db48x 10y ago

Adding a robots.txt file to a domain doesn't cause them to delete their archive of it, only to hide it.

eljimmy 10y ago
Which is one and the same to the public. IMO, they need to stop applying robots.txt retroactively if they want to be considered a valid archive.
- Smerity 10y ago
  Key point given the current climate, if the Trump presidency adds a restrictive robots.txt to all .gov domains they will prevent the Internet Archive Wayback Machine from showing history on any of those domains. Not only would this obliterate public access to the Obama, Bush, Clinton era government websites in the archive, it'd prevent the use of the Wayback Machine for keeping track of Trump's shifting agendas, as demonstrated on his web domain recently.
  - ghaff 10y ago
    Government documents including, I assume, web pages are in the public domain.
    - dragonwriter 10y ago
      That's irrelevant; the issue isn't copyright, but the implementation of the Internet Archive, which applies robots.txt retroactively to archived versions of pages as well as respecting it currently.
      - jeff_tyrrill 10y ago
        But the reason the Internet Archive applies robots.txt retroactively is copyright, as explained in a sibling to the grandparent.
- ghaff 10y ago
  The problem is that the Internet Archive exists on legally shaky ground. Neither they nor anyone else has a right to archive copyrighted web content and display it to the public. They manage to continue doing so in part because they're clearly non-commercial. They also manage to continue doing so because they voluntarily respond to robots.txt, even retroactively. Libraries/archives have no special exemption from copyright law, which is actually a good thing, because otherwise libraries would presumably need to be licensed in some way by the government to qualify for special treatment.
  - OJFord 10y ago
    > the Internet Archive exists on legally shaky ground Not least because of the EU's 'right to be forgotten'.
    - db48x 10y ago
      That doesn't seem likely to apply.
  - CM30 10y ago
    Why not look at WHOIS information when getting an update, and then class a site as 'different' based on whether that changes? In most cases, a new domain owner usually means the site isn't the same as the earlier versions. You'd then just have to stop the archive indexing/showing content after the WHOIS information changed, while leaving the stuff before it intact. Maybe you'd then have a nice form to report pages you want removed/hidden (for the edge cases), or even a seperate robots.txt/meta declaration you can make confirming you're the same person that owns the site. After all, most of the reasons why sites go missing aren't deliberate attempts to rewrite history, but domain squatters not wanting holding pages indexed. Feels like it'd be so easy to implement robots.txt in a more logical way on the Internet Archive.
    - db48x 10y ago
      It's been suggested, but there's no way to automatically do it correctly. The whois info might be anonymized, in which case a change means nothing at all. It might just be someone's name and address, with no way of verifying who that someone works for. Also meaningless. Better just to default to something safe, and spend your manpower on something more important.
deleted 10y ago
[deleted]

makomk 10y ago

Not just the site owners either. I'm guessing not many people on here saw the early days of Gamergate, but one of the things that stoked the fire was the Internet Archive kindly erasing their archives of other websites' pages discussing the mess on request from a person being discussed after their hosting providers were leaned on to give them the boot, changing the past to fit the correct official narrative. It's one of the reasons why no-one involved with that uses them for archiving anything. (Wasn't doxing or anything like that, just explaining what happened was enough.) In our modern hyperpoliticized era, one has to wonder if the Internet Archive is actually trustworthy for anything contentious anymore.

alexpoulsen 10y ago

So someone* should make a clone of archive.org but built on Tor. Maybe call it Torchive. And with no hiding because of robots.txt and possibly prevent users from looking at/editing the data on their server so a person who wants to clear themselves can't flood the net with nodes wanting data on this person and then wipe them all. *someone who has the skills to make it, because I don't, at least right now