4 ms·

PSA/ranty thing: Just because something is archived in the Wayback machine, do not trust that archive.org will keep it there for all time. If you need somethin

by keenerd 12y ago

PSA/ranty thing: Just because something is archived in the Wayback machine, do not trust that archive.org will keep it there for all time. If you need something, make a local copy! A few months ago TIA changed their stance on robots.txt. They now retroactively honor robot blocks. Now any site can completely vanish from the archives. Let's say I died tomorrow. My family lets my domain slip. A squatter buys it and throws up a stock landing page, with a robots.txt that forbids spidering. TIA would delete my entire site from their index. I've already lost a few good sites to this sort of thing. If you depend on a resource, archive it yourself. edit - Official policy: https://archive.org/about/exclude.php https://archive.org/about/exclude.php If I am reading it properly, once blocked they never check later in case of a change of heart? No procedure for getting re-indexed at all?

gojomo 12y ago
A fair rant, but to correct some misperceptions: The retroactive application of robots.txt is not a new policy; it's been in place for at least 11 years, and I believe it arrived very soon after the Wayback Machine was first unveiled. An updated robots.txt does not irreversibly delete prior captures, so if the robots.txt changes again, access to previously-collected material can be re-enabled. This policy has served to minimize risk and automate the most common removal-scenario, when a webmaster wants a total opt-out of current crawling and past display. But, the collateral damage to unrelated content from prior domain-owners has grown as the web has aged, and more domains have changed hands. (The tradeoff that made sense in 2002 probably doesn't make sense in 2014.) Figuring out a system that can automate most legitimate exclusions, while rejecting or reversing those that lack a firm basis in content ownership or personal privacy, is a thorny task, but it would be worth pursuing if/when the Wayback Machine has the necessary staff resources. (My proposal since 2008 has been a DMCA-inspired 'put-back' procedure, where an original content owner can assert, formally, that they are the content owner and do not want the current-day robots.txt applied to captures before a certain date. Then, the current domain-owner would have to counter-notify that to maintain the block. This idea hasn't had legal review, but would reverse some current damage, and any bad-faith blockers would have to go on record with a false claim to maintain the block, potentially exposing them to a third-party legal challenge, with minimal risk to IA.)
- dredmorbius 12y ago
  Logging domain registration information (WHOIS records) and having an alternative method to applying opt-outs for sites whose domains have transferred might help address this. It's not clear-cut. WHOIS records are principally oriented at the domain itself. They may not clearly identify a registrant, registrants' identities may not be clearly evident across renewals, changes, or even registrar-related actions (M&A, failures, splits), the formats (and quality) vary wildly by registrar, some registrars (attempt to) restrict access and use of the records, and more. But it's a start. Hell, a TIA archive of WHOIS registrations might itself be useful....
- keenerd 12y ago
  Whoops, I probably should have checked the exclude.php page in the Wayback Machine before pinning a date on it. My bad. > An updated robots.txt does not irreversibly delete prior captures, so if the robots.txt changes again, access to previously-collected material can be re-enabled. Is there any official statement you can cite? Generally TIA does not mince words and is honest. If they say "removed" then I would assume they are not doing Facebook-style "deletion" shenanigans.
  - gojomo 12y ago
    I don't know of a linkable official statement to that effect. Unfortunately much of the information available (for example in the official onsite FAQ) is incomplete or outdated. The Wayback Machine gets only a tiny fraction of the product-management, public communication, documentation, end-user support, and fundraising that it should.
frik 12y ago
Archive.org honors the robots.txt, at during the indexing period - okay. But current domain owners should not be allowed to remove historic content of the domain at a later date by just modifying the robots.txt. A lot of information is lost as domain squatters take over domains set new robots.txt files. On Wikipedia for example you find a lot of reference-links that point to archived website URLs on archive.org. Every now and then the vital information source is lost - it's a bit surreal like book burning. I really like archive.org, this is the single feature I dislike a lot.

by keenerd 12y ago

gojomo 12y ago

A fair rant, but to correct some misperceptions: The retroactive application of robots.txt is not a new policy; it's been in place for at least 11 years, and I believe it arrived very soon after the Wayback Machine was first unveiled. An updated robots.txt does not irreversibly delete prior captures, so if the robots.txt changes again, access to previously-collected material can be re-enabled. This policy has served to minimize risk and automate the most common removal-scenario, when a webmaster wants a total opt-out of current crawling and past display. But, the collateral damage to unrelated content from prior domain-owners has grown as the web has aged, and more domains have changed hands. (The tradeoff that made sense in 2002 probably doesn't make sense in 2014.) Figuring out a system that can automate most legitimate exclusions, while rejecting or reversing those that lack a firm basis in content ownership or personal privacy, is a thorny task, but it would be worth pursuing if/when the Wayback Machine has the necessary staff resources. (My proposal since 2008 has been a DMCA-inspired 'put-back' procedure, where an original content owner can assert, formally, that they are the content owner and do not want the current-day robots.txt applied to captures before a certain date. Then, the current domain-owner would have to counter-notify that to maintain the block. This idea hasn't had legal review, but would reverse some current damage, and any bad-faith blockers would have to go on record with a false claim to maintain the block, potentially exposing them to a third-party legal challenge, with minimal risk to IA.)

dredmorbius 12y ago
Logging domain registration information (WHOIS records) and having an alternative method to applying opt-outs for sites whose domains have transferred might help address this. It's not clear-cut. WHOIS records are principally oriented at the domain itself. They may not clearly identify a registrant, registrants' identities may not be clearly evident across renewals, changes, or even registrar-related actions (M&A, failures, splits), the formats (and quality) vary wildly by registrar, some registrars (attempt to) restrict access and use of the records, and more. But it's a start. Hell, a TIA archive of WHOIS registrations might itself be useful....
keenerd 12y ago
Whoops, I probably should have checked the exclude.php page in the Wayback Machine before pinning a date on it. My bad. > An updated robots.txt does not irreversibly delete prior captures, so if the robots.txt changes again, access to previously-collected material can be re-enabled. Is there any official statement you can cite? Generally TIA does not mince words and is honest. If they say "removed" then I would assume they are not doing Facebook-style "deletion" shenanigans.
- gojomo 12y ago
  I don't know of a linkable official statement to that effect. Unfortunately much of the information available (for example in the official onsite FAQ) is incomplete or outdated. The Wayback Machine gets only a tiny fraction of the product-management, public communication, documentation, end-user support, and fundraising that it should.

frik 12y ago

Archive.org honors the robots.txt, at during the indexing period - okay. But current domain owners should not be allowed to remove historic content of the domain at a later date by just modifying the robots.txt. A lot of information is lost as domain squatters take over domains set new robots.txt files. On Wikipedia for example you find a lot of reference-links that point to archived website URLs on archive.org. Every now and then the vital information source is lost - it's a bit surreal like book burning. I really like archive.org, this is the single feature I dislike a lot.