5 ms·
A fair rant, but to correct some misperceptions: The retroactive application of robots.txt is not a new policy; it's been in place for at least 11 years, and I
by gojomo 13y ago
A fair rant, but to correct some misperceptions:
The retroactive application of robots.txt is not a new policy; it's been in place for at least 11 years, and I believe it arrived very soon after the Wayback Machine was first unveiled.
An updated robots.txt does not irreversibly delete prior captures, so if the robots.txt changes again, access to previously-collected material can be re-enabled.
This policy has served to minimize risk and automate the most common removal-scenario, when a webmaster wants a total opt-out of current crawling and past display. But, the collateral damage to unrelated content from prior domain-owners has grown as the web has aged, and more domains have changed hands. (The tradeoff that made sense in 2002 probably doesn't make sense in 2014.)
Figuring out a system that can automate most legitimate exclusions, while rejecting or reversing those that lack a firm basis in content ownership or personal privacy, is a thorny task, but it would be worth pursuing if/when the Wayback Machine has the necessary staff resources.
(My proposal since 2008 has been a DMCA-inspired 'put-back' procedure, where an original content owner can assert, formally, that they are the content owner and do not want the current-day robots.txt applied to captures before a certain date. Then, the current domain-owner would have to counter-notify that to maintain the block. This idea hasn't had legal review, but would reverse some current damage, and any bad-faith blockers would have to go on record with a false claim to maintain the block, potentially exposing them to a third-party legal challenge, with minimal risk to IA.)
- dredmorbius 13y agoLogging domain registration information (WHOIS records) and having an alternative method to applying opt-outs for sites whose domains have transferred might help address this. It's not clear-cut. WHOIS records are principally oriented at the domain itself. They may not clearly identify a registrant, registrants' identities may not be clearly evident across renewals, changes, or even registrar-related actions (M&A, failures, splits), the formats (and quality) vary wildly by registrar, some registrars (attempt to) restrict access and use of the records, and more. But it's a start. Hell, a TIA archive of WHOIS registrations might itself be useful....
- keenerd 13y agoWhoops, I probably should have checked the exclude.php page in the Wayback Machine before pinning a date on it. My bad. > An updated robots.txt does not irreversibly delete prior captures, so if the robots.txt changes again, access to previously-collected material can be re-enabled. Is there any official statement you can cite? Generally TIA does not mince words and is honest. If they say "removed" then I would assume they are not doing Facebook-style "deletion" shenanigans.
- gojomo 13y agoI don't know of a linkable official statement to that effect. Unfortunately much of the information available (for example in the official onsite FAQ) is incomplete or outdated. The Wayback Machine gets only a tiny fraction of the product-management, public communication, documentation, end-user support, and fundraising that it should.