6 ms·
It appears that IA applies (or did apply) a new version of robots.txt to pages already in their index, even if they were archived years ago. That's silly, and
by metafunctor 10y ago
It appears that IA applies (or did apply) a new version of robots.txt to pages already in their index, even if they were archived years ago. That's silly, and stopping doing that would probably solve much of this problem.
- LoSboccacc 10y agoyeah it's not even hard to solve of a problem, just use the archived version of robots.txt that matches the crawl date too bad they already lost loads of internet content that way
- tempay 10y agoWhat should happen in the case that a website misconfigures robots.txt and ends up wanting to remove private data? I think I would be tempted to say that the data can't be removed to avoid abuse from future domain owners (or current ones) but I'm not sure if there would be any legal consequences of this attitude.
- LoSboccacc 10y agoSame when an archive crawl illegal/copyrigthed data, a pathway for that needs to exists anyway
- CM30 10y agoProvide a content removal form? It works for DMCA notices, it can work here. Maybe even have a 'reason' textbox to see why someone may want content removed...
- makomk 10y agoOf course, there's the inevitable risk that the Internet Archive's newfound control over who is allowed to make their past disappear into the memory hole and who has it archived forever will be used for political ends, especially since the ability to manually archive pages is already used this way by staff. (Take a look at Jason Scott's Twitter or that of the Archive Team sometime - lots of conspicuous manual archiving of stuff that's embarrassing to a certain US political party.) The issue of curators' views biasing the contents of collections seems to be underappreciated in general in the digital age, for some reason.
- TeMPOraL 10y agoHere I'd lean towards archiving everything indiscriminately. Politicians especially should not have the "right to be forgotten", because what they do is of historical interest.
- escape_goat 10y agoTo clarify the nature of the distortion you are referring to, it would be a sampling bias. In general, the archive spiders the web and ingests information so that there is a certain mean frequency of visits and a certain likelihood of any particular revision of a web page being captured. There would be instances in which data was entered into the archive more certainly and more frequently, on the basis of the nature of that data, than otherwise would have occurred. What one means by bias when one says that this biases the contents of the collection needs to be understood with some care. It would be interesting to hear some historians discuss the matter. I do not think that it is a type of bias that is likely to lead them very far astray. If it mollifies your concerns any, the last time I checked, anyone could manually archive any web page they liked. However, I would recommend writing to The Archive to express your concern. I have an entirely partisan appreciation of the ability of The Archive to prevent redactions from the historical record of material that might later be disavowed. However, I share your more general view that there is no reason that the online history of any single major U.S. political party should be documented any less carefully than any other other.
- textfiles 10y agoJust to idly correct you. Archive Team (not a part of Internet Archive) actually archives piles and piles of web-based material, sometimes in response to current events, sometimes because of known shutting down of services, and sometimes because of speculative worry about longevity. (For an example of the last one, we've been archiving all current FTP sites left.) Meanwhile, Internet Archive's crawlers are bringing in millions (really millions) of URLs every day, just constantly grabbing websites, files, video, you name it. There's certainly a "bias" to the current administration in terms of 1. They're in power 2. They keep removing things new and old. But think of it as us having a few lights shined in specific directions while thousands of other floodlights go literally everywhere.
- paol 10y agoThey didn't lose anything. Content excluded in this manner is only made inaccessible to the public, not deleted from the archive. They can change their policy retroactively.
- patmcguire 10y agoIt may be possible technically. I doubt it's possible in reality: they've probably made promises to a bunch of people over the years that this is how it works. People furious that Their Content is appearing somewhere else and go straight into lawyers on the first email.
- Chaebixi 10y ago> It may be possible technically. I doubt it's possible in reality: they've probably made promises to a bunch of people over the years that this is how it works. People furious that Their Content is appearing somewhere else and go straight into lawyers on the first email. They won't be furious when they're dead. I think the main value of the Internet Archive is not so much in the near term, but in the long term. I hope in the future they enact some policy that ignores any robots.txt for scrapes older than, say, 50 years.
- frik 10y agoUpvoted. That's exactly the problem, and the way to solve it. Example: two months before the movie "The Social Network" got released to theaters in 2010 Facebook decided to add a robots.txt to Facebook.com. Immediately Archive.org deleted/disabled access to the archive how Facebook startpage looked in 2004-2010. BTW. the correct way would be to activate archive access to Facebook.com for the 2004-2010 time-frame again. The "The Accidental Billionaires: The Founding of Facebook" book and the "The Social Network" film based on that book used of course partly Archive.org and various other research methods to get the facts.
- thisacctforreal 10y agoWhat about extending robots.txt to include date ranges? For future domain-owners this is likely far too much control, but maybe that could be mitigated if IA tracks DNS/whois/registration info too
- pimlottc 10y agoIf we did this, the date range would be set to "Forever" 99.99% of the time.
- icebraining 10y agoI don't think it's "silly". IA operates in a sketchy legal environment. There's no fair use exclusion for what they're doing, and it made sense to be extra careful and deferential towards website operators, lest they get hit by a lawsuit.
- deleted 10y ago[deleted]
- the8472 10y ago> There's no fair use exclusion for what they're doing Fair use is not the only exception to copyright. US copyright law has a separate section on exceptions for libraries and archives.
- icebraining 10y agoNo help there, the exception is very limited in the number of copies it may produce, among other factors: https://www.law.cornell.edu/uscode/text/17/108 https://www.law.cornell.edu/uscode/text/17/108
- matt4077 10y agoA "copy", in this context, is a file. They can have three of those, which aligns perfectly with standard backup practices. Serving them is distribution, but not copying.
- icebraining 10y agoYes, it is. Even loading a file to RAM was considered a copy (see MAI Systems Corp. v. Peak Computer, Inc) until Congress made an explicit exception.
- ghaff 10y agoThere are [1] but they seem to be pretty rooted in making a copy of original works in a physical library. The parent's point that the IA operates in a very grey area of law and therefore needs to bend over backwards to comply with requests to remove material still applies. [1] https://www.law.cornell.edu/uscode/text/17/108 https://www.law.cornell.edu/uscode/text/17/108
- mbrookes 10y agoI first came across that issue during one of the many Facebook privacy scandals. I'd found some juicy bits in (IIRC) a much earlier version of their privacy policy. But when I went back to it later, the robots.txt had been updated, and the earlier archives had been obliterated. That just seems wrong.
- turdnagel 10y agoAnother solution might be including http://archive.org http://archive.org in their archives.
- Asparagirl 10y agoIt's turtles all the way down!
- deleted 10y ago[deleted]