26 ms·
Addressing Recent Claims of “Manipulated” Blog Posts in the Wayback Machine
- ry_ry 9y agoThat was... vague Why would the robots file on an active site be applied to the archived content?
- paulie_a 9y agoI agree. That sort of defeats the purpose of something that is literally meant to archive
- ry_ry 9y agoTeMPOraL had the most plausible explanation up thread a bit - apparently it's there as a copyright claim safety-net for Wayback Machine, but making a feature that runs counter to the whole point of the site publicly available makes my head hurt. Perhaps having a bot generate a synopsis of removed content, and showing that in its place would solve any copyright issue fairly elegantly?
- icebraining 9y agoGenerating a synopsis might have other implications (e.g. accusations of libel if the original author considers the synopsis attributes false claims to them).
- occamrazor 9y agoA synopsis is a derivative work. As such, it falls within copyright laws.
- ry_ry 9y agoHuh,I'd always thought they fell under fair use in the same way a review does. You learn something new every day... Multiple times a day in my case!
- pygy_ 9y agoAgreed, it doesn't make any sense to apply those rules retroactively.
- zxcmx 9y agoLegal issues. Who owns the content? There is no real legal basis for a right to mirror, despite how it feels from a techy point of view.
- amelius 9y agoRobots.txt now has legal status?
- SiempreViernes 9y agoIt undoubtedly communicates some intention by the site owner about how the work on the site can be used, so probably?
- amelius 9y agoIt only communicates the intention if the reader is actually obliged to read and interpret the file as a legal document. If I put a file named "legal.txt" in an online folder, is anyone required to read it and act upon it? It might as well be a file intended for some completely unrelated purpose; e.g. a lawyer that put some drafts online, or for all the reader knows, it might even be part of a movie script.
- pseudalopex 9y agorobots.txt has been a de facto standard for over 20 years. Someone might be able to claim ignorance, but the Internet Archive has shown that they know about it. It has a specific format; if it can be parsed, it's safe to assume that it isn't part of a movie script. In most cases, copyright law requires the reader of a document not to republish it, so the robots.txt standard is actually much more permissive.
- occamrazor 9y agoIt represents the expressed preferences of the website owner, so it is legally relevant.
- 9y ago
- skywhopper 9y agoPragmatic decision to avoid big stinks that might risk the extremely high ratio of sites that _don't_ currently block IA.
- myhrvold 9y agoThe way I read it... the Wayback Machine did not allow archives to be taken down arbitrarily -- but a subsequent targeted robots.txt exclusion of the Wayback Machine could render prior archives of that website moot? (Because the Wayback Machine starts from scratch each time?)
- ry_ry 9y agoIf that's the case you could buy up defunct domains, exclude everything via robots.txt and selectively purge sites from archive.org That seems like a pretty glaring flaw in something designed to create an enduring record.
- Piskvorrr 9y agoYup, that already happened in the past. For some reason, this is apparently a feature, not a bug.
- TeMPOraL 9y agoFrom what I understand, this is the deal they make to avoid getting sued by everyone for copyright violation.
- proaralyst 9y agoI don't think they purge the archives, I think they just don't serve them on the wayback machine.
- pronoiac 9y agoYes. Instead of deleting anything, I think the Archive tends to mark stuff as "do not show this for a few decades."
- jake-low 9y agoDo you have a source for this? I didn’t know that but it’s very interesting – a good compromise between the interests of current website owners[1] and future historians. [1]: Sure, some people just want to hide embarrassing or incriminating content, but there’s also cases where someone is being stalked or harassed based on things they shared online, and hiding those things from Archive users may mitigate that.
- rmason 9y agoSo if you find something you'd better make a copy of it yourself because it might be going dark. Doesn't that kind of defeat the whole purpose of the Wayback Machine?
- SllX 9y agoYes, but it's also always been true that if you want to keep something for reference, you make your own copy.
- Buge 9y agoThe problem with that though is that people will think you've manipulated your copy. I've had people accuse me of this when I save pages with screenshots. You need to have a trusted third party make and store the copies. Sometimes I use archive.is , they don't automatically delete because of robots.txt but it's not fully clear to me when they do delete things.
- foota 9y agoThis would be entirely overkill, but in theory if you're accessing the site via https couldn't you record the conversation from your end and later prove it?
- rocqua 9y agoYou'd need a proven time stamp on that conversation, or else the site could just switch certs and then leak their keys. Then, they can claim that you forged the traffic using the leaked keys.
- simias 9y agoI can't believe I'm the one to propose this but being able to unjustifiably timestamp some data is one of the only actual use cases of blockchains. Archive your data, compute a checksum and store that in a Bitcoin block and you can prove later on that you actually owned the data at this point. Of course there are other ways to achieve that such as publishing your checksum to a vast number of neutral third parties such as in a mailing list, bittorent or even a newspaper. You could also rely on a trusted 3rd party who has low incentives to manipulate the data (or would risk a lot if they were caught cheating) such as a bank, insurance or notary for instance. I think archive.org could potentially do something like that by using a merkle tree containing the checksums of the various pages they've archived during the day and publish the top hash every day for anybody to archive (or publish it on a blockchain or whatever, as said above). If later on somebody accuses the archive of manipulation they can publish the merkle tree of the day the site was archived which contains the checksum of the page, and anybody having the top hash of that day can vouch that it is indeed correct. It doesn't stop the archive from storing bogus data but it makes it impossible to "change the past" post-facto, so in this particular situation the journalist could only complain that the archive stored bogus data back in the day and not that it was recently tampered with.
- aviv 9y agoThe robots.txt exlusion loophole has been known for quite a long time.
- jaclaz 9y ago>The robots.txt exlusion loophole has been known for quite a long time. Yes, but it seemed like they had changed their mind, exactly because there is a huge issue with "expired" domains, see: https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websites/ https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websi... https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... They experimentally ignored robots.txt on .mil and .gov domains, and I thought they were going to extend this new policy for all archived sites. The situation/status is not clear, though the retroactive validity of robots.txt remains (at least to me) absurd. It is IMHO only fair to respect a robots.txt since the date it has been put online, it is the retroactivity that is perplexing, as a matter of fact I see it as violating the decisions of the Author, that - at the time some contents was made available - by not posting a robots.txt expressed the intention to have the contents archived and accessible, while there is no guarantee whatsoever that the robots.txt posted years later is still an expression of the same Author. Most probably a middle way would be - if possible technically - that the robots.txt is respected only for the period in which the site has the same owner/registrar, but for the large amount of sites with anonymous or "by proxy" ownership that could not possibly work.
- C4K3 9y agoA middle way could be to only observe robots.txt for crawling, and not for displaying pages. So once a page is grabbed, it's available forever. But if a page is covered by a robots.txt exclusion, it won't be crawled.
- gmueckl 9y agoIs it reasonably feasible to extend the syntax of robots.txt to include date ranges when the entries are specific to the IA bot? That way, specific content from a certain time span could be retroactively suppressed if desired. This would also solve situations where a new owner blocks robot access for a domain where the former owner is OK with the existence of the archived site.
- 21 9y agoWhat's the GDPR impact on this? If I have my own blog on my own domain, and Google and Wayback Machine archives it, can I request them to delete it one year later under GDPR?
- thegeomaster 9y agoYou can already do this, by including an entry in your `robots.txt`.
- chrisvalleybay 9y agoYes, unless it is in the interest of the public to keep it up [0]. [0] https://www.theguardian.com/technology/2018/apr/13/google-loses-right-to-be-forgotten-case https://www.theguardian.com/technology/2018/apr/13/google-lo...
- mercer 9y agoWayback Machine does respect robots.txt, I think. I'm curious what happens if you lose control of your domain, though.
- kevingadd 9y agoIf you lose control the whole history of the site gets nuked by a new robots.txt. This has happened in a few notable cases fairly recently.
- bryanrasmussen 9y agoI guess so if that blog identifies you in some way, which you can probably argue it does.
- pbhjpbhj 9y agoPII is identifiable with you and personal. Though practically I can't see Google checking if particular pages have PII rather than just regular information. Almost all my blogging is about tech, not PII at all.
- 9y ago
- ashelmire 9y agoSo who has the old version of the blog posts in question, so we can see what this journo has to hide?
- disk0 9y agoThis is a good blog post to link to those that actually believe Joy Reid in this case http://ws-dl.blogspot.co.uk/2018/04/2018-04-24-why-we-need-multiple-web.html http://ws-dl.blogspot.co.uk/2018/04/2018-04-24-why-we-need-m...
- exolymph 9y agoIPFS wayback machine when
- acdha 9y agoWhen it scales multiple orders of magnitude better? Web archives are massive and have non-trivial storage costs. What would be interesting would just be storing a Merkle tree for archive hashes so many parties could verify that a much smaller number of copies haven’t been modified.
- lifeisstillgood 9y agoThis highlights to me something about long-term management of my domains. "blog.reidreport.com" is now run by some domain squatter - and knowing nothing about Reid or reid report I took a vaguely generic website at face value - and clicked on the heavily disguised paid adverts. Clearly her domain is defunct - but I got suckered and actually came here to say things like "what terrible journalistic standards" before double checking. As my old domains fall into disrepair I guess I will need to archive them to S3 and keep up the payments just to stop this happening. An interesting problem - and possibly a revenue source for archive.org? EDIT: Hang on - the article on archive says (someone) added a robots.txt to block them. But the blog.reidreport.com is parked on some crappy redirect thing. Whois says that joyannreird@gmail.com still owns the domain - so I think she has got some very very bad advice from her hosting company. And my point still stands - a domain name is a reputation, and it is for life, not just for christmas.
- heartbreak 9y agoAre you sure you're spelling the URL correctly? `blog.reidreport.com` (as you spelled it in your post) redirects to a Blogger.com "Permission denied" page. Not a squatter. I think I'd be concerned about your client redirecting you to a squatter page.
- igorkraw 9y agoFor what it's worth, I have habitually been saving things with https://addons.mozilla.org/en-us/firefox/addon/save-page-we/reviews/847893/ https://addons.mozilla.org/en-us/firefox/addon/save-page-we/... since last year. Though I have some other motivations, having a git repo with auto snapshots every minute and continously snapshotting HN discussions and changing newssites has been giving me a "I'm gonna analyze the shit out of this" vorfreude since 2 months (don't know the english word, german for literally "pre-happiness")
- y4mi 9y agoas a fellow german: giddiness maps wondefully in that sentence. but to stay rational: 'looking forward to', 'excited for' are the more objective terms.
- igorkraw 9y agoDankesehr
- IncRnd 9y agohorripilation
- jdmichal 9y agoAnd "anticipating" could be a more general term that doesn't necessarily have the happiness connotation. "Eagerly anticipating" would give a hit to a certain desire for it to happen, but not necessarily happiness. Of course, anticipatory happiness can be read in via context if it's an obviously positive event.
- acqq 9y agovorfreude == “joyful anticipation“ says the wiktionary.
- celticninja 9y agoI think it would just be anticipation.
- onetimemanytime 9y ago>>...we declined to take down the archives. OK, the way I read, author--one way or another--asked for her blog not being hosted by Wayback machine and they declined. It's my work, as long as I can verify that I wrote, they should take it down or be sued for copyright infringement. I get the "we're archiving the internet," but if I want that post where I said Google is evil taken down because I have a G job interview a week from now, they should take it down. Another thing, just because I have a page online, doesn't mean that I gave them consent to archive it for eternity. I get the robots.txt, but if you're archiving you should ask for permission, they are a gazillion robots out there.
- skywhopper 9y agoThe Internet Archive would be completely unworkable if they had to ask for permission first. So would the Internet for that matter. Anyway, the whole point is to preserve things that wouldn't otherwise be preserved for the future. The Internet Archive folks are thinking in timespans of centuries, not months or years. The service they are providing to current and future researchers and historians is invaluable.
- IAmEveryone 9y agoSection 108 exception to copyright protection for public ally-accessible archives: https://www.law.cornell.edu/uscode/text/17/108 https://www.law.cornell.edu/uscode/text/17/108
- ghaff 9y agoThat exception is for physical works within a library. It's basically the only thing that makes libraries/archives special relative to you or I with respect to copyright.
- deleted 9y ago[deleted]
- prepend 9y agoThe author released content for public consumption. While author has copyright, it’s not legal to revoke usage rights years after the fact. Not a lawyer, but think it’s related to usage rights under a copyright. Imagine an author giving you a copy of the book and then 15 years later coming to your home library and asking for it back. It’s cool to not post publicly or to restrict access. But releasing and then yanking doesn’t make sense.
- nkurz 9y agoThey aren't very prominent, but the two links at the top of the blog post provide useful context: https://www.mediaite.com/online/exclusive-joy-reid-claims-newly-discovered-homophobic-posts-from-her-blog-were-fabricated/ https://www.mediaite.com/online/exclusive-joy-reid-claims-ne... https://theintercept.com/2018/04/24/msnbcs-joy-reid-claims-her-website-was-hacked-and-bigoted-anti-lgbt-content-added-a-bizarre-story-liberal-outlets-ignore/ https://theintercept.com/2018/04/24/msnbcs-joy-reid-claims-h... The summary is that last year Reid (a journalist for a major US television news network) publicly apologized for a series of blog posts that were characterized as 'homophobic'. More posts of a similar nature were recently discovered on archive.org, and instead of apologizing for these as well, Reid has disavowed them. She and her lawyers claim that unlike the previous occasion, these newly discovered posts were altered by 'hackers' either before or after being archived. The linked blog post is making the limited claim the posts on archive.org accurately represent the posts present on Reid's site at the time they were archived, and do not appear to have been altered post-archiving.
- makomk 9y agoApparently there are also archived copies of some of the posts on the Library of Congress site that were saved back in 2006 when they were originally posted: http://ws-dl.blogspot.co.uk/2018/04/2018-04-24-why-we-need-multiple-web.html http://ws-dl.blogspot.co.uk/2018/04/2018-04-24-why-we-need-m... I'm not sure how old the Internet Archive copies were since they're no longer available, but at least one of the ones saved elsewhere was originally archived in 2012.
- onion2k 9y agoThe linked blog post is making the limited claim the posts on archive.org accurately represent the posts present on Reid's site at the time they were archived, and do not appear to have been altered post-archiving. This might actually be a good use case for a blockchain. Hashing the data that's added to the archive and then putting the hash in the blockchain would reasonably prove the data in the archive hasn't been modified at a later date.
- etskinner 9y ago
- wpdev_63 9y agoI believe it.
- csomar 9y agoSigned hashes with blockchain time stamping could be a solution to that. A standard format for content and publishing content (whether a blogpost or a tweet), then based on that block of text you provide a digital signature linked to your identity. The signature is published on the blockchain, though, this is optional. The author will then be unable to refute (or able to) the authorship of the relevant text.
- deleted 9y ago[deleted]
- JustSomeNobody 9y agoI don't see this ending well for her. She should have just come clean and apologized ... again.
- logfromblammo 9y agoApplying the various razors, I find that the hypothesis that would need to be refuted first is that Reid wrote the posts, they were archived as written, unaltered since then, and she simply does not wish to take responsibility for their contents now. Who had motive to alter the posts in question? Who had the opportunity? When could it have happened? What method did they use to do so? If Reid's team cannot plausibly answer those questions, we are still examining the simplest hypothesis, and have seen no plausible evidence that it should be refuted. If we are to believe that those posts were written by someone else posing as Reid, would that suspicion not apply equally to everything appearing on her blog now? In which case, the solution has always been to sign the post using public-private asymmetric cryptography and to employ a public timestamp server to verify the time of publication.
- themihai 9y agoBottom line: You can remove the content you own using robots.txt
- zeveb 9y agoThis, right here, is why the GDPR's 'right to be forgotten' is so pernicious. Were Mrs. Reid an EU person, the Internet Archive could be forced to disappear her previous posts. Or given that she's also a public figure, would it be permitted to retain them? Only a court could decide.
- parliament32 9y agoWait so adding a robots.txt exclusion for the Wayback Machine makes all previous archives of the site inaccessible? That's very odd behaviour, and really not the point of a robots.txt file... I would expect a robots.txt to control a bot's visits / scraping behaviour, not a site's history.
- adventured 9y agoYes. They use the robots.txt file to essentially ascertain ownership or control of the domain. Archive.org doesn't want to delete the content they have (for whatever reason), so the compromise they came up with is to read the robots.txt and then hide the content they have archived if the present domain owner/controller wants it to be that way. If you remove the robots.txt setting, the archives become available again.
- RIMR 9y agoCome on, Joy. You blog posts weren't even that bad, they were just in poor taste. You didn't say anything particularly vitriolic or hateful. This is your opportunity to admit that these were once your views, and emphasize your personal growth since then. Instead, you are just going to pretend that your past self never existed... "I find gay sex to be gross" isn't that controversial of an opinion. Plenty of open-minded, accepting people agree with you. It just wasn't a worthwhile opinion to espouse... Own it, Joy. Don't just play dumb, because now you just look dumb.
- magic_beans 9y agoYou're not wrong. Her blog is tame compared to some of the comments I see DAILY in ANY subreddit on reddit. What Joy SHOULD have done was admitted that she "used" to be a homophone, and apologized.
- apetresc 9y agoShe's still a homophone; Reid sounds exactly like "read".
- accoil 9y agoIsn't that the point though? She wasn't a homophone in the past.
- kleiba 9y agoIs the use of the word "homophone" some in-joke that I'm not aware of?
- pchristensen 9y ago"homophone" - each of two or more words having the same pronunciation but different meanings, origins, or spelling, e.g., new and knew. "homophobe" - a person with an extreme and irrational aversion to homosexuality and homosexual people. Just a little joke about a linguistic mixup.
- BeetleB 9y agoA bit surprised at the comments here. As far as I can remember, robots.txt has always been used by the Wayback Machine this way. Often a bummer when the original domain expires and a domain squatter takes over - many have robots.txt
- dredmorbius 9y agoThat's changed. A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly. We see the future of web archiving relying less on robots.txt file declarations geared toward search engines, and more on representing the web as it really was, and is, from a user’s perspective. https://teleread.org/2017/04/24/the-internet-archive-will-soon-stop-honoring-robots-txt-files/ https://teleread.org/2017/04/24/the-internet-archive-will-so... https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...