12 ms·
Reflections as the Internet Archive turns 25
- endisneigh 5y agoI'm curious - how does Archive.org get around DMCA? I assume it's because Fair Use and the fact that they're a not-for-profit, but more details would be great. Other sites like outline.com (which I guess is a for-profit) entity don't really allow you to get around paywalls the way the Wayback Machine does. As someone interested in building a site that gets around paywalls for semi-educational purposes I'm curious if anyone has details!
- joe_the_user 5y agoMy impression archive.org follows whatever directions are in robots.txt. If those directions allow a site to be archived, they do it, otherwise they don't. That's from looking at not-archived sites, ezboard, which mention in another post.
- toomuchtodo 5y agoTakedowns cause the content to be “darked.” It’s no longer publicly available, but still archived on disk until a future date.
- pabs3 5y agoIA are a designated library, which confers a set of privileges under US copyright law.
- DoingIsLearning 5y agoYes as a non-US based user this is one of the most amazing things for me with IA. in my head IA was just the wayback machine with cached pages. But during the pandemic I realized that there is a plethora of actual books that one can checkout with far less friction than in a standard library. It is such a cool concept, also there are IA satellite projects (not sure if they are owned by or in partnership), for different non-english languages e.g. arquivo.pt so you can have the same plethora of content in other non-english languages as well.
- ghaff 5y agoA very limited set of privileges. And AFAIK it's only a "designated library" in California. I'm not sure there is such a thing at the federal level other than the Library of Congress. Basically, most people are fine with what the Wayback Machine does and they'll take down any mirror that the domain owner asks them to.
- joe_the_user 5y agoNice they're there. At the same time, it's amazingly easy for content to be removed from there - if someone objects or even if things are murky. For example, all content from the old ezboard site was been removed based on the configuration of the current URL owners' robots.txt, and current URL owner is just a domain parker. Ezboard hosted a lot of content back in the day. https://archive.org/post/560730/ezboard-is-there-any-hope https://archive.org/post/560730/ezboard-is-there-any-hope
- SilverRed 5y agoHopefully it is just hidden and not deleted. But this is the main reason why alternative archive sites exists which ignore the original posters requests. Frequently used to archive posts from public figures which are suspected to be attempted to be scrubbed later.
- joe_the_user 5y agoGot any examples of these alternative archives?
- SilverRed 5y agoThis is the main one https://archive.is/ https://archive.is/ From the FAQ, they do not respect robots.txt since they only archive on request by a user and they do not remove archives unless they contain illegal content.
- 1vuio0pswjnm7 5y agoHN commenters like to use archive.is but I always wonder if people are aware that archive.is is (a) blocked in some countries^1 and (b) may block access to itself in a country if it feels threatened.^2 There is also the issue of EDNS subnet.^3 archive.is tries to require it; it wants to know what location a request is coming from. In addition to EDNS, archive.is inserts the IP address and geolocation of the incoming request into the HTML of the returned page as a tracking pixel.^4 Thus archive.is does some things archive.org does not do besides just ignoring robots.txt One of the things archive.org does that archive.is does not do is that archive.org inserts an HTTP response header intended to disable Chrome FLoC.^5 I add this header for all sites in a local proxy; however I do not see many sites adding it as a courtesy. Thanks archive.org for doing that. 1. https://en.wikipedia.org/wiki/Archive.is https://en.wikipedia.org/wiki/Archive.is 2. https://www.reddit.com/r/KotakuInAction/comments/3e29vm/archiveis_possibly_banned_in_finland/ https://www.reddit.com/r/KotakuInAction/comments/3e29vm/arch... 3. https://webapps.stackexchange.com/questions/135222/why-does-1-1-1-1-not-resolve-archive-is https://webapps.stackexchange.com/questions/135222/why-does-... 4. https://news.ycombinator.com/item?id=27498902 https://news.ycombinator.com/item?id=27498902 5. permissions-policy: interest-cohort=()
- deleted 5y ago[deleted]
- TedDoesntTalk 5y agoHow many books have you read that were published in the 1800s? I’m betting close to zero. Unfortunately, most people 200 years from now won’t care about the 70 petabytes the Internet Archive has saved. Don’t misunderstand: I am glad they do this and love their work. I just think we overestimate the long-term value of this info beyond a very small set of future historians or social historians. Most people have their lives to live in this moment, and if they have a chance to look backwards before they were born, it’s not a big piece of their time.
- someguy101010 5y agoI was reading every newspaper that mentioned spanish flu from the 1910's when covid was starting. I never would have thought that I would have been wishing for easier access to full text search newspaper archives. Here we are :p
- jfoutz 5y agoEh, Shelly, Dickens, Twain and Wells are all I can think of. Not zero, but a vanishingly small percentage.
- TheCowboy 5y ago> How many books have you read that were published in the 1800s? I’m betting close to zero. Quite a few actually, and I'm not an outlier. Plus there are many adaptations and derivative works that exist.
- dleslie 5y ago19th century literature is full of treasures.
- SilverRed 5y agoHow many times have you clicked a link and found the page 404 or redirect you somewhere else? That is the real value of the web archive to me. Wikipedia uses it a lot as well, they automatically snapshot any link used as a reference and if the page goes away they automatically swap the link for a web archive version. Reading a reference 50 or even 200 years old is not absurd. A post detailing some research findings which is referenced in wikipedia is still greatly valuable. Youtube historians often reference ancient patents to uncover the history of old items.
- soheil 5y agoFor anyone interested I made a quick and dirty way to pull up the archive of any URL by prefixing it with arxiv.link e.g., http://arxiv.link/https://news.ycombinator.com/ http://arxiv.link/https://news.ycombinator.com/
- fouc 5y agoNice, thanks! Ever since I started using archive.is last year, I've always wanted something like this! Could be paired with a bookmarklet too. javascript:(function(){window.open('http://arxiv.link/'+location.href)})(); http://arxiv.link/'+location.href)})();
- alexislours 5y agoYou have always been able to do this with archive.is, I’ve been using this bookmarklet for a quite some time. javascript: (() => { window.open("https://archive.is/ https://archive.is/" + window.location.href, '_blank')})();
- fouc 5y agoI meant I wanted to go directly to the most recent result in the wayback engine for the given page.
- alexislours 5y agoIn this case you can just use the following and it will redirect on the latest archive without being dependant on a 3rd party website: javascript: (() => { window.open("https://web.archive.org/web/ https://web.archive.org/web/" + window.location.href, '_blank')})();
- codethief 5y agoarchive.is is not related to the Internet Archive's Wayback Machine, though, is it?
- 5y ago
- deleted 5y ago[deleted]
- dleslie 5y agoI love the Internet Archive; I worry that its utility will wane as content becomes more dynamic than static. What does it mean to archive the experience of scrolling through a social feed?
- petertodd 5y agoThe paid, legal-oriented, archiving service Perma.cc that Harvard Law runs actually lets you upload your own PDFs and screenshots in addition to allowing Perma.cc's bots capture webpages. Of course, since you could upload anything the difference is made clear in the UI. In a legal context, simply attesting to the validity of a screenshot is really common. So when that functionality is used Perma.cc is operating more as a permanent file storage service than a trusted archive. Regardless, this does go a long way to solving the problem of dynamic sites.
- cxr 5y ago> actually lets you upload your own PDFs and screenshots in addition to allowing Perma.cc's bots capture webpages FWIW: the Wayback Machine is just one part of the Internet Archive. The quoted bit accurately describes things you can do with an archive.org account, too. Readers here may be familiar with the archive.org-affiliated effort by a team specifically working to recreate the playability of old PC (and otherwise) video games with JSMESS. > this does go a long way to solving the problem of dynamic sites Maybe, but the "dynamic" aspect that I'm sure the other person had in mind doesn't have much to do with the D in DHTML so much as it has to do with the dynamism that arises when you have a smart server responding to requests from a fat, JS-powered frontend. It would be possible to accurately model this in and execute it from a series of static assets, in some cases, but it's rarely done. Even many sites built with static site generators today are not going to be usable in the future. There's too much tight coupling to the environment/deployment configuration and not enough semantic richness to properly hint to the crawler what resources are necessary to archive. In the heydey of XML, it used to be a big deal to strive for machine readable documents. Today's resume-driven development-obsessed webdevs effectively cast a vote of no confidence even in HTML, doing an end-run around it daily, and figuratively holding up a middle finger to the Principle of Least Power. https://www.w3.org/DesignIssues/Principles.html#PLP https://www.w3.org/DesignIssues/Principles.html#PLP To some extent, even a bunch of the projects associated with TBL's Solid initiative are guilty of doing the same.
- tkgally 5y agoIn his reflections, Brewster Kahle mentions his goal of creating “a library available to anybody, anywhere in the world.” He doesn’t mention, though, the costs of making that library available to the world for free or the fact that the Internet Archive accepts donations. So I will: https://archive.org/donate/ https://archive.org/donate/
- walterbell 5y agoAnyone know the relative budgets/donations/staff of Wikipedia vs. Archive.org?
- tkgally 5y agoProPublica has a database of tax filing information for nonprofits. Internet Archive: https://projects.propublica.org/nonprofits/organizations/943242767 https://projects.propublica.org/nonprofits/organizations/943... Wikimedia Foundation: https://projects.propublica.org/nonprofits/organizations/200049703 https://projects.propublica.org/nonprofits/organizations/200... Mozilla Foundation: https://projects.propublica.org/nonprofits/organizations/200097189 https://projects.propublica.org/nonprofits/organizations/200... Electronic Frontier Foundation: https://projects.propublica.org/nonprofits/organizations/43091431 https://projects.propublica.org/nonprofits/organizations/430...
- ezequiel-garzon 5y agoThis library would be available not only to those who could pay the $1 per minute that LexusNexus charged Any idea on what LexusNexus is, or was back then? Thanks!
- Teever 5y agohttps://en.wikipedia.org/wiki/LexisNexis https://en.wikipedia.org/wiki/LexisNexis
- lifekaizen 5y agoVery expensive research tool places like large law firms would use.
- fumeux_fume 5y agoI think it's a reference to LexisNexis which still exists today in a similar capacity. Back in the day (1990s), it was an online service to search for and read virtually any legal or news article among many other things. It was very expense, but was usually access thru a corporate or educational institution's license.
- Mountain_Skies 5y agoIf you live in the US, they have a file on you and it's probably very thick. A credit report will run a couple of pages long, the LexusNexus report on me (a complete nobody) cost them about $8 to mail. It contained many errors, mostly about property I don't own but also about an insurance claim I never made. Like your credit report, you can get a free copy by writing them and requesting a copy. IIRC when I did it a few years ago, I had to make the request in writing, I wasn't able to order it online, at least not for free.
- jacquesm 5y agoThey ought to be liable for those errors.
- IfOnlyYouKnew 5y agoBig fan of the archive, especially openlibrary. But the lofty talk of “democratizing” this-or-that is a bit overblown: in authoritarian countries, the archive does zero democratizing, because it’s either just not available, or all the relevant individual documents are blocked. For democratic countries that have democratically elected to restrict the distribution of some materials the society considers harmful, such as Germany with Nazi propaganda, the archive happily decides to undermine those clear, longstanding, and — disproving the single argument for free speech absolutism — not slippery-sloping anywhere over decades. Why? Because laws you disagree with are, apparently, illegitimate. It probably helps their bold defense of all that is holy to intimately know that these really are democratic countries, which aren’t going to just send a wet team to dismember them, Saudis-style, or to spend millions on an elaborate plan whose only purpose is to let you live for another month, with a clear mind that has complete certainty that you will die, and who did it. The anti-semitism and racism on archive.org, plus some copyright violations isn’t a byproduct of their “freedom”. It’s all of it. There are plenty of free hosts for video or documents, and an hour at minimum wage would pay for hosting quite a lot for quite a while for most of the archive’s audience. But the killer feature is immunity, through anonymity and DMCA’s Safe Harbour. Sure, everyone here is only defending “free speech” and would never agree with the swastika-fetishists. Only, somehow, they never complain about ISIS having a hard time on Twitter, or porn being censored on Facebook. It’s the scans of Der Stürmer, especially of the 38 to 44 vintage, are the chosen symbols of “democratization”. [0]: https://www.sfgate.com/california-politics/article/Internet-Archive-Neo-Nazis-white-supremacist-SF-16320465.php https://www.sfgate.com/california-politics/article/Internet-...
- textfiles 5y ago"Big Fan" is doing a lot of heavy lifting there.
- ZeroGravitas 5y agoI was intrigued by this comment, but couldn't really figure out what was being claimed so read the linked article. It's still not 100% clear but elements of it include: - Historical racist texts (Mein Kampf, 1930s newspapers) which I feel on balance is a good thing to preserve in a library. I would assume having those papers available does more to combat fascism than to encourage it (though who knows really) - archives of websites that are dodgy (ISIS and Neo-Nazi are mentioned but there must be all sorts of crazy and or bad stuff on the web that gets archived) - following US law rather than local law (a general conundrum for internet sites like this, especially if you do it for countries whose laws you like but not for other countries) - providing a place for people to upload and share files anonymously So yeah, I'm not really a Free Speech absolutist myself (doesn't seem like many that claim they are actually believe it when it comes to things they disagree with) but doesn't feel like they're in the same boat as social media platforms who actively spread bad things if it increases ad impressions. Some similar issues around policing large amounts of content and dealing with different legal, political and moral frameworks at scale though.
- shrubby 5y agoPictures and touchscreens have ruined the internet ;-)
- spiritplumber 5y agoThe Internet Archive location is beautiful. it's a church that has been partially turned into a server farm. Big Neuromancer energy when you go inside and look.
- DoingIsLearning 5y agoI was too curious not to look this up: https://www.atlasobscura.com/places/internet-archive-headquarters https://www.atlasobscura.com/places/internet-archive-headqua...
- N3cr0ph4g1st 5y agoDid a hackathon there in 2016, it is so cool!
- DonHopkins 5y agoA search of youtube for "wayback machine" produces pages of stuff about the Internet Archive, and only the 24th result has anything to do with the origin of the term. People who didn't spend their Saturday mornings glued in front of the TV screen as a child of the 1970's might not remember how American kids learned about history back then: Peabody's Improbable History - Surrender of Cornwallis Peabody and Sherman travel back to October 19, 1781 to witness when Cornwallis surrendered for Washington. However, when they got there, then he didn't show up. https://www.youtube.com/watch?v=3E8zmaOiCVw&ab_channel=bullwinklecanada https://www.youtube.com/watch?v=3E8zmaOiCVw&ab_channel=bullw...
- ConceptJunkie 5y agoActually, "The Bullwinkle Show" premiered in 1959, but I discovered it in the early 70s, as I suspect you did.
- pwdisswordfish0 5y agoFYI, it's not nowadays obscure. There's a current series The Mr. Peabody & Sherman Show on Netflix, and Hollywood made a Mr. Peabody and Sherman movie (by Dreamworks) in 2014.
- ZeroGravitas 5y agoThis is a bit of a tangent, but there was mention of non-advertising based funding models. Is anyone working on an advertising model that achieves the basic goal of advertising, but without the centralising aspect which seems to be the root cause of many of the issues? Giant monopolies are always going to subvert regulation but the same industry as disconnected units might be easier to police. Obviously you can just try to split them up or limit their size with regulation after the fact but a good technical basis might help out. Does web advertising just not make sense unless you can amass lots of private user data and track people across the web? If so can we subcontract that data to smaller companies we can trust with our data and effectively punish if they break the rules?
- coldpie 5y ago> Does web advertising just not make sense unless you can amass lots of private user data and track people across the web? It does make sense, but it has to compete with the invasive-style advertising, and it will always lose. If you want the "good advertising" you have to kill the "bad advertising".
- elric 5y agoWithout trying to be contrarian, I don't think that everything should be archived. Random tweets, random blog posts, random personal web sites. Let them wither and die and be forgotten. Notable content by notable people? Sure. Everyone else ought to have the right to be forgotten, including some drunk tweet they wrote 10 years ago and regret, or an old personal page which contained too much PII. Archive no longer has a way to opt-out, which is bad enough, but I still think they should be opt-in.
- jacquesm 5y agoYou never know in advance what will and what will not be worthwhile archiving, you only know that at some unspecified point in the future.
- elric 5y agoThat's lovely from the perspective of a historian 200 years in the future. But it does not nothing to alleviate the pain of people living in the present. People whose prospective employers comb over their embarrassing past. Or bullies. Or any other number of evildoers whose life is made easier by unfettered access to indelible information.
- pwdisswordfish8 5y agoI don't like your proposed solution that uses right to be forgotten as a blunt instrument to paper over serious problems, to used a mixed metaphor. When you appeal to the right to be forgotten to fix problems like people and "prospective employers comb over their embarrassing past", it's a way of throwing up your hands and neglecting deeper issues. In a right-to-be-forgotten world, the way it would end up going is: 1. problematic potentates punish pitiable proles 2. someone invokes right to be forgotten 3. this is considered "good enough" 4. the problem conditions that allowed #1 to fester remain uncorrected I feel this way about a lot of stuff these days (especially the where the erosion of the tenets of a liberal society is involved), where people argue vociferously for a "solution" that can at best be considered an indirect way of handling the problem. You see this with a lot of contemporary calls for the dismantling of tradition of free speech/free inquire/freedom of association, for example. People end up chafing in the direction of proposals that have dual-use effects in the first instance and perniciously "null" effects in the second instance.
- akkartik 5y agoI wish IA hosted Usenet archives. Even if they stopped at year 2000 and didn't update further.
- generationP 5y agoThey host some: https://archive.org/details/usenet https://archive.org/details/usenet As far as my own old posts are concerned, it looks complete :) But it isn't easily findable or searchable; the intended way of interaction is apparently to download an entire hierarchy and grep.
- akkartik 5y agoOh good to know! That's adequate.
- cxr 5y agoAnything particular about Usenet that makes it attractive? Modern mailing lists (and even ones that are now dead but accessible) are a treasure trove of information, and I've often privately mulled over my concern that there doesn't seem to be a concerted effort to save them. Having tried to track down old copies of the Linux mailing lists, and having run into indicators of others' quests (and dead ends), I know how fragile the situation is. As I've recently come to understand, the Internet Archive itself used to have its own mailing lists for handling discussion, which interestingly enough seem to no longer be accessible—perhaps even lost.
- TekMol 5y agoHow can IA just go out there and copy+republish the content of others and get away with it? Aren't they breaching copyright on a massive massive scale?
- ghaff 5y agoBecause the vast majority of people don't care if someone archives something they've made public. And the Internet Archive bends over (to some too far) backwards with robots.txt to exclude anything someone wants removed from public view.
- causi 5y agoAs a question related to archival, what's the best tool for local archiving? HTTRACK is getting long in the tooth and it just work for all the dynamic content on modern web pages.
- ArtDev 5y agoI think we need a new src attribute that can have a fallback. There are so many parts of the internet where most of the links are dead. It's pretty sad actually.
- ovebepari 5y agoBrewster Kahle mentions his goal of creating “a library available to anybody, anywhere in the world.” Fun fact: Archive.org is blocked in Bangladesh for god knows what reasons.
- ignoramous 5y agoA little known trivia: Apache Hadoop (and the multi-billion dollar open source big-data ecosystem it spawned) was worked upon at first at Internet Archive [0]. Speaking of billions: According to Kahle, Alexa Internet's compute infrastructure informed Amazon's take on IaaS (AWS) [1]. Another perhaps lost nugget is Amazon once funded (either in part or in full) the development of the Wayback Machine, Internet Archive's most impactful product. In addition, till date (if I'm not mistaken) Amazon continues to donate data it fetches from Alexa Toolbar installations to the Wayback Machine. [0] https://archive.is/Le3id https://archive.is/Le3id [1] https://archive.is/EnzHq https://archive.is/EnzHq
- Thoreandan 5y agoIt's a shame that the mirror in Alexandria appears to be long abandoned.
- nonbirithm 5y agoIsn't the publisher's lawsuit still going forward in November of this year? It would be terrible if the Archive had to shut down because they decided to do an unnecessary experiment with copyright while simultaneously being tied to an irreplaceable, yet centralized, historical resource. I donate to them with the hopes that they won't try to do anything that carries that kind of risk to their continued existence again.
- musicale 5y ago> This library would have all the published works of humankind. This library would be available not only to those who could pay the $1 per minute that LexusNexus charged, or only at the most elite universities. This would be a library available to anybody, anywhere in the world. Could we take the role of a library a step further, so that everyone’s writings could be included–not only those with a New York book contract? Could we build a multimedia archive that contains not only writings, but also songs, recipes, games, and videos? For every Sci-Hub trying to create the library of Alexandria, there's an Elsevier trying to burn it down. Current copyright law is largely on the side of the arsonists rather than the archivists. (note: recipes are not copyrighted, though cookbooks are)