20 ms·

A Thread about Internet Archive's “Silent Killer”

589 points by danso 7y ago

toomuchtodo 7y ago
Internet Archive donations are matched 2-1 currently, for those interested. https://archive.org/donate/ https://archive.org/donate/
- deleted 7y ago
  [deleted]
- xur17 7y ago
  Are bitcoin donations included in that?
- Akababa 7y ago
  Who's matching the donations?
  - aeyes 7y ago
    I guess the Pineapple Fund https://pineapplefund.org/ https://pineapplefund.org/
    - textfiles 7y ago
      It is not the pineapple fund, although it has been a lovely time working with them. Regardless of the cryptocurrency discussion, the fact that someone who feels some sort of windfall would immediately have an urge to share it with organizations that need it is laudable.
maxton 7y ago
I donated a week or so ago. The Internet Archive has come in handy many times for me, not just for the Wayback Machine but also things like their live music archives. They're an indispensable resource.
- giancarlostoro 7y ago
  They also archive ROMs for different consoles.
  - _sbrk 7y ago
    Which is a copyright violation, right?
    - giancarlostoro 7y ago
      ROMs are legal at least here in the USA. Doesn't stop big companies from bullying little sites and becoming copyright trolls. The Internet archive will not be easily bullied to take down legally archived items thankfully. The trick is you have to create the ROM yourself apparently. Sharing it is bad. I would assume copies made by anybody would be identical, so it would be interesting to see such a case in court against The Internet Archive, not that I want them to get sued, but would love for them to win if they did. Nintendo is the most aggressive about ROMs than anybody.
      - chongli 7y ago
        Nintendo is the most aggressive about ROMs than anybody They are, but they’re also the most proactive in making their old games available on new hardware. It wouldn’t be fair to call Super Mario Bros “abandonware” because Nintendo has done a lot to keep the game alive and accessible to new generations.
        vadansky 7y ago
        They have been caught putting other people's "illegal" roms on VC
        giancarlostoro 7y ago
        https://www.eurogamer.net/articles/2017-01-18-did-nintendo-download-a-mario-rom-and-sell-it-back-to-us https://www.eurogamer.net/articles/2017-01-18-did-nintendo-d... Here's an article that asks this very question, not sure if it was posted on HN prior or not.
gatherhunterer 7y ago
Twitter is the worst choice for publishing. This involved so much more text-scrolling and photo-expanding than is necessary. Edit: If anyone would care to explain why they feel that chopping an article up into pieces of arbitrary size makes for a good user experience, please do so. I know Twitter gets a pass from HN in general but it is simply not suited to this type of content by design. This was a conscious choice by its creators.
- toomuchtodo 7y ago
  https://threadreaderapp.com/thread/1204428311553642496.html https://threadreaderapp.com/thread/1204428311553642496.html
  - sprite-1 7y ago
    Thanks! This is great, and even better coupled with Firefox's Reader View
- decebalus1 7y ago
  No, I learned it's the best choice for publishing [0] [0] https://news.ycombinator.com/item?id=21684008 https://news.ycombinator.com/item?id=21684008
- kick 7y ago
  I'm sure if you offer to donate someone will put it in a blog format for you. Hint hint. Edit: Twitter is obviously the correct platform for this because of reach. Hacker News is reading it, his 33.3k twitter followers are reading it, reddit is probably reading it, everyone's reading it! So if the goal was to draw more donations, he's winning.
- drdrey 7y ago
  On Twitter you get built-in liking, sharing and commenting, people will also DM you with questions, it's all one tap away. If you just post a link to a blog post, almost nobody will click it and read it there.
  - 7y ago
toyg 7y ago
TIA should get money from the UN. Or be the only beneficiary of a flat tax on network ports - I bet even just $5 on every small router sold in the US (which is basically nothing) would generate a ton of money for them.
starsinspace 7y ago
Although efforts like internet archive are noble (and I find it occasionally useful), I'm not sure it's always so great that everything anyone does online will be permanently archived. I know many people feel that everything should be available forever. But for me... it's pushing me away from doing much on the web. I liked it in the 90s when things were more ephemeral. When you could make mistakes and not have them easily found by anyone with a few clicks, forever.
- tcd 7y ago
  I'm inclined to agree with this position. Does every Youtube, Reddit, Twitter, Hackernews, Facebook comment need to be archived and stored for the next thousand years? I'd argue no, and there's a huge amount of waste in there - so many bot posts, or just spam. But here we are, people want to archive every byte of information that traverses through the internet. These days you have to approach cautiously, as every thing you do or post may be archived.
  - mirimir 7y ago
    > These days you have to approach cautiously, as every thing you do or post may be archived. That was obvious in the 90s. But sadly enough, it hasn't worked out that way. Before the Internet Archive anyway.
    - ghaff 7y ago
      I don't really worry too much. Between all the stuff that doesn't get archived, the fact that the sheer volume tends to effectively hide any single piece of information, and the frequent difficulty of connecting online identities to specific real people, it's not like most people have a perfectly discoverable complete digital record online.
      - mikepurvis 7y ago
        Which is fine as long as you remain anonymous and unimportant in the present day. It becomes much more of an issue if you abruptly become a person of interest, and all of a sudden there's a team of people motivated to go through that archive with a comb looking for the few juicy tidbits needed to publicly humiliate you. And obviously this is already happening— in the Canadian election a few months ago, there was a big scandal where some pictures turned up of the prime minister wearing blackface at a holiday party he attended in 2001. And on top of that there was a pretty steady stream of candidates (some of whom were indeed booted from their parties) who were challenged over social media comments/posts made 5+ years earlier, especially on hot social topics where national sentiment has rapidly evolved in recent times. Perhaps we will all just become inured to this, and rightfully be able to accept our leaders as human, judging them for who they are today and not past words and deeds. But is it possible to still retain the ability to fairly judge someone in the present while forgiving the past? Or will we lose the ability to discern the difference? The GOP's attitude toward president Trump does not give me hope on this.
tyingq 7y ago
Pretty amazing what they do with 1/10th the revenue of Wikimedia and quite a lot more data to manage.
- dredmorbius 7y ago
  IA are a capture process. Wikipedia supports active updates to content, under contentious circumstances, as well as structuring content for useful access (something IA also does, but under looser conditions). Both are highly worthwhile projects -- some of the best of the Web. But a straight apples-pomellos comparision is difficult.
  - tyingq 7y ago
    Wikipedia also gets a lot of that done with free labor. And, over the years, their spend has increased quite a lot more than their content, update frequency, and so on. I think it's a fair enough comparison.
tetha 7y ago
I've discovered their efforts to archive old dos games... fully playable in the browser through a dosbox build in WASM as far as I know. That's a really impressive cooperation of very old and very new technology - 16 bit up to the edge of JS (- edit: or rather, the edge of browser based computing). And on that train of thought, I just had a little flashback about storage sizes. There was some time when 3.14 MB was a big unit of measurement, and some 50MB drive was huge. The classical example: Monkey Island, Indiana Jones or Day of the Tentacle on ~12 floppy disks. But you had to choose which 1 or 2 to install because you didn't have enough space on your hard drive, or you had to swap disks every few screens :) Or, they are archiving a lot of video playthroughs in the lets-play style from video sites, in case those sites go through a meltdown like viddler does. I guess I'm rambling. Point is: This isn't just a storage dump. There are also interesting projects around the internet archive to make the old things accessible on new systems. Very worthwhile donating to.
- fyp 7y ago
  I relived a part of my childhood by playing Chip's Challenge again the other day: https://archive.org/details/chips_challenge_windows_3.x https://archive.org/details/chips_challenge_windows_3.x
  - ocdtrekkie 7y ago
    Note that you can buy Chip's Challenge 1, 2, and Chuck's Challenge 3D from the guy who originally created it. https://store.steampowered.com/developer/niffler https://store.steampowered.com/developer/niffler
    - acomjean 7y ago
      There is an interesting podcast with a "Chuck" Somerville interview. He started on the Apple ][. Its pretty fun walk through the industry in the 80s, and touches on getting the "Chips Challenge" name back from the group that bought it. http://appletimewarp.libsyn.com/apple-time-warp-podcast-episode-2-editedm4a http://appletimewarp.libsyn.com/apple-time-warp-podcast-epis...
- 7y ago
mirimir 7y ago
It's too bad that there's not a vaguely-somehow-related-but-not-really and impossible-to-censor service that retains stuff that sites have excluded using robots.txt or whatever.
- Rebelgecko 7y ago
  The good news is that the Wayback Machine stopped making robots.txt retroactive. That handles most of the cases I was running into where content I wanted was removed from the archive.
- 333c 7y ago
  Is this supposed to be a veiled reference to archive.is? I'm not sure what point you're making.
  - mirimir 7y ago
    No, I wasn't thinking of that site per se. I was thinking more about the 90s era dream of uncensurable "data havens". That led to Freenet, for example. Which is slow, and forgets stuff that doesn't get accessed. And Tor onion services, which are more readily taken down. But the problem is that there's no way to know in advance whether something is about to disappear from the Internet Archive. So you'd need someone inside who'd discreetly alert the backup service.
nym 7y ago
Setup a monthly $5 donation. It's not much, but I know how valuable the archive is... they're doing the work the library of congress SHOULD be doing.
- criddell 7y ago
  > I know how valuable the archive is How valuable is it?
  - bigwavedave 7y ago
    $5/mo ;)
  - Avamander 7y ago
    Data is generally hard to put a value on.
    - criddell 7y ago
      The implication through a lot of this discussion is that it's greater than zero and even that I question.
    - tingletech 7y ago
      When I took an animal communication class in college, there was a formula for the value of information (not data). I don't remember the exact formula, but it was similar Shannon's equation. Basically, the more valuable information is, the larger the change it affects in the probably of what an organisms next behavioral state is. So, if information signaling didn't change the behavior of another organism, it wasn't considered communication.
msla 7y ago
Here's what I thought this would be about: There's a problem with the Wayback Machine in specific which can kill your ability to access it quite silently, unless you know how to use the browser's development tools and interpret headers. It has to do with cookies: Somehow, the Wayback Machine sets cookies... and sets cookies... and keeps setting cookies, until it overflows its own ability to accept cookies. At that point, your browser tries to access a Wayback Machine page, handing the server all of the cookies it currently has, and the server refuses to deal. It absolutely denies everything, sending an error header and a blank page. You have to clear all web.archive.org cookies to get anything at all, at which point it works perfectly. I've completely solved this problem by blacklisting web.archive.org in browser cookie blacklists. I haven't had it happen since then. As far as I'm concerned, the problem is diagnosed and just needs to be solved. At their end.
- topherPedersen 7y ago
  I've run into this problem as well. I used to use a Chrome extension which I authored that saved all of the pages I browsed to the wayback machine (in another tab), and I would frequently need to clear my cookies to keep the thing working.
peterwwillis 7y ago
It seems like all those things they backed up didn't seem to impact anyone's actual life. People lost files, websites, communications, whatever, and yet they kept right on living life. So I can't help but think that this is all a waste, like people who take pictures everywhere they go, and never look at them later. Is there any useful purpose to all this other than nostalgia? And at what point does the nostalgia overwhelm the useful purpose?
- jborichevskiy 7y ago
  Seems to me like it could come in very useful in a legal case, with very significant and real impacts on someone's life.
  - Zitrax 7y ago
    Also useful for various research, in the same way as going to the archives in a library can be useful. Even if most data is uninteresting doesn't mean that there are data there that is very useful to someone.
redisman 7y ago
I encourage everyone to also do their own thinking of what did you enjoy in the internet or computer software in the 80's, 90's and 00's and see if it is still available somewhere. I found out that the local games scene I used to love as a kid had been almost wiped off the face off the earth. These were small non-commercial and shareware games localized in just one language so it was already a niche. The free hosting services of the 90s are gone so those sites are down and no one wanted to keep paying for hosting for 20 years+ on sites that get very few visitors nowadays. The only way to get these games again was to find a Discord group and a friendly stranger who agreed to seed a torrent (which had 0 seeds when I found it). I'm looking to upload them to a couple of different places and compile a basic website catalogue (static site on CDN) one of these days. For the layperson, these games are already gone from the internet. The internet archive does a great service but it is breadth-first and quite surface level. The depth has to come from people who were familiar with the sites at their peak. And there's a big change that no one is doing that for your specific interest.
- oneepic 7y ago
  YouTube has many channels with playthroughs of old games as well. (old for me is the 90's and early 00's since i'm in my twenties) You can find things like the old Lego PC games, old educational games like Physicus/Bioscopia, kids' games like Putt-Putt (not in the least because they have a speedrunning community), and much more. It's been a great resource to go back to the stuff I played growing up.
- foxthatruns 7y ago
  BlueMaxima's Flashpoint is a fabulous archival project that is saving as many Adobe Flash games/animations as they can before browsers pull support at the end of 2020. Really cool, since Flash games are a similarly concrete slice of culture/history that will just be gone if they're not archived. https://bluemaxima.org/flashpoint/ https://bluemaxima.org/flashpoint/
  - baroffoos 7y ago
    Looks like its a huge amount of work to strip the DRM out of the games which stop functioning when the original website stops working.
jedberg 7y ago
It seems they could save some money by moving a bunch of infrequently accessed data to warm storage. The entire archive does not need to be accessible 24/7. I would be perfectly ok if I was trying to see a copy of a web page from five years ago, and it said that I had to make a request and it would be available in five or ten minutes. I think I could wait five or ten minutes for a web page to get pulled from the archives.
- archi42 7y ago
  Oh heck, I would be okay waiting for a few hours, or even a day for huge files. Though I suppose electricity isn't a huge driver for costs, afaict from that thread, the disks are.
- anamexis 7y ago
  Practically speaking, how would warm storage like that be implemented?
  - dredmorbius 7y ago
    Usually tape libraries. A problem with most forms of warm storage is that they involve moving media around, which itself can lead to degradation. I would like to see some write-one and really last forever form of storage to come into widespread use. That seems ... like it's still a few years off. And you still have the issue of how such storage is read, and how that affects total lifetime.
    - ccostes 7y ago
      Sounds like what they're doing at Microsoft with Project Silica https://news.microsoft.com/innovation-stories/ignite-project-silica-superman/ https://news.microsoft.com/innovation-stories/ignite-project...
      - dredmorbius 7y ago
        That definitely is in the direction I'm considering.
notacoward 7y ago
The real "silent killer" I see here is the reliance on mirroring. One-failure protection, with a 2x expansion factor. As it happens, I work on large storage systems, where 2x is our maximum expansion factor and for that we get resistance to as many as nine simultaneous failures. Across power and network failure domains, with multiple kinds of background scrubbing to detect loss of that redundancy. Oh, and 60PB is something we might add to an existing cluster for a day to absorb transient I/O load. There's also a bunch of monitoring and automation stuff that should be considered "table stakes" for storage at these scales. Seems like an opportunity to use what I and others have learned for a good cause, to make this valuable resource more efficient and more durable all at once.
- scarejunba 7y ago
  Is that different techniques or you just have vastly more resources? I'm sure they'd accept your help in either case if you offered directly.
  - jeremysalwen 7y ago
    I think he's referring to error-correcting codes.
  - notacoward 7y ago
    It's different techniques (erasure coding). As it turns out, applying those techniques would reduce need for additional physical resources as well, since it allows data to be stored more efficiently with the same physical resources. It does require more CPU and memory relative to bytes stored or spindles to store them on, but it's easy to make that tradeoff and still come out ahead.
- diego 7y ago
  Did you get to the part where they explain that they are strapped for resources and in need of donations? They have plenty of expertise, money is what they need.
  - notacoward 7y ago
joe_the_user 7y ago
So, copyright conditions are apparently another silent killer [1]. A website can be archived, vanish from the web, and then vanish from the archive for technical copyright reasons (new owner's robot.txt file on the root). So "archiving the archive" might be useful. Or something. Ezboard was an old discussion site that contained much of interest - archived and now the archive is not accessible. https://archive.org/post/389127/ezboard-content-suddenly-not-available-in-the-new-system-why https://archive.org/post/389127/ezboard-content-suddenly-not...
- btrettel 7y ago
  A while back I tried looking up a particular ezboard forum I used to participate in, only to find it was blocked. This is a real shame. I have some of the posts saved but it's only a fraction of the full forum.
- deleted 7y ago
  [deleted]
- tgsovlerkhgsel 7y ago
  TIL they still follow robots.txt -- https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... mentioned that they were planning to stop doing that (and I remember reading a news article based on it claiming that they already stopped following robots.txt, hence my confusion). Truly a shame. I get following robots.txt at collection time, I don't get following a robots.txt that was added later.
  - joe_the_user 7y ago
    Well that was 2 and 3/4 years ago and ezboard is still gone. I hope they didn't erase but rather excluded but legality and all that.
- tingletech 7y ago
  I think they still keep the data. I've always wondered why it was called the "The Oakland Archive Policy". http://groups.ischool.berkeley.edu/archive/aps/removal-policy http://groups.ischool.berkeley.edu/archive/aps/removal-polic...
somesortofsystm 7y ago
I hope the first human settlement on another planet takes a backup of the entire Internet Archive and has it safe and sound on another planet. Well, I can imagine that happening in the next 5 years, anyway.
mrmonkeyman 7y ago
But who will archive the archive?
commoner 7y ago
The Internet Archive is absolutely essential to Wikipedia, whose articles are required to be verifiable to reliable sources. When pages go offline, the link rot makes it harder for articles to be verified. However, if the Wayback Machine has an archived copy, that copy can then be cited as the source and made available to readers and editors. The Wayback Machine now automatically archives every new external link added to the English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Link_rot https://en.wikipedia.org/wiki/Wikipedia:Link_rot The Wikimedia Foundation's budget is about 10 times that of the Internet Archive. If you see the fundraising banner on Wikipedia and want to help out the site, but don't think the Wikimedia Foundation needs the donation, consider donating to the Internet Archive instead. https://archive.org/donate https://archive.org/donate
- Bartweiss 7y ago
  This is a great point, thank you. It's been awesome to see archiving and Archive links become standard practice on Wikipedia. For a long time, links just rotted and were replaced. Then later, dead links would be switched to Archive content if it was available. But it's so much easier and more reliable to archive a page when it's referenced, and provide both links even while the page is still up. It's one thing to automate dead link detection, but archiving defends against a lot more. Universities seem to redo their sitemaps and redirect all their old links about once a year. And reference pages get edited, which goes unnoticed for a long time and then leads to frustrating [Not in citation given] tags. It's incredible what a boon the Internet Archive has been to wiki sourcing.
- ignoramous 7y ago
  > The Wayback Machine now automatically archives every new external link added to the English Wikipedia. Links submitted to Hacker News should be auto-archived, too: I often stumbled upon dead-links [0] which otherwise had generated insightful discussion on news.yc. Adding archive next to web would work nicely. [0] Launch HNs, in particular.
  - scrollaway 7y ago
    It is possible to do that nowadays and that is one of the things I suggested implementing in tildes. https://gitlab.com/tildes/tildes/issues/586 https://gitlab.com/tildes/tildes/issues/586 I believe all link aggregators should do this, they're a great way of getting defacto quality/interesting content discovered by the archive.
corporate_shi11 7y ago
I found out about the Wayback Machine back in 2007 or so. I was young and just starting to explore the internet and found it incredibly fascinating to explore news sites as they were on the day of major historical events (9/11, etc). The amount of times I've relied on the Internet Archive by way of the Wayback Machine is numerous. I'll be adding them to my modest donation list, along with Wikipedia and similar public service sites.
lazylizard 7y ago
What does marie kondo think?
- catalogia 7y ago
  What does lazylizard think?
HenryKissinger 7y ago
Why not just let old data die? What's the point in devoting 10TB to saving data which maybe 300 people will check in the next 25 years? Information destruction is not necessarily a bad thing.
- tingletech 7y ago
  Selection is destruction, and destruction is selection. (but not even a blackhole can destroy information?)
bpaddock 7y ago
The Food and Drug Administration (FDA) has been archiving the FDA.gov site at archive-it.org . Leaving a lot of dead links on the FDA site. Sometimes they tell you to look in the archives for the old information, without giving you a link to it, and sometimes they don't, they just expect you to know. Now why can't the FDA afford the space to keep their pages forever on their own site? Fill in your favorite conspiracy theory... Some of the information that has been removed, such as the 2015 hearings on Fluoroquinolone antibiotics, are important health research as just one example. https://archive-it.org/organizations/1137 https://archive-it.org/organizations/1137
- Analemma_ 7y ago
  > Now why can't the FDA afford the space to keep their pages forever on their own site? Fill in your favorite conspiracy theory... This seems like a prime example of Hanlon's razor. Tight government budgets and lowest-bidder contractors not bothering with page permanence strike me as the most likely explanation.
  - ghaff 7y ago
    It may also be worth observing that it's how pretty much any private company website operates. Arguably the FDA should be different and archive old, even outdated, content. But, while private companies may explicitly archive some materials like press releases and earnings reports, 99.9% of their focus is on the current content and they'll mostly just delete anything that's not in service of today.
- kevingadd 7y ago
  It's not limited to the FDA, in the last 3-4 years many US agencies have purged data from their websites, and in some cases the employees themselves have protested this in public and begged people to make archives. It's great to have multiple archive sources for that stuff because you never know when it's going to disappear if politics are involved.
gravypod 7y ago
Is there any way to download warcs collected by the internet archive? I'd like to try and back some up and use them for some search algorithm benchmarks.
mastazi 7y ago
> A shutdown announcement put these at risk. We worked with the founder, @tedr (RIP), who'd left the company, to save as much as we could. 3.5 Terabytes I work for a company that owns a website containing information that in my opinion is valuable to the public. The website may go offline forever in the coming months. How can I get in touch with the Archive, to ensure that the content is saved? Parts of the content are not easy to index (e.g. there are "hidden" pages that you will only find if you have the exact URL), I can assist with that.
- ohithereyou 7y ago
  The fastest way is probably the ArchiveTeam IRC channel[1]. [1] https://www.archiveteam.org/index.php?title=Archiveteam:IRC https://www.archiveteam.org/index.php?title=Archiveteam:IRC
  - mastazi 7y ago
    Thank you!
- rahuldottech 7y ago
  Other than irc, you can send a DM to https://twitter.com/textfiles https://twitter.com/textfiles, or email at jason@textfiles.com.
  - mastazi 7y ago
    Thanks!
- lioeters 7y ago
  > a company that owns a website containing information that in my opinion is valuable to the public Please do post your efforts on HN if you succeed in getting the material archived. I'd love to hear more about it.
  - mastazi 7y ago
    I will!
PostOnce 7y ago
I don't understand why archive.org keeps all kinds of formats, .ogg and .mp3, .pdf and .jpg for the same resource? Why not just whatever the original format was? Sometimes its a dozen or so formats it seems like. There are probably a lot of reasons, but I don't know where to chat about it to learn more. edit: now that I think about it, if the original is pdf, then jpeg makes sense for loading one-page-at-a-time in the browser, but it seems like mp3 transcoding to ogg is reasonable to leave up to the user?
- textfiles 7y ago
  The derivations are auto generated for easier access by various clients. In the metadata, they are marked as original versus not original for later discernment and perhaps rerendering.
  - PostOnce 7y ago
    So, then, the convenience outweighs the disk space cost? And also, I guess, transcoding the least-popular format on-the-fly is still too CPU intensive for large files (zips of hundreds of jp2 images, etc) Guess I'm looking for a magic bullet where there might not be one, I just want to see Archive.org keep doing what it's doing far into the future. Who knows, maybe we will stumble on some magic bullet, new compression algos (Zstandard? AVIF? AV1?), user clouds for compression (like boinc; archive already lets users assist via bittorrent for bandwidth costs). Thinking out loud. Anyway, keep doin' what you're doing. And podcasting. That's good too :)
echelon 7y ago
A linear amount of data could be saved if we extricated text content from the HTML skeleton that contains it. I wish Semantic Web had taken off. "Pages with styling" was suboptimal. Web apps are a such a weird evolutionary branch we've descended into that don't relate to documents. Content instead should have fallen under a type of ontology: news item, blog post, technical reference, comment, status update, ... If we'd adopted such a markup grammar and styled around it, we could parse out meaning, have stronger links in the graph, and compress. Semantic Web would have happened if commercial web didn't outpace it.
- account42 7y ago
  Once you start transforming a document you always risk discarding information that later will turn out to be useful. Besides, I bet images, videos and other non-text data dwarf the space required by HTML markup.
  - echelon 7y ago
    Yes, that's what I expect to be the case. I'm just musing over a world built in semantic markup rather than presentational with arbitrary structure.
wolfgke 7y ago
Don't forget https://www.ft.com/content/5be1f2ee-d60b-11e9-a0bd-ab8ec6435630 https://www.ft.com/content/5be1f2ee-d60b-11e9-a0bd-ab8ec6435... (discussed under https://news.ycombinator.com/item?id=21007476 https://news.ycombinator.com/item?id=21007476).
saagarjha 7y ago
One of the replies on Twitter exhorts them to "charge people" in all caps. It'd be a sad to a library put behind a paywall :(
- ghaff 7y ago
  While I tend to agree in this case, lots of libraries are behind a "paywall" in some form or other. The are explicitly membership-only private libraries but many, e.g. university libraries, also restrict access.
azinman2 7y ago
PSA - Many companies will match donations. Apple, my employer, does this through a site called Benevity.
rebuilder 7y ago
The Internet Archive is amazing in many ways. Kind if sadly, for me, the most amazing thing is that they haven't been sued out of existence. How do they manage to operate with copyright laws being what they are?
- earenndil 7y ago
  They've been granted a DMCA exemption.
mellosouls 7y ago
The Internet Archive is a fantastic and important initiative and we should definitely support it. But: Let's also support the public service players in its space that often get forgotten or marginalised by it's well funded marketing. I'm thinking particularly of the various national libraries that preserve content under incredibly tight budgetary, PR and legal constraints that the IA is relatively free of. On IA-aware media like HN, there is a tendency to present it as the only preservation initiative out there, which is absolutely not the case. https://en.m.wikipedia.org/wiki/List_of_Web_archiving_initiatives https://en.m.wikipedia.org/wiki/List_of_Web_archiving_initia...
Bnshsysjab 7y ago
I’m somewhat curious about their backup methods - mirrored drives aren’t great even if they’re stored at two seperate locations. Surely this would get enough support that they could host torrents of the content stored in chunks, and have many peers download and seed many chunks making the backups entirely distributed? I’d gladly seed a few hundred gigs of data to ensure they maintain good backup procedures.
- jve 7y ago
  Not sure why are you being downvoted. That was actually the first thing I thought - what is the backup strategy? Mirroring is for performance + service continuity. Let me remind that there already was a fire that destroyed some equipment for internet archive but luckily not digitalized data: https://www.theverge.com/2013/11/7/5076166/the-internet-archive-seeks-donations-after-fire-destroys-equipment https://www.theverge.com/2013/11/7/5076166/the-internet-arch... Luckily that article from 2013 mentions: > None of the Internet Archive's digitized data was lost in the fire as backups are held in multiple locations.
sdhankar 7y ago
Some things about internet archives which isnt as obvious unless one has been there in person - Housed in a church its not what a typical software development shop might look like, its mix of open office and workshop floor. - A big workforce actually does a lot of manual work of scanning books and transferring from different media types - Its a interesting experience to tour the data center which is in the office itself. I cannot remember but there was something special about those machines to control the heating
duelingjello 7y ago
IA and WBM are great and essential, like a Library of Congress/Smithsonian. What's frustrating about some old websites like Microsoft or Borland's FTP download area is that dynamic links weren't followed and can't be followed and websites that used user-agent filtering. CDN links also weren't captured well. There's so many retro patches that just don't exist publicly. For example, a number of files on SciTech's IA's WBM have zero captures. Most FTP sites weren't captured in WBM adequately either. There are spots of FTP archives hosted here and there on IA and elsewhere, but they're not like WBM for static content sites, and a single snapshot archive lacks the history and the changes, before and after. It is what it is, unless folks donate their vintage personal/work local mirrors to add to the collective.
- nine_k 7y ago
  Donating your copy whenno authoritative source exists is a great chance to retroactively update the past. Anything that was not securely scanned by WBM but was donated should have a different and clearly denoted status, if admitted.
alwillis 7y ago
Let’s not forget that we can all participate in archiving web data using IPFS[1], which the Internet Archive is also using. And coming soon, you’ll be able to get paid to make content available using IPFS and FileCoin[2]. [1]: https://ipfs.io/ https://ipfs.io/ [2]: https://filecoin.io/ https://filecoin.io/
prirun 7y ago
I wrote a Prime minicomputer emulator some years ago and put it online (telnet em.prirun.com 8001). The Prime was a minicomputer from the 80's. I worked with Primes for many years and also worked at the company as an OS specialist for 18 months. Seven versions of the Primos operating system, from rev 18 to rev 24, have been recovered from disks, 9-track tapes, and 8mm backup tapes. The company died around 1992. There has been a huge amount of Prime software that as far as I know, is lost to time. Oracle ran on it. SPSS. A native DBMS. Every one of the major OS revs had 5-10 minor revs. Some software products are only available for certain revs. I actually used rev 12, so at least half of the OS versions are completely missing. And Prime released source for their products. Most of that is missing. For me personally, the emulator was / is a very rewarding project, and maybe for a handful of others who are still alive and used Primes in high school, college, or work. It's been really fascinating to "relive" my Prime days now and then, and others have made similar comments. But how valuable is it really? Is it more than just a curiosity now? Sure, for a current or future computer historian, it might be viewed as a gold mine, but as time goes on, it has less and less value IMO, especially as the people who actual used one die off. If in a few years, only 10 people in the world care about such a thing, does it make sense to save it for all time? I have my doubts. And if I did have all of the versions of Primos, and all of the software ever written for Primes, would it make a difference even now, when there are people alive who actually used this computer system? Also seems doubtful. To me, an "archive everything and let someone else figure out if it's useful" strategy seems impractical. If I had my choice, sure, I'd like to have every major version of the OS and all of the products for each version. But all of the dot-revs too? Nope - not important. Would I like all of the manuals? Yep! Some of the products like Oracle, DBMS, etc. were a bitch to configure back in the day, and certainly would be even harder now with very few people around to help. So even if I did have them, I doubt I could get them running. And even if they were running, they were a specialty product at the time, with few experts, so finding someone today that would be interested in them would be a needle / haystack thing. It all reminds me of family pictures. Before my grandma died, she went through a large box of B/W photos, explaining to us who all the people were, and we wrote on the back of the photos. But just 3 generations after her, the generation after me, no one knows the people in the photos (except my mom is in some - their grandma). In a way it seems that this family history is somehow important, but no one younger than me is interested at all. And truthfully, I never look at the pictures myself. I ended up scanning the photos with my mom or grandma, putting them on a digital picture frame for my nephews, and leaving the rest out. I asked them first - they didn't care about any of the others. I'm not an archivist, but it seems to me that curation is a much more important aspect than grabbing everything and keeping it alive forever. Not curating seems like a "scale to infinity" problem. Those tend to not end well. I think the IA is great project and am not intending to be critical, but giving my own perspective about preserving some very small piece of computer history.
phendrenad2 7y ago
The Internet Archive is great, but it risks becoming a single failure point if we rely on it too much. Also it would be good to take some of the server load off of them. One possible solution is for smaller archives to exist. So if you're interested in archival, and you have some spare time and cash, consider not only donating to IA, but also setting up your own archive site with content on whatever category or topic that you found interesting enough to archive.
- britmob 7y ago
  I like this attitude. This is also why sites like the-eye exist.
- weystrom 7y ago
  Exactly, decentralization is the way to go. Data hoarding is a hobby. Just checkout r/datahoarder.
rikroots 7y ago
The British Library's web archive[1] does similar archiving work, but limits itself to 'British' sites - in other words, sites with a .uk domain. I've had good dealings with the admins before, when I submitted some of my personal sites for inclusion in the archive. Interestingly, the British Library uses web crawlers based on the Internet Archive's Heritrix web crawler[2], which demonstrates how important IA's work is for many other archival organisations' work. [1] https://www.webarchive.org.uk/ https://www.webarchive.org.uk/ [2] https://github.com/internetarchive/heritrix3 https://github.com/internetarchive/heritrix3
S_A_P 7y ago
This is one of the last bastions of the internet I remember as a kid. People publishing knowledge because they could. Sharing things that were interesting. Giving to other people. Sure there has always been seedier elements or commerce, but by and large this was a place to grow and share knowledge. By most every measure today the internet is a better place than it was but I still miss those days...
maxander 7y ago
Perhaps they could get a grant from OpenAI or some similar AI research firm; someday, when the technology arrives, the Internet Archive will be the ultimate corpus.

589 points by danso 7y ago

Internet Archive donations are matched 2-1 currently, for those interested. https://archive.org/donate/ https://archive.org/donate/

deleted 7y ago
[deleted]
xur17 7y ago
Are bitcoin donations included in that?
Akababa 7y ago
Who's matching the donations?
- aeyes 7y ago
  I guess the Pineapple Fund https://pineapplefund.org/ https://pineapplefund.org/
  - textfiles 7y ago
    It is not the pineapple fund, although it has been a lovely time working with them. Regardless of the cryptocurrency discussion, the fact that someone who feels some sort of windfall would immediately have an urge to share it with organizations that need it is laudable.

maxton 7y ago

I donated a week or so ago. The Internet Archive has come in handy many times for me, not just for the Wayback Machine but also things like their live music archives. They're an indispensable resource.

giancarlostoro 7y ago
They also archive ROMs for different consoles.
- _sbrk 7y ago
  Which is a copyright violation, right?
  - giancarlostoro 7y ago
    ROMs are legal at least here in the USA. Doesn't stop big companies from bullying little sites and becoming copyright trolls. The Internet archive will not be easily bullied to take down legally archived items thankfully. The trick is you have to create the ROM yourself apparently. Sharing it is bad. I would assume copies made by anybody would be identical, so it would be interesting to see such a case in court against The Internet Archive, not that I want them to get sued, but would love for them to win if they did. Nintendo is the most aggressive about ROMs than anybody.
    - chongli 7y ago
      Nintendo is the most aggressive about ROMs than anybody They are, but they’re also the most proactive in making their old games available on new hardware. It wouldn’t be fair to call Super Mario Bros “abandonware” because Nintendo has done a lot to keep the game alive and accessible to new generations.
      - vadansky 7y ago
        They have been caught putting other people's "illegal" roms on VC
        giancarlostoro 7y ago
        https://www.eurogamer.net/articles/2017-01-18-did-nintendo-download-a-mario-rom-and-sell-it-back-to-us https://www.eurogamer.net/articles/2017-01-18-did-nintendo-d... Here's an article that asks this very question, not sure if it was posted on HN prior or not.

gatherhunterer 7y ago

Twitter is the worst choice for publishing. This involved so much more text-scrolling and photo-expanding than is necessary. Edit: If anyone would care to explain why they feel that chopping an article up into pieces of arbitrary size makes for a good user experience, please do so. I know Twitter gets a pass from HN in general but it is simply not suited to this type of content by design. This was a conscious choice by its creators.

toomuchtodo 7y ago
https://threadreaderapp.com/thread/1204428311553642496.html https://threadreaderapp.com/thread/1204428311553642496.html
- sprite-1 7y ago
  Thanks! This is great, and even better coupled with Firefox's Reader View
decebalus1 7y ago
No, I learned it's the best choice for publishing [0] [0] https://news.ycombinator.com/item?id=21684008 https://news.ycombinator.com/item?id=21684008
kick 7y ago
I'm sure if you offer to donate someone will put it in a blog format for you. Hint hint. Edit: Twitter is obviously the correct platform for this because of reach. Hacker News is reading it, his 33.3k twitter followers are reading it, reddit is probably reading it, everyone's reading it! So if the goal was to draw more donations, he's winning.
drdrey 7y ago
On Twitter you get built-in liking, sharing and commenting, people will also DM you with questions, it's all one tap away. If you just post a link to a blog post, almost nobody will click it and read it there.
- 7y ago

toyg 7y ago

TIA should get money from the UN. Or be the only beneficiary of a flat tax on network ports - I bet even just $5 on every small router sold in the US (which is basically nothing) would generate a ton of money for them.

starsinspace 7y ago

Although efforts like internet archive are noble (and I find it occasionally useful), I'm not sure it's always so great that everything anyone does online will be permanently archived. I know many people feel that everything should be available forever. But for me... it's pushing me away from doing much on the web. I liked it in the 90s when things were more ephemeral. When you could make mistakes and not have them easily found by anyone with a few clicks, forever.

tcd 7y ago
I'm inclined to agree with this position. Does every Youtube, Reddit, Twitter, Hackernews, Facebook comment need to be archived and stored for the next thousand years? I'd argue no, and there's a huge amount of waste in there - so many bot posts, or just spam. But here we are, people want to archive every byte of information that traverses through the internet. These days you have to approach cautiously, as every thing you do or post may be archived.
- mirimir 7y ago
  > These days you have to approach cautiously, as every thing you do or post may be archived. That was obvious in the 90s. But sadly enough, it hasn't worked out that way. Before the Internet Archive anyway.
  - ghaff 7y ago
    I don't really worry too much. Between all the stuff that doesn't get archived, the fact that the sheer volume tends to effectively hide any single piece of information, and the frequent difficulty of connecting online identities to specific real people, it's not like most people have a perfectly discoverable complete digital record online.
    - mikepurvis 7y ago
      Which is fine as long as you remain anonymous and unimportant in the present day. It becomes much more of an issue if you abruptly become a person of interest, and all of a sudden there's a team of people motivated to go through that archive with a comb looking for the few juicy tidbits needed to publicly humiliate you. And obviously this is already happening— in the Canadian election a few months ago, there was a big scandal where some pictures turned up of the prime minister wearing blackface at a holiday party he attended in 2001. And on top of that there was a pretty steady stream of candidates (some of whom were indeed booted from their parties) who were challenged over social media comments/posts made 5+ years earlier, especially on hot social topics where national sentiment has rapidly evolved in recent times. Perhaps we will all just become inured to this, and rightfully be able to accept our leaders as human, judging them for who they are today and not past words and deeds. But is it possible to still retain the ability to fairly judge someone in the present while forgiving the past? Or will we lose the ability to discern the difference? The GOP's attitude toward president Trump does not give me hope on this.

tyingq 7y ago

Pretty amazing what they do with 1/10th the revenue of Wikimedia and quite a lot more data to manage.

dredmorbius 7y ago
IA are a capture process. Wikipedia supports active updates to content, under contentious circumstances, as well as structuring content for useful access (something IA also does, but under looser conditions). Both are highly worthwhile projects -- some of the best of the Web. But a straight apples-pomellos comparision is difficult.
- tyingq 7y ago
  Wikipedia also gets a lot of that done with free labor. And, over the years, their spend has increased quite a lot more than their content, update frequency, and so on. I think it's a fair enough comparison.

tetha 7y ago

I've discovered their efforts to archive old dos games... fully playable in the browser through a dosbox build in WASM as far as I know. That's a really impressive cooperation of very old and very new technology - 16 bit up to the edge of JS (- edit: or rather, the edge of browser based computing). And on that train of thought, I just had a little flashback about storage sizes. There was some time when 3.14 MB was a big unit of measurement, and some 50MB drive was huge. The classical example: Monkey Island, Indiana Jones or Day of the Tentacle on ~12 floppy disks. But you had to choose which 1 or 2 to install because you didn't have enough space on your hard drive, or you had to swap disks every few screens :) Or, they are archiving a lot of video playthroughs in the lets-play style from video sites, in case those sites go through a meltdown like viddler does. I guess I'm rambling. Point is: This isn't just a storage dump. There are also interesting projects around the internet archive to make the old things accessible on new systems. Very worthwhile donating to.

fyp 7y ago
I relived a part of my childhood by playing Chip's Challenge again the other day: https://archive.org/details/chips_challenge_windows_3.x https://archive.org/details/chips_challenge_windows_3.x
- ocdtrekkie 7y ago
  Note that you can buy Chip's Challenge 1, 2, and Chuck's Challenge 3D from the guy who originally created it. https://store.steampowered.com/developer/niffler https://store.steampowered.com/developer/niffler
  - acomjean 7y ago
    There is an interesting podcast with a "Chuck" Somerville interview. He started on the Apple ][. Its pretty fun walk through the industry in the 80s, and touches on getting the "Chips Challenge" name back from the group that bought it. http://appletimewarp.libsyn.com/apple-time-warp-podcast-episode-2-editedm4a http://appletimewarp.libsyn.com/apple-time-warp-podcast-epis...
7y ago

mirimir 7y ago

It's too bad that there's not a vaguely-somehow-related-but-not-really and impossible-to-censor service that retains stuff that sites have excluded using robots.txt or whatever.

Rebelgecko 7y ago
The good news is that the Wayback Machine stopped making robots.txt retroactive. That handles most of the cases I was running into where content I wanted was removed from the archive.
333c 7y ago
Is this supposed to be a veiled reference to archive.is? I'm not sure what point you're making.
- mirimir 7y ago
  No, I wasn't thinking of that site per se. I was thinking more about the 90s era dream of uncensurable "data havens". That led to Freenet, for example. Which is slow, and forgets stuff that doesn't get accessed. And Tor onion services, which are more readily taken down. But the problem is that there's no way to know in advance whether something is about to disappear from the Internet Archive. So you'd need someone inside who'd discreetly alert the backup service.

nym 7y ago

Setup a monthly $5 donation. It's not much, but I know how valuable the archive is... they're doing the work the library of congress SHOULD be doing.

criddell 7y ago
> I know how valuable the archive is How valuable is it?
- bigwavedave 7y ago
  $5/mo ;)
- Avamander 7y ago
  Data is generally hard to put a value on.
  - criddell 7y ago
    The implication through a lot of this discussion is that it's greater than zero and even that I question.
  - tingletech 7y ago
    When I took an animal communication class in college, there was a formula for the value of information (not data). I don't remember the exact formula, but it was similar Shannon's equation. Basically, the more valuable information is, the larger the change it affects in the probably of what an organisms next behavioral state is. So, if information signaling didn't change the behavior of another organism, it wasn't considered communication.

msla 7y ago

Here's what I thought this would be about: There's a problem with the Wayback Machine in specific which can kill your ability to access it quite silently, unless you know how to use the browser's development tools and interpret headers. It has to do with cookies: Somehow, the Wayback Machine sets cookies... and sets cookies... and keeps setting cookies, until it overflows its own ability to accept cookies. At that point, your browser tries to access a Wayback Machine page, handing the server all of the cookies it currently has, and the server refuses to deal. It absolutely denies everything, sending an error header and a blank page. You have to clear all web.archive.org cookies to get anything at all, at which point it works perfectly. I've completely solved this problem by blacklisting web.archive.org in browser cookie blacklists. I haven't had it happen since then. As far as I'm concerned, the problem is diagnosed and just needs to be solved. At their end.

topherPedersen 7y ago
I've run into this problem as well. I used to use a Chrome extension which I authored that saved all of the pages I browsed to the wayback machine (in another tab), and I would frequently need to clear my cookies to keep the thing working.

peterwwillis 7y ago

It seems like all those things they backed up didn't seem to impact anyone's actual life. People lost files, websites, communications, whatever, and yet they kept right on living life. So I can't help but think that this is all a waste, like people who take pictures everywhere they go, and never look at them later. Is there any useful purpose to all this other than nostalgia? And at what point does the nostalgia overwhelm the useful purpose?

jborichevskiy 7y ago
Seems to me like it could come in very useful in a legal case, with very significant and real impacts on someone's life.
- Zitrax 7y ago
  Also useful for various research, in the same way as going to the archives in a library can be useful. Even if most data is uninteresting doesn't mean that there are data there that is very useful to someone.

redisman 7y ago

I encourage everyone to also do their own thinking of what did you enjoy in the internet or computer software in the 80's, 90's and 00's and see if it is still available somewhere. I found out that the local games scene I used to love as a kid had been almost wiped off the face off the earth. These were small non-commercial and shareware games localized in just one language so it was already a niche. The free hosting services of the 90s are gone so those sites are down and no one wanted to keep paying for hosting for 20 years+ on sites that get very few visitors nowadays. The only way to get these games again was to find a Discord group and a friendly stranger who agreed to seed a torrent (which had 0 seeds when I found it). I'm looking to upload them to a couple of different places and compile a basic website catalogue (static site on CDN) one of these days. For the layperson, these games are already gone from the internet. The internet archive does a great service but it is breadth-first and quite surface level. The depth has to come from people who were familiar with the sites at their peak. And there's a big change that no one is doing that for your specific interest.

oneepic 7y ago
YouTube has many channels with playthroughs of old games as well. (old for me is the 90's and early 00's since i'm in my twenties) You can find things like the old Lego PC games, old educational games like Physicus/Bioscopia, kids' games like Putt-Putt (not in the least because they have a speedrunning community), and much more. It's been a great resource to go back to the stuff I played growing up.
foxthatruns 7y ago
BlueMaxima's Flashpoint is a fabulous archival project that is saving as many Adobe Flash games/animations as they can before browsers pull support at the end of 2020. Really cool, since Flash games are a similarly concrete slice of culture/history that will just be gone if they're not archived. https://bluemaxima.org/flashpoint/ https://bluemaxima.org/flashpoint/
- baroffoos 7y ago
  Looks like its a huge amount of work to strip the DRM out of the games which stop functioning when the original website stops working.

jedberg 7y ago

It seems they could save some money by moving a bunch of infrequently accessed data to warm storage. The entire archive does not need to be accessible 24/7. I would be perfectly ok if I was trying to see a copy of a web page from five years ago, and it said that I had to make a request and it would be available in five or ten minutes. I think I could wait five or ten minutes for a web page to get pulled from the archives.

archi42 7y ago
Oh heck, I would be okay waiting for a few hours, or even a day for huge files. Though I suppose electricity isn't a huge driver for costs, afaict from that thread, the disks are.
anamexis 7y ago
Practically speaking, how would warm storage like that be implemented?
- dredmorbius 7y ago
  Usually tape libraries. A problem with most forms of warm storage is that they involve moving media around, which itself can lead to degradation. I would like to see some write-one and really last forever form of storage to come into widespread use. That seems ... like it's still a few years off. And you still have the issue of how such storage is read, and how that affects total lifetime.
  - ccostes 7y ago
    Sounds like what they're doing at Microsoft with Project Silica https://news.microsoft.com/innovation-stories/ignite-project-silica-superman/ https://news.microsoft.com/innovation-stories/ignite-project...
    - dredmorbius 7y ago
      That definitely is in the direction I'm considering.

notacoward 7y ago

The real "silent killer" I see here is the reliance on mirroring. One-failure protection, with a 2x expansion factor. As it happens, I work on large storage systems, where 2x is our maximum expansion factor and for that we get resistance to as many as nine simultaneous failures. Across power and network failure domains, with multiple kinds of background scrubbing to detect loss of that redundancy. Oh, and 60PB is something we might add to an existing cluster for a day to absorb transient I/O load. There's also a bunch of monitoring and automation stuff that should be considered "table stakes" for storage at these scales. Seems like an opportunity to use what I and others have learned for a good cause, to make this valuable resource more efficient and more durable all at once.

scarejunba 7y ago
Is that different techniques or you just have vastly more resources? I'm sure they'd accept your help in either case if you offered directly.
- jeremysalwen 7y ago
  I think he's referring to error-correcting codes.
- notacoward 7y ago
  It's different techniques (erasure coding). As it turns out, applying those techniques would reduce need for additional physical resources as well, since it allows data to be stored more efficiently with the same physical resources. It does require more CPU and memory relative to bytes stored or spindles to store them on, but it's easy to make that tradeoff and still come out ahead.
diego 7y ago
Did you get to the part where they explain that they are strapped for resources and in need of donations? They have plenty of expertise, money is what they need.
- notacoward 7y ago

joe_the_user 7y ago

So, copyright conditions are apparently another silent killer [1]. A website can be archived, vanish from the web, and then vanish from the archive for technical copyright reasons (new owner's robot.txt file on the root). So "archiving the archive" might be useful. Or something. Ezboard was an old discussion site that contained much of interest - archived and now the archive is not accessible. https://archive.org/post/389127/ezboard-content-suddenly-not-available-in-the-new-system-why https://archive.org/post/389127/ezboard-content-suddenly-not...

btrettel 7y ago
A while back I tried looking up a particular ezboard forum I used to participate in, only to find it was blocked. This is a real shame. I have some of the posts saved but it's only a fraction of the full forum.
deleted 7y ago
[deleted]
tgsovlerkhgsel 7y ago
TIL they still follow robots.txt -- https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... mentioned that they were planning to stop doing that (and I remember reading a news article based on it claiming that they already stopped following robots.txt, hence my confusion). Truly a shame. I get following robots.txt at collection time, I don't get following a robots.txt that was added later.
- joe_the_user 7y ago
  Well that was 2 and 3/4 years ago and ezboard is still gone. I hope they didn't erase but rather excluded but legality and all that.
tingletech 7y ago
I think they still keep the data. I've always wondered why it was called the "The Oakland Archive Policy". http://groups.ischool.berkeley.edu/archive/aps/removal-policy http://groups.ischool.berkeley.edu/archive/aps/removal-polic...

somesortofsystm 7y ago

I hope the first human settlement on another planet takes a backup of the entire Internet Archive and has it safe and sound on another planet. Well, I can imagine that happening in the next 5 years, anyway.

mrmonkeyman 7y ago

But who will archive the archive?

commoner 7y ago

The Internet Archive is absolutely essential to Wikipedia, whose articles are required to be verifiable to reliable sources. When pages go offline, the link rot makes it harder for articles to be verified. However, if the Wayback Machine has an archived copy, that copy can then be cited as the source and made available to readers and editors. The Wayback Machine now automatically archives every new external link added to the English Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Link_rot https://en.wikipedia.org/wiki/Wikipedia:Link_rot The Wikimedia Foundation's budget is about 10 times that of the Internet Archive. If you see the fundraising banner on Wikipedia and want to help out the site, but don't think the Wikimedia Foundation needs the donation, consider donating to the Internet Archive instead. https://archive.org/donate https://archive.org/donate

Bartweiss 7y ago
This is a great point, thank you. It's been awesome to see archiving and Archive links become standard practice on Wikipedia. For a long time, links just rotted and were replaced. Then later, dead links would be switched to Archive content if it was available. But it's so much easier and more reliable to archive a page when it's referenced, and provide both links even while the page is still up. It's one thing to automate dead link detection, but archiving defends against a lot more. Universities seem to redo their sitemaps and redirect all their old links about once a year. And reference pages get edited, which goes unnoticed for a long time and then leads to frustrating [Not in citation given] tags. It's incredible what a boon the Internet Archive has been to wiki sourcing.
ignoramous 7y ago
> The Wayback Machine now automatically archives every new external link added to the English Wikipedia. Links submitted to Hacker News should be auto-archived, too: I often stumbled upon dead-links [0] which otherwise had generated insightful discussion on news.yc. Adding archive next to web would work nicely. [0] Launch HNs, in particular.
- scrollaway 7y ago
  It is possible to do that nowadays and that is one of the things I suggested implementing in tildes. https://gitlab.com/tildes/tildes/issues/586 https://gitlab.com/tildes/tildes/issues/586 I believe all link aggregators should do this, they're a great way of getting defacto quality/interesting content discovered by the archive.

corporate_shi11 7y ago

I found out about the Wayback Machine back in 2007 or so. I was young and just starting to explore the internet and found it incredibly fascinating to explore news sites as they were on the day of major historical events (9/11, etc). The amount of times I've relied on the Internet Archive by way of the Wayback Machine is numerous. I'll be adding them to my modest donation list, along with Wikipedia and similar public service sites.

lazylizard 7y ago

What does marie kondo think?

catalogia 7y ago
What does lazylizard think?

HenryKissinger 7y ago

Why not just let old data die? What's the point in devoting 10TB to saving data which maybe 300 people will check in the next 25 years? Information destruction is not necessarily a bad thing.

tingletech 7y ago
Selection is destruction, and destruction is selection. (but not even a blackhole can destroy information?)

bpaddock 7y ago

The Food and Drug Administration (FDA) has been archiving the FDA.gov site at archive-it.org . Leaving a lot of dead links on the FDA site. Sometimes they tell you to look in the archives for the old information, without giving you a link to it, and sometimes they don't, they just expect you to know. Now why can't the FDA afford the space to keep their pages forever on their own site? Fill in your favorite conspiracy theory... Some of the information that has been removed, such as the 2015 hearings on Fluoroquinolone antibiotics, are important health research as just one example. https://archive-it.org/organizations/1137 https://archive-it.org/organizations/1137

Analemma_ 7y ago
> Now why can't the FDA afford the space to keep their pages forever on their own site? Fill in your favorite conspiracy theory... This seems like a prime example of Hanlon's razor. Tight government budgets and lowest-bidder contractors not bothering with page permanence strike me as the most likely explanation.
- ghaff 7y ago
  It may also be worth observing that it's how pretty much any private company website operates. Arguably the FDA should be different and archive old, even outdated, content. But, while private companies may explicitly archive some materials like press releases and earnings reports, 99.9% of their focus is on the current content and they'll mostly just delete anything that's not in service of today.
kevingadd 7y ago
It's not limited to the FDA, in the last 3-4 years many US agencies have purged data from their websites, and in some cases the employees themselves have protested this in public and begged people to make archives. It's great to have multiple archive sources for that stuff because you never know when it's going to disappear if politics are involved.

gravypod 7y ago

Is there any way to download warcs collected by the internet archive? I'd like to try and back some up and use them for some search algorithm benchmarks.

mastazi 7y ago

> A shutdown announcement put these at risk. We worked with the founder, @tedr (RIP), who'd left the company, to save as much as we could. 3.5 Terabytes I work for a company that owns a website containing information that in my opinion is valuable to the public. The website may go offline forever in the coming months. How can I get in touch with the Archive, to ensure that the content is saved? Parts of the content are not easy to index (e.g. there are "hidden" pages that you will only find if you have the exact URL), I can assist with that.

ohithereyou 7y ago
The fastest way is probably the ArchiveTeam IRC channel[1]. [1] https://www.archiveteam.org/index.php?title=Archiveteam:IRC https://www.archiveteam.org/index.php?title=Archiveteam:IRC
- mastazi 7y ago
  Thank you!
rahuldottech 7y ago
Other than irc, you can send a DM to https://twitter.com/textfiles https://twitter.com/textfiles, or email at jason@textfiles.com.
- mastazi 7y ago
  Thanks!
lioeters 7y ago
> a company that owns a website containing information that in my opinion is valuable to the public Please do post your efforts on HN if you succeed in getting the material archived. I'd love to hear more about it.
- mastazi 7y ago
  I will!

PostOnce 7y ago

I don't understand why archive.org keeps all kinds of formats, .ogg and .mp3, .pdf and .jpg for the same resource? Why not just whatever the original format was? Sometimes its a dozen or so formats it seems like. There are probably a lot of reasons, but I don't know where to chat about it to learn more. edit: now that I think about it, if the original is pdf, then jpeg makes sense for loading one-page-at-a-time in the browser, but it seems like mp3 transcoding to ogg is reasonable to leave up to the user?

textfiles 7y ago
The derivations are auto generated for easier access by various clients. In the metadata, they are marked as original versus not original for later discernment and perhaps rerendering.
- PostOnce 7y ago
  So, then, the convenience outweighs the disk space cost? And also, I guess, transcoding the least-popular format on-the-fly is still too CPU intensive for large files (zips of hundreds of jp2 images, etc) Guess I'm looking for a magic bullet where there might not be one, I just want to see Archive.org keep doing what it's doing far into the future. Who knows, maybe we will stumble on some magic bullet, new compression algos (Zstandard? AVIF? AV1?), user clouds for compression (like boinc; archive already lets users assist via bittorrent for bandwidth costs). Thinking out loud. Anyway, keep doin' what you're doing. And podcasting. That's good too :)

echelon 7y ago

A linear amount of data could be saved if we extricated text content from the HTML skeleton that contains it. I wish Semantic Web had taken off. "Pages with styling" was suboptimal. Web apps are a such a weird evolutionary branch we've descended into that don't relate to documents. Content instead should have fallen under a type of ontology: news item, blog post, technical reference, comment, status update, ... If we'd adopted such a markup grammar and styled around it, we could parse out meaning, have stronger links in the graph, and compress. Semantic Web would have happened if commercial web didn't outpace it.

account42 7y ago
Once you start transforming a document you always risk discarding information that later will turn out to be useful. Besides, I bet images, videos and other non-text data dwarf the space required by HTML markup.
- echelon 7y ago
  Yes, that's what I expect to be the case. I'm just musing over a world built in semantic markup rather than presentational with arbitrary structure.

wolfgke 7y ago

Don't forget https://www.ft.com/content/5be1f2ee-d60b-11e9-a0bd-ab8ec6435630 https://www.ft.com/content/5be1f2ee-d60b-11e9-a0bd-ab8ec6435... (discussed under https://news.ycombinator.com/item?id=21007476 https://news.ycombinator.com/item?id=21007476).

saagarjha 7y ago

One of the replies on Twitter exhorts them to "charge people" in all caps. It'd be a sad to a library put behind a paywall :(

ghaff 7y ago
While I tend to agree in this case, lots of libraries are behind a "paywall" in some form or other. The are explicitly membership-only private libraries but many, e.g. university libraries, also restrict access.

azinman2 7y ago

PSA - Many companies will match donations. Apple, my employer, does this through a site called Benevity.

rebuilder 7y ago

The Internet Archive is amazing in many ways. Kind if sadly, for me, the most amazing thing is that they haven't been sued out of existence. How do they manage to operate with copyright laws being what they are?

earenndil 7y ago
They've been granted a DMCA exemption.

mellosouls 7y ago

The Internet Archive is a fantastic and important initiative and we should definitely support it. But: Let's also support the public service players in its space that often get forgotten or marginalised by it's well funded marketing. I'm thinking particularly of the various national libraries that preserve content under incredibly tight budgetary, PR and legal constraints that the IA is relatively free of. On IA-aware media like HN, there is a tendency to present it as the only preservation initiative out there, which is absolutely not the case. https://en.m.wikipedia.org/wiki/List_of_Web_archiving_initiatives https://en.m.wikipedia.org/wiki/List_of_Web_archiving_initia...

Bnshsysjab 7y ago

I’m somewhat curious about their backup methods - mirrored drives aren’t great even if they’re stored at two seperate locations. Surely this would get enough support that they could host torrents of the content stored in chunks, and have many peers download and seed many chunks making the backups entirely distributed? I’d gladly seed a few hundred gigs of data to ensure they maintain good backup procedures.

jve 7y ago
Not sure why are you being downvoted. That was actually the first thing I thought - what is the backup strategy? Mirroring is for performance + service continuity. Let me remind that there already was a fire that destroyed some equipment for internet archive but luckily not digitalized data: https://www.theverge.com/2013/11/7/5076166/the-internet-archive-seeks-donations-after-fire-destroys-equipment https://www.theverge.com/2013/11/7/5076166/the-internet-arch... Luckily that article from 2013 mentions: > None of the Internet Archive's digitized data was lost in the fire as backups are held in multiple locations.

sdhankar 7y ago

Some things about internet archives which isnt as obvious unless one has been there in person - Housed in a church its not what a typical software development shop might look like, its mix of open office and workshop floor. - A big workforce actually does a lot of manual work of scanning books and transferring from different media types - Its a interesting experience to tour the data center which is in the office itself. I cannot remember but there was something special about those machines to control the heating

duelingjello 7y ago

IA and WBM are great and essential, like a Library of Congress/Smithsonian. What's frustrating about some old websites like Microsoft or Borland's FTP download area is that dynamic links weren't followed and can't be followed and websites that used user-agent filtering. CDN links also weren't captured well. There's so many retro patches that just don't exist publicly. For example, a number of files on SciTech's IA's WBM have zero captures. Most FTP sites weren't captured in WBM adequately either. There are spots of FTP archives hosted here and there on IA and elsewhere, but they're not like WBM for static content sites, and a single snapshot archive lacks the history and the changes, before and after. It is what it is, unless folks donate their vintage personal/work local mirrors to add to the collective.

nine_k 7y ago
Donating your copy whenno authoritative source exists is a great chance to retroactively update the past. Anything that was not securely scanned by WBM but was donated should have a different and clearly denoted status, if admitted.

alwillis 7y ago

Let’s not forget that we can all participate in archiving web data using IPFS[1], which the Internet Archive is also using. And coming soon, you’ll be able to get paid to make content available using IPFS and FileCoin[2]. [1]: https://ipfs.io/ https://ipfs.io/ [2]: https://filecoin.io/ https://filecoin.io/

prirun 7y ago

I wrote a Prime minicomputer emulator some years ago and put it online (telnet em.prirun.com 8001). The Prime was a minicomputer from the 80's. I worked with Primes for many years and also worked at the company as an OS specialist for 18 months. Seven versions of the Primos operating system, from rev 18 to rev 24, have been recovered from disks, 9-track tapes, and 8mm backup tapes. The company died around 1992. There has been a huge amount of Prime software that as far as I know, is lost to time. Oracle ran on it. SPSS. A native DBMS. Every one of the major OS revs had 5-10 minor revs. Some software products are only available for certain revs. I actually used rev 12, so at least half of the OS versions are completely missing. And Prime released source for their products. Most of that is missing. For me personally, the emulator was / is a very rewarding project, and maybe for a handful of others who are still alive and used Primes in high school, college, or work. It's been really fascinating to "relive" my Prime days now and then, and others have made similar comments. But how valuable is it really? Is it more than just a curiosity now? Sure, for a current or future computer historian, it might be viewed as a gold mine, but as time goes on, it has less and less value IMO, especially as the people who actual used one die off. If in a few years, only 10 people in the world care about such a thing, does it make sense to save it for all time? I have my doubts. And if I did have all of the versions of Primos, and all of the software ever written for Primes, would it make a difference even now, when there are people alive who actually used this computer system? Also seems doubtful. To me, an "archive everything and let someone else figure out if it's useful" strategy seems impractical. If I had my choice, sure, I'd like to have every major version of the OS and all of the products for each version. But all of the dot-revs too? Nope - not important. Would I like all of the manuals? Yep! Some of the products like Oracle, DBMS, etc. were a bitch to configure back in the day, and certainly would be even harder now with very few people around to help. So even if I did have them, I doubt I could get them running. And even if they were running, they were a specialty product at the time, with few experts, so finding someone today that would be interested in them would be a needle / haystack thing. It all reminds me of family pictures. Before my grandma died, she went through a large box of B/W photos, explaining to us who all the people were, and we wrote on the back of the photos. But just 3 generations after her, the generation after me, no one knows the people in the photos (except my mom is in some - their grandma). In a way it seems that this family history is somehow important, but no one younger than me is interested at all. And truthfully, I never look at the pictures myself. I ended up scanning the photos with my mom or grandma, putting them on a digital picture frame for my nephews, and leaving the rest out. I asked them first - they didn't care about any of the others. I'm not an archivist, but it seems to me that curation is a much more important aspect than grabbing everything and keeping it alive forever. Not curating seems like a "scale to infinity" problem. Those tend to not end well. I think the IA is great project and am not intending to be critical, but giving my own perspective about preserving some very small piece of computer history.

phendrenad2 7y ago

The Internet Archive is great, but it risks becoming a single failure point if we rely on it too much. Also it would be good to take some of the server load off of them. One possible solution is for smaller archives to exist. So if you're interested in archival, and you have some spare time and cash, consider not only donating to IA, but also setting up your own archive site with content on whatever category or topic that you found interesting enough to archive.

britmob 7y ago
I like this attitude. This is also why sites like the-eye exist.
weystrom 7y ago
Exactly, decentralization is the way to go. Data hoarding is a hobby. Just checkout r/datahoarder.

rikroots 7y ago

The British Library's web archive[1] does similar archiving work, but limits itself to 'British' sites - in other words, sites with a .uk domain. I've had good dealings with the admins before, when I submitted some of my personal sites for inclusion in the archive. Interestingly, the British Library uses web crawlers based on the Internet Archive's Heritrix web crawler[2], which demonstrates how important IA's work is for many other archival organisations' work. [1] https://www.webarchive.org.uk/ https://www.webarchive.org.uk/ [2] https://github.com/internetarchive/heritrix3 https://github.com/internetarchive/heritrix3

S_A_P 7y ago

This is one of the last bastions of the internet I remember as a kid. People publishing knowledge because they could. Sharing things that were interesting. Giving to other people. Sure there has always been seedier elements or commerce, but by and large this was a place to grow and share knowledge. By most every measure today the internet is a better place than it was but I still miss those days...

maxander 7y ago

Perhaps they could get a grant from OpenAI or some similar AI research firm; someday, when the technology arrives, the Internet Archive will be the ultimate corpus.