11 ms·

Internet is losing its memory: Cerf

134 points by adrian_mrd 8y ago

kwhitefoot 8y ago
The comment about Archimedes and infinitesimals makes reminds me of something that has been bothering me for a long time: what brilliant ideas are there lurking in the world unknown to almost all of us that could change the course of history if only they were publicised?
- trevyn 8y ago
  Another take: What brilliant ideas exist publicly today that have not yet been recognized as brilliant? And how might you go about systematically identifying them? And what might your incentives be for doing so?
  - AnIdiotOnTheNet 8y ago
    “These days, the problem isn’t how to innovate; it’s how to get society to adopt the good ideas that already exist.” — Douglas Engelbart
- bcaa7f3a8bbc 8y ago
  I have a similar thought experiment (or Sci-Fi plot?) for a long time. If an advanced alien civilization came to Earth, and helped ancient human civilizations to built and operate a computer system, which can share, store and translate scientific discoveries all across the globe (while they are not allowed to help humans besides operating the system) throughout 5000 years of history - how could it change the human history?
  - gsanghera 8y ago
    The ancient humans would then also (eventually) develop patents because someone somewhere would have thought about it even back then - and with worldwide distribution - it would catch on. Effectively, what I'm saying is - when faced with a common greater good that is a game changer for humanity, humanity then has to collectively want the greater good. And humans can be selfish - ancient or not.
devilmoon 8y ago
Slightly related to this article, I recommend this Tedx talk by Cerf himself: https://www.youtube.com/watch?v=GV0A82TCrf0 https://www.youtube.com/watch?v=GV0A82TCrf0
stillkicking 8y ago
In many ways he's understating the problem. At least there used to be file formats to support, and files to lose. But how much of the data you interact with daily is even directly accessible to you? How much of it can you access when you're offline? Take Slack. You can install it locally, but that doesn't matter, because it won't start without an internet connection, and its local storage is completely opaque. Compare this to a mail client, where everything is stored locally, indexed and searchable offline, and can be exported into a universal albeit messy MIME format, as well as imported into other accounts. Of course this is necessary for the business model, the Slack free-plan event horizon wouldn't be effective if it only applied to which new data you can sync down... and if users only discovered this when e.g. moving to a new computer, they would quickly start to wonder why they can't just transfer the data they already have themselves. With Google Drive, your 'docs' are just placeholders pointing to the cloud. You can export them individually to Word or print them to PDF, but it's a manual process, which you will likely only think of when it's too late. For an example of how this can really matter: an acquaintance is embroiled in a legal dispute with an ex-employer. Thanks to Apple's sane implementation of IMAP Mail on iOS, she still has access to all her company communications there to use as evidence. Unlike on her PC, where she was just using webmail the entire time and has nothing. The incentive in the cloud age is to create dysfunctional products that provide an illusion of permanence instead of the reality of tangibility. I expect this is only going to become a bigger problem over time.
- knorker 8y ago
  > You can export them individually to Word or print them to PDF Actually it's really really easy to do in bulk: https://takeout.google.com/settings/takeout?pli=1 https://takeout.google.com/settings/takeout?pli=1 Not invalidating the rest of what you said.
  - dredmorbius 8y ago
    Google's data take-out has proven fairly robust.
  - 013a 8y ago
    I think the rest of what they said matters, though. Its not "preservable by default". If you need access to that data in any situation where your connectivity to Google or that account is severed, you're shit out of luck. The best case example is if you don't have internet. The bad case is really that "legal dispute" argument, where you're dealing with a bad actor who has power over you and your Google account. The worst case: Google themselves severs your access.
WalterBright 8y ago
By contrast, before the internet, most everything was routinely lost anyway. Only papers from famous people were kept. Even so, for example, HP's historical archive was all on paper and placed in a single building. Which burned down. The WTC collapse destroyed the unpublished archive of Kennedy pictures.
- JdeBP 8y ago
  David Granick wasn't famous, yet his works were preserved, by a library. * https://news.ycombinator.com/item?id=16472614 https://news.ycombinator.com/item?id=16472614 * https://news.ycombinator.com/item?id=16363283 https://news.ycombinator.com/item?id=16363283
- vertere 8y ago
  It is true that a lot more stuff gets kept these days. But we seem to be much more liable to losing important information. As Bret Victor [0] has pointed out, we use largely the same mechanisms for managing the most trivial to the most important information, and most people don't really understand how they work. > a 2013 survey of law- and policy-related publications found that, at the end of six years, nearly fifty per cent of the URLs cited in those publications no longer worked. According to a 2014 study conducted at Harvard Law School, "more than 70% of the URLs within the Harvard Law Review and other journals, and 50% of the URLs within United States Supreme Court opinions, do not link to the originally cited information." [1] I doubt that the US Supreme Court was regularly losing track of cited documents before "information technology" came along. A lot of documents may be recoverable even if the original URLs are broken, but is this really the best we can do to manage information in the information age? [0]: http://worrydream.com/TheWebOfAlexandria/2.html http://worrydream.com/TheWebOfAlexandria/2.html [1]: https://www.newyorker.com/magazine/2015/01/26/cobweb https://www.newyorker.com/magazine/2015/01/26/cobweb
  - c12 8y ago
    The fact that urls cited in publications no longer work over time is why I often included their content as apendix's to my essays. That way even if the original source is lost the referenced material isn't. Then again material published in journals and books is greater than material published on the internet, if for nothing other than it can be relied upon to be found in decades to come.
pmlnr 8y ago
http://longbets.org/601/ http://longbets.org/601/ https://adactio.com/journal/11937 https://adactio.com/journal/11937 "The original URL for this prediction (www.longbets.org/601) will no longer be available in eleven years." This has been a known problem for a while, but it's always nice to see more voices crying out about it to the more general public.
bcaa7f3a8bbc 8y ago
Think about it, we still have almost-complete versions of Usenet archives from the 70s to the late-90s, the whole network has been preserved as if it was a time capsule. You can visit https://olduse.net https://olduse.net (maintained by a retired Debian developer), and see the heyday of hackers and early technological adopters as if it was just yesterday, the entire society is archived here. Discussing about the 4-color theorem recently proved, latest version of C compiler, difference between a vacuum tube amplifier and a solid-state amplifier, where GNU Project and Linux kernel was launched, and early online culture and tons of colorful, hilarious, but forgotten and buried memes, and weird phenomena emerged from the collective (un)consensus... Sci-Fi fandom being an integral part of online and hacker culture, millions of lewd story written in alt.sex, "Immediate Death of Usenet Predicted!", "There is no cabal", alt.french.captain.borg.borg.borg, Coffee and Cat warning, The church of Kibology, Anti-spam Movement, creationism vs evolutionism debates at talk.origin, Meow Wars - the first meme war online, all the personal attacks, trolls, flame wars, and "cyber-stalking", etc. Then centralized WWW replaced distributed Usenet, crappy HTML replaced perfect machine-readable data format. Would we have a similar archive for Reddit or Hacker News? Possibly not. So Hacker News, just come and create one! You can make it! Anther unique challenge created by WWW is the inaccessibility of server-side software - exporting and preserving the data is NOT enough, unlike Usenet which you can just load any data. The user-interface and functionality of one website itself is also the collective memory that needs to be preserved - we need replicated software of a website, which has identical user-interface, which has all the functions from the original website: users to click an username and see the posts, karma of this user, etc. I don't think anyone even noticed the existence of this problem. Luckily, major websites online such as Reddit or 4chan, all use FLOSS software which would make the work easier, but still a huge challenge due to the inaccessibility of raw database. Also, to make some contents meaningful in the future, external resources such as hyperlinks to other websites and images should also be preserved, considering this, the chance of creating an authentic and complete archive is even lower. --- But even if we're still using a distributed network where data preservation is still technically possible, and there is no walled garden, it may still be difficult to implement. In the era of Usenet, you often attach your name, address and phone number - there was virtually no threats except for a few trolls - this is why archiving Usenet was possible in the first place. But the Internet is not the Net anymore, now not only humans - almost every piece of equipment involved on the route may be your enemy. The ongoing security and privacy movement is a huge threat of historical records. From my observation, at least of infosec hackers community - After Snowden's revelation, public and open discussion is slowing being transformed into private, closed, encrypted and temporary activity, plus self-hosted platforms like ActivityPub, GNU/Social, Mastodon. This is indeed good from a security and privacy perspective and it is exactly what we need now. But we are also creating a huge gap of knowledge, information and history on the Internet. After my death, none of my self-hosted code, or my blog, or my GNU/Social posts will survive. In conclusion, "collect 'em all" is both an malicious NSA dragnet surveillance, and a glorified act of history preservation. This is where the contradiction lies. I don't know what to do. For WWW, archive.org is a workaround and I think it needs more donation. But for all the other self-hosted things like Mastodon and git server, there is no solution at all.
deleted 8y ago
[deleted]
imhoguy 8y ago
Adobe Flash player (closed SWF format) death takes toll also with uncountable number of human artistic creations, games etc. I wonder if there is some way to preserve that relic of 2000s.
- icebraining 8y ago
  Seems like there are a few people still working on Shumway, the flash engine written in JavaScript.
  - theandrewbailey 8y ago
    I doubt anyone is working on Shumway. Its in the Firefox graveyard. https://bugzilla.mozilla.org/describecomponents.cgi?product=Firefox%20Graveyard https://bugzilla.mozilla.org/describecomponents.cgi?product=...
    - icebraining 8y ago
      From Mozilla yes, but there was some activity by others this year: https://github.com/mozilla/shumway/pull/2442 https://github.com/mozilla/shumway/pull/2442
- MaxBarraclough 8y ago
  > I wonder if there is some way to preserve that relic of 2000s. Flash is fading from mainstream use on the web (thankfully), but not hard to run Flash today if you want to.
- RandallBrown 8y ago
  Seems like virtual machines will be able to run flash for a long long time.
deleted 8y ago
[deleted]
AnIdiotOnTheNet 8y ago
I'm probably weird in that my philosophy doesn't jive well with the current societal norm, but I think it is important to come to terms with a simple truth: All things are ultimately lost. Physics tells that even the universe itself will end, even if different models disagree as to precisely how. Things begin, they exist, and then they end. That they are not eternal does not mean that their existence was meaningless. Which isn't to say that preservation is pointless or wrong, but I think it is unhealthy to try and hang on too firmly to the past.
- 013a 8y ago
  I think that's fighting human nature. One of the things that separates us from other creatures is our constant desire to fight entropy. Sentient life itself is the ultimate reversal of entropy; while the universe becomes more complex around us, we reduce complexity in every environment we enter by organizing matter to build more and more efficient systems. Its not crazy to think that, if we live that long, we will figure out a way to escape even the end of the universe (I'm reminded of Asimov's The Last Question). But that progress definitely won't happen if future generations can't build on what we've learned. I actually think living with the idea that everything is temporary is very very dangerous. Nothing is temporary, except you. We need to live with the humility to recognize that our life on this planet is incomprehensibly short, and we need to spend it doing everything we can to give our children the headstart they need to do everything they can to give their children a headstart... into infinity. That's the only way we improve as a species. That's why there is no greater sin than destroying or limiting access to information, and conversely why the internet is literally the most substantial development in the history of mankind.
  - AnIdiotOnTheNet 8y ago
    That doesn't separate us from other living things, or even natural processes, in any way. Many things appear to sew order in the universe on the local scale, but the math is pretty clear that this isn't actually fighting entropy at all. The Last Question is a work of fiction. All works of fact seem to support the concept that it is not possible. That isn't to say that we will never discover properties of the universe that open new possibilities, but A) I think eternity is just a different kind of ending anyway, and B) stubbornly grasping to such a concept seems a desperate attempt to deny one's own mortality. I'm not advocating for forgetting things for the sake of it, I'm advocating against the idea that everything must be preserved. Certainly we should try to keep in mind the lessons of our past, but we should also be willing to let go of things that haven't given us good reason to keep them around.
jacquesm 8y ago
Due to the Curse of SaaS soon the web won't be about documents any more but about applications that have no obligation to preserve anything. The only thing you can be sure of is that your advertising profiles will be stored for the next 100 years or so.
013a 8y ago
Frankly, its somewhat hilarious to hear all this coming from Cerf, who works for the worst offenders of data centralization of all time. Think of the data that gets lost on Drive every day due to their product decision around the SaaS model + proprietary document formats, or the data that was lost/hidden on Code, Wave, Plus, their shattered IM platforms, etc.
dahart 8y ago
At first I thought this was saying all information on the internet should be preserved (which is a growing privacy concern), but after reading more carefully I think it’s about having a way to preserve the information that is intended to be public and permanent from the beginning. We don’t have any good ways yet to guarantee future access to public information. I’d never heard of the Digital Object Architecture (DOA). The article itself makes light of the unfortunate acryonym with “History pronounced DOA”. That actually left me confused about what they were talking about for a minute. No idea if it’s a good way to preserve academic papers on the internet, but the business model side of it is a pure open question. That makes me wonder whether it solves anything at all. The problem with information on the internet is that the people who publish eventually lose the interest or the ability to continue paying for storage and access. “Economics/Business Model While the Handle System has been used for many years in publishing and library systems, generalizing to other applications, e.g., Internet of Things, will likely generate economic concerns related to the business model of the system, especially at the Global Handle Registry. Will organizations be charged for each identifier? Will organizations that acquire a prefix be able to create unlimited sub-prefixes or will they be charged for each sub-prefix? How will these policies be developed? How will the money flow? What will be the impact on developing countries or small businesses?” https://www.internetsociety.org/resources/doc/2016/overview-of-the-digital-object-architecture-doa/ https://www.internetsociety.org/resources/doc/2016/overview-...
unquietcode 8y ago
It's worth noting that in complex systems, forgetting can be just as important as remembering. The ability to evolve and change is in part predicated on the ability to selectively forget some elements that are no longer helpful. As another commenter pointed out, some aspects of our society's cultural memory are, probably, best left in the past, if preserved at all.
- notabee 8y ago
  Knowing what should be forgotten requires either foreknowledge of the future or relies on inaccurate predictions based on current assumptions. Thus the less information of uncertain usefulness that is retained, the more that those assumptions dictate what will be considered useful in the future without having to rediscover things entirely. That limits adaptability. Just try to imagine how many things were invented in the past that weren't seen as useful at the time and had to be rediscovered later. How many ideas were lost in the Dark Ages, for instance, because they offended religious sensibility at the time? If there is capacity to retain information in an organized way without undue cost, it should be retained as a hedge against future uncertainty. No single generation of humans should be trusted to make such decisions without being unduly influenced by the biases of their time.
sulam 8y ago
We have been forgetting things for thousands of years. Last I looked, we still don’t know how to make Damascus steel or Roman concrete. People will reinvent the things they need and maybe never even know that it’s a reinvention. I reinvented a couple fundamental graphics techniques because I needed them and didn’t study computer graphics in school. I’m sure this happens all the time. I’m all for making it easier to recover data and generally make it easier to store stuff. And I believe you have a right to your data (yay GDPR). But this isn’t a catastrophe.
- arbitrage 8y ago
  > we still don’t know how to make Damascus steel or Roman concrete That's an oft-repeated but ultimately meaningless statement. We know how to make things better than Damascus steel or Roman concrete. Our processes have exceeded the ancients. The quality of Damascus steel is likely overrated, since it wasn't quite as absolute shit as what was being regularly traded at the time. Same with concrete. We might not have exquisite written recipes and procedures for these materials, but they have been reverse-engineered, and it turns out, they weren't that great compared to modern chemical and material engineered products. We don't need to 'rediscover' Damascus steel because it is obselete. A romantic idea, and poetic in how it was 'lost' to time, but consider that it was 'lost' to time, similar to Japanese steel-working, because nobody wanted to buy it anymore. It became economically and culturally irrelevant.
  - sulam 8y ago
    Exactly. We forgot how to do a thing and invented better. The “lost” knowledge isn’t really worth much to us today. And that will be true of essentially everything we forget in the future as well — the forgetting is the sign that it wasn’t needed.
paidleaf 8y ago
Losing memory via attrition is certainly a concern. But I think the bigger concern is losing memory via censorship and copyright. For most people, the internet is now google search and a couple of social media companies. Google has been curating search results for a while now. We no longer see what is out there in the internet, but a narrow google approved view of the internet. For example, 10 years ago, google showed us the entire elephant ( more or less ), today, we only get to see the tail or the trunk or whichever part of the elephant google decides to frame. Now all of the social media companies are now curating. And with the loss of neutrality, ISPs will undoubtedly curate. And then there is the copyright issues where archiving is getting more difficult. Kids today don't know what a great tool google cache was because it's gone. And archiving sites are being attack from news, media, politicians, etc. In other words, the internet dark age isn't going to be a result of formats getting old ( though that is an issue ). It's going to be a result of us only being allowed to see through the frame that a handful of companies deem appropriate. The dark ages didn't happen because formats got old. The dark ages happened because the powerful decided that we should view the world within a certain narrow religious frame and censored everything else.
- rezeroed 8y ago
  In the olden days, everyone was slapping together their own websites, rather than using the same old templated wordpress, hugo, etc, so there was far more character, and not everything was a blog. And so people surfed the web. These days the web is more purpose oriented, people looking for something, the "surfing" all within walls, wave pools.
  - floren 8y ago
    The days of the web ring, which often as not was broken or took you through dozens of content-free sites but could occasionally drop you into something fantastic. Browsing Internet Directories. Just plain finding some guy's massive meticulously-maintained HTML-only site full of fascinating articles, bookmarking it, slowly working your way through it over the course of your evening browsing sessions.
  - TeMPOraL 8y ago
ravenstine 8y ago
People need to keep their own personal archives. While there are great projects like archive.org, it's becoming more clear that it's not enough. There's not enough resources for an organization(that isn't spying on everyone) to archive YouTube, or Slack history, etc., so it's up to you to back up that data if it's important to you. Culture can be easily lost to the void if we don't do this. As I've mentioned in another HN thread, I've been using youtube-dl to back up YouTube videos that are important to me since it's increasingly obvious that YouTube is merely a profit machine that would delete all of its user's videos in one fell swoop if they made an extra buck off it. I've also begun to back up some blogs that have vast mounts of great content, but I've found that HTTrack is... well, terrible. That's not to say it doesn't technically do its job, but it's way too aggressive. I wan't something that's easier to configure and can do a better job of taking "snapshots" of blog entries while blocking ads, javascripts, being able to resume properly if my power goes out, etc.
- abathur 8y ago
  There may be more modern projects, but there's also "Lots of Copies Keep Stuff Safe" at https://www.lockss.org https://www.lockss.org When you say personal archives, I wonder a little about what we'd lose to filtering by people who know they're building an archive for posterity, versus people in the future rediscovering things that were safely forgotten for some long span of time. At the delta of personal archives and projects like this, I imagine something a little like SETI or folding for home archival (is this just describing IPFS?), but I think one of the challenges is how we preserve the right kind of data. I did some spring-data-cleaning this year while building a new system, and one of my tasks was addressing duplicate files. This forced me to confront hundreds of gigabytes of VM images, snapshots, and massive piles of duplicate files from various installations of Ruby and Python. But perhaps I underestimate the value of even the duplicate VM images or language installs. Perhaps, centuries hence, researchers will hopefully sift yottabyte of data looking for all of the dependencies to successfully resurrect applications they need to experience in order to understand references to them in well-stewarded tweets, posts, or articles.
- txsh 8y ago
  I noticed old articles I had bookmarked were disappearing so I’ve started using a self-hosted Bookstack to save a copy of everything interesting I read. I enter reader mode and copy eveeything on the page, and then paste it into the WYSIWYG editor. Everything is formatted nicely and it even creates a menu using the head tags of the sections. The only problem is, while images copy correctly, they’re still hosted on the original server so I have to replace them with an uploaded copy. Only takes a couple of minutes though. I could probably automate this with a script.
rmorey 8y ago
In case anyone hasn't already heard if it, IPFS(https://ipfs.io https://ipfs.io) is a really cool project that aims to help solve this problem
- mbowcutt 8y ago
  Glad someone else mentioned it. I haven't seen much news from them recently other than a picture-sync application. Looking forward to the next few years and their (anticipated) growth
OnlyRepliesToBS 8y ago
"How can this be accelerated?"
walterbell 8y ago
If you're on Windows, https://www.mailstore.com https://www.mailstore.com has a mature, free version that will backup all your local and cloud mail to a searchable offline archive. Uses standard formats. Can also move your mail between cloud providers.
_salmon 8y ago
The Archive Team has been doing this work for at least 10 years. If you'd like to help archive the internet, they have an easy-to-use tool you just run and let work in the background. https://archiveteam.org/index.php?title=ArchiveTeam_Warrior https://archiveteam.org/index.php?title=ArchiveTeam_Warrior
wolfgke 8y ago
A large problem in archiving is actually the copyright law. Sites like sci-hub or Library Genesis do a great job in archiving scientific papers and books, but are illegal. Also archive.org is sometimes on the brink of legal/illegal, e.g. not all material that can be found on archive.org is really legal; though for lots of the formally illegal archived content, the copyright holders do not care or do not want to cause an outcry.

134 points by adrian_mrd 8y ago

The comment about Archimedes and infinitesimals makes reminds me of something that has been bothering me for a long time: what brilliant ideas are there lurking in the world unknown to almost all of us that could change the course of history if only they were publicised?

trevyn 8y ago
Another take: What brilliant ideas exist publicly today that have not yet been recognized as brilliant? And how might you go about systematically identifying them? And what might your incentives be for doing so?
- AnIdiotOnTheNet 8y ago
  “These days, the problem isn’t how to innovate; it’s how to get society to adopt the good ideas that already exist.” — Douglas Engelbart
bcaa7f3a8bbc 8y ago
I have a similar thought experiment (or Sci-Fi plot?) for a long time. If an advanced alien civilization came to Earth, and helped ancient human civilizations to built and operate a computer system, which can share, store and translate scientific discoveries all across the globe (while they are not allowed to help humans besides operating the system) throughout 5000 years of history - how could it change the human history?
- gsanghera 8y ago
  The ancient humans would then also (eventually) develop patents because someone somewhere would have thought about it even back then - and with worldwide distribution - it would catch on. Effectively, what I'm saying is - when faced with a common greater good that is a game changer for humanity, humanity then has to collectively want the greater good. And humans can be selfish - ancient or not.

devilmoon 8y ago

Slightly related to this article, I recommend this Tedx talk by Cerf himself: https://www.youtube.com/watch?v=GV0A82TCrf0 https://www.youtube.com/watch?v=GV0A82TCrf0

stillkicking 8y ago

In many ways he's understating the problem. At least there used to be file formats to support, and files to lose. But how much of the data you interact with daily is even directly accessible to you? How much of it can you access when you're offline? Take Slack. You can install it locally, but that doesn't matter, because it won't start without an internet connection, and its local storage is completely opaque. Compare this to a mail client, where everything is stored locally, indexed and searchable offline, and can be exported into a universal albeit messy MIME format, as well as imported into other accounts. Of course this is necessary for the business model, the Slack free-plan event horizon wouldn't be effective if it only applied to which new data you can sync down... and if users only discovered this when e.g. moving to a new computer, they would quickly start to wonder why they can't just transfer the data they already have themselves. With Google Drive, your 'docs' are just placeholders pointing to the cloud. You can export them individually to Word or print them to PDF, but it's a manual process, which you will likely only think of when it's too late. For an example of how this can really matter: an acquaintance is embroiled in a legal dispute with an ex-employer. Thanks to Apple's sane implementation of IMAP Mail on iOS, she still has access to all her company communications there to use as evidence. Unlike on her PC, where she was just using webmail the entire time and has nothing. The incentive in the cloud age is to create dysfunctional products that provide an illusion of permanence instead of the reality of tangibility. I expect this is only going to become a bigger problem over time.

knorker 8y ago
> You can export them individually to Word or print them to PDF Actually it's really really easy to do in bulk: https://takeout.google.com/settings/takeout?pli=1 https://takeout.google.com/settings/takeout?pli=1 Not invalidating the rest of what you said.
- dredmorbius 8y ago
  Google's data take-out has proven fairly robust.
- 013a 8y ago
  I think the rest of what they said matters, though. Its not "preservable by default". If you need access to that data in any situation where your connectivity to Google or that account is severed, you're shit out of luck. The best case example is if you don't have internet. The bad case is really that "legal dispute" argument, where you're dealing with a bad actor who has power over you and your Google account. The worst case: Google themselves severs your access.

WalterBright 8y ago

By contrast, before the internet, most everything was routinely lost anyway. Only papers from famous people were kept. Even so, for example, HP's historical archive was all on paper and placed in a single building. Which burned down. The WTC collapse destroyed the unpublished archive of Kennedy pictures.

JdeBP 8y ago
David Granick wasn't famous, yet his works were preserved, by a library. * https://news.ycombinator.com/item?id=16472614 https://news.ycombinator.com/item?id=16472614 * https://news.ycombinator.com/item?id=16363283 https://news.ycombinator.com/item?id=16363283
vertere 8y ago
It is true that a lot more stuff gets kept these days. But we seem to be much more liable to losing important information. As Bret Victor [0] has pointed out, we use largely the same mechanisms for managing the most trivial to the most important information, and most people don't really understand how they work. > a 2013 survey of law- and policy-related publications found that, at the end of six years, nearly fifty per cent of the URLs cited in those publications no longer worked. According to a 2014 study conducted at Harvard Law School, "more than 70% of the URLs within the Harvard Law Review and other journals, and 50% of the URLs within United States Supreme Court opinions, do not link to the originally cited information." [1] I doubt that the US Supreme Court was regularly losing track of cited documents before "information technology" came along. A lot of documents may be recoverable even if the original URLs are broken, but is this really the best we can do to manage information in the information age? [0]: http://worrydream.com/TheWebOfAlexandria/2.html http://worrydream.com/TheWebOfAlexandria/2.html [1]: https://www.newyorker.com/magazine/2015/01/26/cobweb https://www.newyorker.com/magazine/2015/01/26/cobweb
- c12 8y ago
  The fact that urls cited in publications no longer work over time is why I often included their content as apendix's to my essays. That way even if the original source is lost the referenced material isn't. Then again material published in journals and books is greater than material published on the internet, if for nothing other than it can be relied upon to be found in decades to come.

pmlnr 8y ago

http://longbets.org/601/ http://longbets.org/601/ https://adactio.com/journal/11937 https://adactio.com/journal/11937 "The original URL for this prediction (www.longbets.org/601) will no longer be available in eleven years." This has been a known problem for a while, but it's always nice to see more voices crying out about it to the more general public.

bcaa7f3a8bbc 8y ago

Think about it, we still have almost-complete versions of Usenet archives from the 70s to the late-90s, the whole network has been preserved as if it was a time capsule. You can visit https://olduse.net https://olduse.net (maintained by a retired Debian developer), and see the heyday of hackers and early technological adopters as if it was just yesterday, the entire society is archived here. Discussing about the 4-color theorem recently proved, latest version of C compiler, difference between a vacuum tube amplifier and a solid-state amplifier, where GNU Project and Linux kernel was launched, and early online culture and tons of colorful, hilarious, but forgotten and buried memes, and weird phenomena emerged from the collective (un)consensus... Sci-Fi fandom being an integral part of online and hacker culture, millions of lewd story written in alt.sex, "Immediate Death of Usenet Predicted!", "There is no cabal", alt.french.captain.borg.borg.borg, Coffee and Cat warning, The church of Kibology, Anti-spam Movement, creationism vs evolutionism debates at talk.origin, Meow Wars - the first meme war online, all the personal attacks, trolls, flame wars, and "cyber-stalking", etc. Then centralized WWW replaced distributed Usenet, crappy HTML replaced perfect machine-readable data format. Would we have a similar archive for Reddit or Hacker News? Possibly not. So Hacker News, just come and create one! You can make it! Anther unique challenge created by WWW is the inaccessibility of server-side software - exporting and preserving the data is NOT enough, unlike Usenet which you can just load any data. The user-interface and functionality of one website itself is also the collective memory that needs to be preserved - we need replicated software of a website, which has identical user-interface, which has all the functions from the original website: users to click an username and see the posts, karma of this user, etc. I don't think anyone even noticed the existence of this problem. Luckily, major websites online such as Reddit or 4chan, all use FLOSS software which would make the work easier, but still a huge challenge due to the inaccessibility of raw database. Also, to make some contents meaningful in the future, external resources such as hyperlinks to other websites and images should also be preserved, considering this, the chance of creating an authentic and complete archive is even lower. --- But even if we're still using a distributed network where data preservation is still technically possible, and there is no walled garden, it may still be difficult to implement. In the era of Usenet, you often attach your name, address and phone number - there was virtually no threats except for a few trolls - this is why archiving Usenet was possible in the first place. But the Internet is not the Net anymore, now not only humans - almost every piece of equipment involved on the route may be your enemy. The ongoing security and privacy movement is a huge threat of historical records. From my observation, at least of infosec hackers community - After Snowden's revelation, public and open discussion is slowing being transformed into private, closed, encrypted and temporary activity, plus self-hosted platforms like ActivityPub, GNU/Social, Mastodon. This is indeed good from a security and privacy perspective and it is exactly what we need now. But we are also creating a huge gap of knowledge, information and history on the Internet. After my death, none of my self-hosted code, or my blog, or my GNU/Social posts will survive. In conclusion, "collect 'em all" is both an malicious NSA dragnet surveillance, and a glorified act of history preservation. This is where the contradiction lies. I don't know what to do. For WWW, archive.org is a workaround and I think it needs more donation. But for all the other self-hosted things like Mastodon and git server, there is no solution at all.

deleted 8y ago

[deleted]

imhoguy 8y ago

Adobe Flash player (closed SWF format) death takes toll also with uncountable number of human artistic creations, games etc. I wonder if there is some way to preserve that relic of 2000s.

icebraining 8y ago
Seems like there are a few people still working on Shumway, the flash engine written in JavaScript.
- theandrewbailey 8y ago
  I doubt anyone is working on Shumway. Its in the Firefox graveyard. https://bugzilla.mozilla.org/describecomponents.cgi?product=Firefox%20Graveyard https://bugzilla.mozilla.org/describecomponents.cgi?product=...
  - icebraining 8y ago
    From Mozilla yes, but there was some activity by others this year: https://github.com/mozilla/shumway/pull/2442 https://github.com/mozilla/shumway/pull/2442
MaxBarraclough 8y ago
> I wonder if there is some way to preserve that relic of 2000s. Flash is fading from mainstream use on the web (thankfully), but not hard to run Flash today if you want to.
RandallBrown 8y ago
Seems like virtual machines will be able to run flash for a long long time.

deleted 8y ago

[deleted]

AnIdiotOnTheNet 8y ago

I'm probably weird in that my philosophy doesn't jive well with the current societal norm, but I think it is important to come to terms with a simple truth: All things are ultimately lost. Physics tells that even the universe itself will end, even if different models disagree as to precisely how. Things begin, they exist, and then they end. That they are not eternal does not mean that their existence was meaningless. Which isn't to say that preservation is pointless or wrong, but I think it is unhealthy to try and hang on too firmly to the past.

013a 8y ago
I think that's fighting human nature. One of the things that separates us from other creatures is our constant desire to fight entropy. Sentient life itself is the ultimate reversal of entropy; while the universe becomes more complex around us, we reduce complexity in every environment we enter by organizing matter to build more and more efficient systems. Its not crazy to think that, if we live that long, we will figure out a way to escape even the end of the universe (I'm reminded of Asimov's The Last Question). But that progress definitely won't happen if future generations can't build on what we've learned. I actually think living with the idea that everything is temporary is very very dangerous. Nothing is temporary, except you. We need to live with the humility to recognize that our life on this planet is incomprehensibly short, and we need to spend it doing everything we can to give our children the headstart they need to do everything they can to give their children a headstart... into infinity. That's the only way we improve as a species. That's why there is no greater sin than destroying or limiting access to information, and conversely why the internet is literally the most substantial development in the history of mankind.
- AnIdiotOnTheNet 8y ago
  That doesn't separate us from other living things, or even natural processes, in any way. Many things appear to sew order in the universe on the local scale, but the math is pretty clear that this isn't actually fighting entropy at all. The Last Question is a work of fiction. All works of fact seem to support the concept that it is not possible. That isn't to say that we will never discover properties of the universe that open new possibilities, but A) I think eternity is just a different kind of ending anyway, and B) stubbornly grasping to such a concept seems a desperate attempt to deny one's own mortality. I'm not advocating for forgetting things for the sake of it, I'm advocating against the idea that everything must be preserved. Certainly we should try to keep in mind the lessons of our past, but we should also be willing to let go of things that haven't given us good reason to keep them around.

jacquesm 8y ago

Due to the Curse of SaaS soon the web won't be about documents any more but about applications that have no obligation to preserve anything. The only thing you can be sure of is that your advertising profiles will be stored for the next 100 years or so.

013a 8y ago

Frankly, its somewhat hilarious to hear all this coming from Cerf, who works for the worst offenders of data centralization of all time. Think of the data that gets lost on Drive every day due to their product decision around the SaaS model + proprietary document formats, or the data that was lost/hidden on Code, Wave, Plus, their shattered IM platforms, etc.

dahart 8y ago

At first I thought this was saying all information on the internet should be preserved (which is a growing privacy concern), but after reading more carefully I think it’s about having a way to preserve the information that is intended to be public and permanent from the beginning. We don’t have any good ways yet to guarantee future access to public information. I’d never heard of the Digital Object Architecture (DOA). The article itself makes light of the unfortunate acryonym with “History pronounced DOA”. That actually left me confused about what they were talking about for a minute. No idea if it’s a good way to preserve academic papers on the internet, but the business model side of it is a pure open question. That makes me wonder whether it solves anything at all. The problem with information on the internet is that the people who publish eventually lose the interest or the ability to continue paying for storage and access. “Economics/Business Model While the Handle System has been used for many years in publishing and library systems, generalizing to other applications, e.g., Internet of Things, will likely generate economic concerns related to the business model of the system, especially at the Global Handle Registry. Will organizations be charged for each identifier? Will organizations that acquire a prefix be able to create unlimited sub-prefixes or will they be charged for each sub-prefix? How will these policies be developed? How will the money flow? What will be the impact on developing countries or small businesses?” https://www.internetsociety.org/resources/doc/2016/overview-of-the-digital-object-architecture-doa/ https://www.internetsociety.org/resources/doc/2016/overview-...

unquietcode 8y ago

It's worth noting that in complex systems, forgetting can be just as important as remembering. The ability to evolve and change is in part predicated on the ability to selectively forget some elements that are no longer helpful. As another commenter pointed out, some aspects of our society's cultural memory are, probably, best left in the past, if preserved at all.

notabee 8y ago
Knowing what should be forgotten requires either foreknowledge of the future or relies on inaccurate predictions based on current assumptions. Thus the less information of uncertain usefulness that is retained, the more that those assumptions dictate what will be considered useful in the future without having to rediscover things entirely. That limits adaptability. Just try to imagine how many things were invented in the past that weren't seen as useful at the time and had to be rediscovered later. How many ideas were lost in the Dark Ages, for instance, because they offended religious sensibility at the time? If there is capacity to retain information in an organized way without undue cost, it should be retained as a hedge against future uncertainty. No single generation of humans should be trusted to make such decisions without being unduly influenced by the biases of their time.

sulam 8y ago

We have been forgetting things for thousands of years. Last I looked, we still don’t know how to make Damascus steel or Roman concrete. People will reinvent the things they need and maybe never even know that it’s a reinvention. I reinvented a couple fundamental graphics techniques because I needed them and didn’t study computer graphics in school. I’m sure this happens all the time. I’m all for making it easier to recover data and generally make it easier to store stuff. And I believe you have a right to your data (yay GDPR). But this isn’t a catastrophe.

arbitrage 8y ago
> we still don’t know how to make Damascus steel or Roman concrete That's an oft-repeated but ultimately meaningless statement. We know how to make things better than Damascus steel or Roman concrete. Our processes have exceeded the ancients. The quality of Damascus steel is likely overrated, since it wasn't quite as absolute shit as what was being regularly traded at the time. Same with concrete. We might not have exquisite written recipes and procedures for these materials, but they have been reverse-engineered, and it turns out, they weren't that great compared to modern chemical and material engineered products. We don't need to 'rediscover' Damascus steel because it is obselete. A romantic idea, and poetic in how it was 'lost' to time, but consider that it was 'lost' to time, similar to Japanese steel-working, because nobody wanted to buy it anymore. It became economically and culturally irrelevant.
- sulam 8y ago
  Exactly. We forgot how to do a thing and invented better. The “lost” knowledge isn’t really worth much to us today. And that will be true of essentially everything we forget in the future as well — the forgetting is the sign that it wasn’t needed.

paidleaf 8y ago

Losing memory via attrition is certainly a concern. But I think the bigger concern is losing memory via censorship and copyright. For most people, the internet is now google search and a couple of social media companies. Google has been curating search results for a while now. We no longer see what is out there in the internet, but a narrow google approved view of the internet. For example, 10 years ago, google showed us the entire elephant ( more or less ), today, we only get to see the tail or the trunk or whichever part of the elephant google decides to frame. Now all of the social media companies are now curating. And with the loss of neutrality, ISPs will undoubtedly curate. And then there is the copyright issues where archiving is getting more difficult. Kids today don't know what a great tool google cache was because it's gone. And archiving sites are being attack from news, media, politicians, etc. In other words, the internet dark age isn't going to be a result of formats getting old ( though that is an issue ). It's going to be a result of us only being allowed to see through the frame that a handful of companies deem appropriate. The dark ages didn't happen because formats got old. The dark ages happened because the powerful decided that we should view the world within a certain narrow religious frame and censored everything else.

rezeroed 8y ago
In the olden days, everyone was slapping together their own websites, rather than using the same old templated wordpress, hugo, etc, so there was far more character, and not everything was a blog. And so people surfed the web. These days the web is more purpose oriented, people looking for something, the "surfing" all within walls, wave pools.
- floren 8y ago
  The days of the web ring, which often as not was broken or took you through dozens of content-free sites but could occasionally drop you into something fantastic. Browsing Internet Directories. Just plain finding some guy's massive meticulously-maintained HTML-only site full of fascinating articles, bookmarking it, slowly working your way through it over the course of your evening browsing sessions.
- TeMPOraL 8y ago

ravenstine 8y ago

People need to keep their own personal archives. While there are great projects like archive.org, it's becoming more clear that it's not enough. There's not enough resources for an organization(that isn't spying on everyone) to archive YouTube, or Slack history, etc., so it's up to you to back up that data if it's important to you. Culture can be easily lost to the void if we don't do this. As I've mentioned in another HN thread, I've been using youtube-dl to back up YouTube videos that are important to me since it's increasingly obvious that YouTube is merely a profit machine that would delete all of its user's videos in one fell swoop if they made an extra buck off it. I've also begun to back up some blogs that have vast mounts of great content, but I've found that HTTrack is... well, terrible. That's not to say it doesn't technically do its job, but it's way too aggressive. I wan't something that's easier to configure and can do a better job of taking "snapshots" of blog entries while blocking ads, javascripts, being able to resume properly if my power goes out, etc.

abathur 8y ago
There may be more modern projects, but there's also "Lots of Copies Keep Stuff Safe" at https://www.lockss.org https://www.lockss.org When you say personal archives, I wonder a little about what we'd lose to filtering by people who know they're building an archive for posterity, versus people in the future rediscovering things that were safely forgotten for some long span of time. At the delta of personal archives and projects like this, I imagine something a little like SETI or folding for home archival (is this just describing IPFS?), but I think one of the challenges is how we preserve the right kind of data. I did some spring-data-cleaning this year while building a new system, and one of my tasks was addressing duplicate files. This forced me to confront hundreds of gigabytes of VM images, snapshots, and massive piles of duplicate files from various installations of Ruby and Python. But perhaps I underestimate the value of even the duplicate VM images or language installs. Perhaps, centuries hence, researchers will hopefully sift yottabyte of data looking for all of the dependencies to successfully resurrect applications they need to experience in order to understand references to them in well-stewarded tweets, posts, or articles.
txsh 8y ago
I noticed old articles I had bookmarked were disappearing so I’ve started using a self-hosted Bookstack to save a copy of everything interesting I read. I enter reader mode and copy eveeything on the page, and then paste it into the WYSIWYG editor. Everything is formatted nicely and it even creates a menu using the head tags of the sections. The only problem is, while images copy correctly, they’re still hosted on the original server so I have to replace them with an uploaded copy. Only takes a couple of minutes though. I could probably automate this with a script.

rmorey 8y ago

In case anyone hasn't already heard if it, IPFS(https://ipfs.io https://ipfs.io) is a really cool project that aims to help solve this problem

mbowcutt 8y ago
Glad someone else mentioned it. I haven't seen much news from them recently other than a picture-sync application. Looking forward to the next few years and their (anticipated) growth

OnlyRepliesToBS 8y ago

"How can this be accelerated?"

walterbell 8y ago

If you're on Windows, https://www.mailstore.com https://www.mailstore.com has a mature, free version that will backup all your local and cloud mail to a searchable offline archive. Uses standard formats. Can also move your mail between cloud providers.

_salmon 8y ago

The Archive Team has been doing this work for at least 10 years. If you'd like to help archive the internet, they have an easy-to-use tool you just run and let work in the background. https://archiveteam.org/index.php?title=ArchiveTeam_Warrior https://archiveteam.org/index.php?title=ArchiveTeam_Warrior

wolfgke 8y ago

A large problem in archiving is actually the copyright law. Sites like sci-hub or Library Genesis do a great job in archiving scientific papers and books, but are illegal. Also archive.org is sometimes on the brink of legal/illegal, e.g. not all material that can be found on archive.org is really legal; though for lots of the formally illegal archived content, the copyright holders do not care or do not want to cause an outcry.