19 ms·

Link rot and content drift are endemic to the web

• budibase 5y ago
  I think it's important to support tools that are driven with people in mind, rather than money. I like where Brave is going: https://news.ycombinator.com/item?id=27593360 https://news.ycombinator.com/item?id=27593360
• chovybizzass 5y ago
  yes. i am down to HN and Reddit. Google calendar and telegram. I don't even know how to find cool stuff online anymore. Google SERPs are all business driven now unless you're research news.
  • ct0 5y ago
    All the cool stuff has gone back to IRL. Once people stopped making potato gun websites, the internet really stopped blossoming into an amazingly vibrant space. I would recommend hackaday.com because it hasn't changed in quite some time.
    • agent008t 5y ago
      I used to go online to escape the boring, unimaginative people and the world they create. Now I have to disconnect to escape them.
      • TheFreim 5y ago
        You have to know where to look online, very rare these days.
    • jabl 5y ago
      FWIW, Siemens recently bought hackaday (or well, Siemens bought a company called Supplyframe which was the owner of Hackaday), so lets see how long that will last..
    • Avamander 5y ago
      > Once people stopped making potato gun websites, the internet really stopped blossoming into an amazingly vibrant space. Have they or have you stopped looking? I'd say the former.
    • krapp 5y ago
      >Once people stopped making potato gun websites, the internet really stopped blossoming into an amazingly vibrant space. Entire new genres of creative output - music, fiction, fandom, films, cosplay, hobbyist and enthusiast communities have been spawned by the modern web. It's never been more vibrant. I'll never understand why people on Hacker News seem to believe the internet stopped evolving as an expressive space just because services replaced the need to design websites by hand. That's like believing literature ended once scribes were replaced by the printing press.
      • ulber 5y ago
        Yeah, the internet is bigger now - all of the weird fringe stuff from enthusiasts is still there and there's even more of it. One thing that has happened is that the mainstream is there too now, so if you don't venture outside that its easier to not see the more interesting stuff. In the old days, when just having a "homepage" placed you a bit outside the norm, stumbling on something non-mainstream was more of a given.
        • krapp 5y ago
          And the mainstream has become more interesting as a result. It's not uncommon to watch anime now, D&D and video games are no longer niche, people's interests are becoming more diverse as media is no longer being gatekept by communities, geography or publishers, and everything becomes available to everyone. People might see that as a negative, the Eternal September effect eating their favorite thing, but I see it as a positive.
  • AnIdiotOnTheNet 5y ago
    I wish there was something like Reddit that could be organized by topic, but had the simplicity of HN's design instead of the monstrocity Reddit has become. My guess is it would still succumb to the Reddit Hive Mind effect without a reasonably benevolent moderation team though. For all the times I've said that HN basically does the same thing, I have to admit that it is much better about keeping it in check.
    • mrunseen 5y ago
      I don’t know about userbase but old.reddit with disabled subreddit CSS is quite usable for me.
    • zanderwohl 5y ago
      Reddit has been pushing to be like other social media sites now. They've added profile pictures and avatars, not to mention they're pushing video content like nothing else. They've got livestreaming and try to saturate your front page with as much video as possible. They've also changed the way their app handles video links to be more like TikTok or YouTube. It used to be a lot like HN - discussions around links to articles. I wish there was a community with the feel of HN with the wide net of Reddit. It feels like all social media is converging; Snapchat, Instagram, Facebook, Reddit, TikTok, Youtube, all an endless stream of ai-curated short videos that you can swipe through over and over.
      • Avamander 5y ago
        > It feels like all social media is converging; Snapchat, Instagram, Facebook, Reddit, TikTok, Youtube, all an endless stream of ai-curated short videos that you can swipe through over and over. It's likely that AI curation is the future because no humans can shift through the vast amounts of data and content being made. Things that can't keep up without curation or with just human curation have died or will die. Good AI curation can bring you the exact content you're looking for, can but not will. I've seen it work times and times again, but I've also noticed that you have to be aware of the flaws of the tool to be able to use them or it gets really bad really quick. You can't let the AI take control, if it derails to content you don't like you must know what it uses as a quality signal and give it a thumbs down, if it is intentionally derailed, you must stop using the platform. TikTok recently released an update to their algorithm, it ruined my FYP and replaced my content with inane videos made by people nearby - hyperlocal garbage. The feedback mechanisms given no longer work, before that update they did. I do think that even people being nostalgic here about the "old internet" should try and learn how to turn AI curation for their own advantage instead of just being sad and nostalgic.
  • throwaway_egbs 5y ago
    The article is about link rot, not cultural rot.
    • shadowgovt 5y ago
      And it seems to overlook a pretty straightforward question: in the era of the search engine, how much of an issue is link rot? I've hit bad links before. Four out of five times, I can do a general search for the title of the document that should have been at the link or the quoted excerpt that the document I'm reading pulled from the link, and I get a clone of the document posted somewhere else.
  • mycall 5y ago
    Google still works, you just need to be very specific on your search criteria (e.g. site:). I do agree on the premise that good quality data, art and content has been lost to the winds of time.
    • rchaud 5y ago
      Needing to use 'site:______.com' in the search query defeats the purpose of an internet search engine. I agree that it is necessary today, due to the sheer amount of useless sites that pop up on page 1 of the search. I wish Reddit invested more into making their internal site search better. If people did their searches directly on websites, Google would have an incentive to improve search results so it wasn't always the same 10-20 websites topping the list for nearly every query.
  • Avamander 5y ago
    You either have to visit aggregating sites that have likeminded people (HN/Reddit/Obscure FB groups/Group chats/Forums) or know exactly what you're looking for.
• djoldman 5y ago
  > People tend to overlook the decay of the modern web, when in fact these numbers are extraordinary—they represent a comprehensive breakdown in the chain of custody for facts. This is a particularly good quote to sum up the article. The internet is not a repository of facts, it is a repository of facts, spam, junk, and things. Moreover, it is not the only repository of these. Link rot happens. Content is subject to the will of the publisher to spend the time and/or money to continue to host it. Depending on links to work eternally is a mistake. The problem is not the link rot, it is the bad assumption.
  • PaulHoule 5y ago
    It's depressing. I know somebody who started a business that was successful for a while and then failed. Spammers got control of the domain and now it is full of ads for a dangerous diet drug. What makes my blood boil is that it impugns the integrity of the founder who is a decent person who has nothing to do with that scam.
    • JKCalhoun 5y ago
      Domains, if not outright sold by the owner, should die then.
      • Wowfunhappy 5y ago
        And never be possible to register ever again? I feel like most easy-to-type domains would have been permanently expended in the early days of the web.
        • PaulHoule 5y ago
          You wish there was some way you could make the links go away...
      • junon 5y ago
        Or ICANN can create policies about domain squatting like GP described.
• CountDrewku 5y ago
  This is why there are sites like WayBackMachine. I suggest people keep donating to them as well if they want to preserve internet history.
• shoto_io 5y ago
  TL;DR The second law of thermodynamics applies to the Internet as well.
• JKCalhoun 5y ago
  I expected the article to deal more with the rotting landscape of the internet, the rotting of our choices of content, the poor selection of links on the 1st page of any search engine.... I am less concerned with the fact that a link breaks than I am with what it says about the content that was there, is no longer there.
  • rchaud 5y ago
    Links are the backbone of the internet. Archive.org is a huge asset, but it relies on individuals being prescient enough to archive pages that might be lost to time. That's not scalable. Plenty of people will visit Wayback Machine to pull up an old page that's gone to the big 404 in the sky, but they won't actively submit links to archive themselves. The bad design and low quality content is a symptom of the Internet's broken underlying economics. That's a human problem, not a tech problem.
• uniqueuid 5y ago
  By the way, the technical side of this is very interesting. If you look at the tools mentioned (the wayback machine, but also perma.cc and other archival solutions), almost all of them rely on a single semi-modern tech stack that produces WARCs (web archives - ISO - ISO 28500:2017 https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/ https://iipc.github.io/warc-specifications/specifications/wa...). The main crawler still seems to be heritrix3 (https://github.com/internetarchive/heritrix3 https://github.com/internetarchive/heritrix3), but there's a great little ecosystem with tools such as webrecorder and warcprox. Still, I've read through the code of these tools and am feeling that they are failing in the face of the modern web with single page apps, mobile phone apps and walled gardens. Even newer iterations with browser automation are getting increasingly throttled and blocked and excluded from walled gardens. Perhaps the time has come for a coordinated, decentralized but omnipresent approach to archival.
  • black_puppydog 5y ago
    > increasingly throttled and blocked and excluded from walled garden I keep thinking back to Jacob Applebaum's stance of "facebook and the other walled gardens are the real dark web."
  • shadowgovt 5y ago
    Honestly, it would be a better use of surplus resources than crypto mining. If only there are a way to algorithmically tie a proof of work for a new cryptocurrency to archival of the internet in a way that wouldn't be easily gamed (by people archiving easy to access content or highly redundant archival of trivia).
    • meowkit 5y ago
      https://spec.filecoin.io/algorithms/pos/ https://spec.filecoin.io/algorithms/pos/
  • PaulHoule 5y ago
    WARC can record and replay single-page apps, but it struggles with knowing where a "page" begins and ends. There was a time when I was furious with the web going to hell and I investigated the possibility of "web without browsers" that started with making a WARC capture of page and putting pages through extensive filtering and classification before the user sees anything. With interactive capturing you can push a button to indicate that a page is done "loading" but with automated capturing you can't really know that the page is done or that you got a good capture. That ended the project right there.
• have_faith 5y ago
  Knowing that it decays is what prompts us to try and save the bits worth saving. I don't sit in the camp that everything digital must be preserved and that it's a disaster if it isn't. I try not to fight entropy in it's many manifestations. It's a shame when content disappears but I think it's also healthy to just accept it. We tend to only frame information disappearing in a negative light because we can always imagine a scenario where that information could have been valuable to someone, and that is a valid concern, I just don't think it's helpful to view it as the internet going into some downward rotting spiral and therefore every single 0 and 1 must be preserved. The major problems of the internet seem to be almost entirely cultural currently.
  • Robotbeat 5y ago
    Ironically, deleting stuff on the web is technically REDUCING its entropy.
• FridayoLeary 5y ago
  But at least it's apparent that an end must be in sight for internet as we know it. That there's still other forms of internet waiting to be discovered.
  • dredmorbius 5y ago
    Such as / directions / goals to seek / pitfalls to avoid?
• okareaman 5y ago
  Buddhists chuckle at the notion of permanence and go back to constructing sand mandalas
  • datameta 5y ago
    I know your comment is probably tongue-in-cheek but I must say (without disagreeing, regardless) that perhaps modern day digital infrastructure can transcend the considerations early buddhists may have had for the natural world and then-contemporary human-built structures.
    • okareaman 5y ago
      I used to be a data hoarder but I learned to let it go. I save the important stuff, just like the Buddhist monks do. How much of the internet is really worth saving? What will Geocities mean to anyone 50 years from now? The internet is a dynamic process evolving in real time. No man ever steps in the same river twice, for it's not the same river and he's not the same man Heraclitus
      • datameta 5y ago
        I am a fan of that quote and I now personally eschew clutter. However I'm referring to the examples in the article such as the supreme court justice referencing links that no longer existed. Paraphrasing the article, >75% of links from the 90s are defunct. Sure Geocities may not have value to many, but an astonishing number of links in court rulings and law documents are leading to dead ends. I can see how this could lead to shaky ground upon which it would be more difficult to defend certain internet freedoms.
        • okareaman 5y ago
          Google's recent invention of text links should help this: https://en.wikipedia.org/wiki/Filler_text#:~:text=%22Now%20is%20the%20time%20for,21%20(1918) https://en.wikipedia.org/wiki/Filler_text#:~:text=%22Now%20i.... # signifies an anchor :~:text= signifies a text link %22Now%20is%20the%20time%20for,21%20(1918). says show me the text between "Now is the time for" and "21 (1918)."
• kylestlb 5y ago
  It's an Atlantic article so it's long, and several of the comments here show that people aren't actually reading the whole thing... but I did and it's worth the time. It's not only about links being dead, it's about the lack of transparency & audit when content is changed via takedown requests, it's about dead links showing up in decades-old supreme court decisions, it's about private industry's lack of incentive for wanting to improve any of these issues, etc.
  • ballenf 5y ago
    Maybe the reason it feels impossible to stem the tide of link rot is that it's takes tremendous energy to constantly increase the entropy of a system. And that energy has opportunity cost that no one really wants to talk about. The article has the unstated assumption that eternal preservation of all writing ever is a net benefit. I think it's worth having a discussion on that point.
    • kylestlb 5y ago
      I disagree that the unstated assumption is eternal preservation of all writing. IMO the author is clearly focusing on official or semi-official published information, not necessarily what you and I write on places like HN.
      • bentcorner 5y ago
        And there's the irony that's pointed out in the article as well - official documentation on government websites may not last past an elected official's term, yet blithe comments on a social media site that are later regretted may last forever.
    • Sr_developer 5y ago
      > Maybe the reason it feels impossible to stem the tide of link rot is that it's takes tremendous energy to constantly increase the entropy of a system. I suppose you meant "decrease"
      • 5y ago
• MomoXenosaga 5y ago
  Visiting an old forum and all the pictures will be gone because, surprise, free image hosting doesn't make economic sense.
  • Robotbeat 5y ago
    The web forum that I frequent most, https://forum.nasaspaceflight.com https://forum.nasaspaceflight.com , has a policy of not allowing embedded images but requiring them to be attached on each post. The forum has been active for well over a decade now (site was founded in 2004) and has a thriving community that continues to grow. It is the community that keeps the forum alive instead of just one random company (although it technically is a company). This helps prevent link rot. Forums seem anachronistic in 2021, but they have massive benefits versus the gigantic platforms like Facebook or Reddit or Twitter, and the quality of the discourse and analysis is far higher. There are a few ads, but they’re very unobtrusive. (Also, Twitter posts are often linked, but usually the text is copied for archival purposes.) My experience with Wikipedia, forums such as those, and Wayback Machine and arxiv.org make me think that people will do a lot of stuff basically for free and that by building communities, you don’t need extremely clever trustless incentive systems like Blockchain or major paywalls (although granted, the forum does have something like a paywall for unverified pre-public info) or massive platforms with multi billion dollar companies in order to disseminate information, analysis, news, etc. Best practices of web forums from the 2000s (active moderation, a sense of common purpose, expectations of non-toxicness, etc), are a really good solution.
    • deleted 5y ago
      [deleted]
    • EricE 5y ago
      Forums are not anachronistic - they are the future again. The only way to guard against the censorship of the current social media platforms is to not rely on them. What always baffled me is why forums didn't embrace technologies we had in the BBS days. Offline readers were the greatest thing since sliced bread - you could use whatever interface to a message board you liked, whatever editor you liked, etc. It was a lot easier to quickly scan through literally thousands of messages with a native, local client that relying on the constant ping pong between your client and a remote server. Decentralized aggregation is what we really need. A combination of RSS and DNS. Ways to foster creation and discovery of hand curated lists like the original Yahoo - but thousands of them. No reliance on Google, Facebook, Twitter, etc. It's a nice dream anyway...
• platz 5y ago
  Fungi serve an important role in nature.
• ericra 5y ago
  For those hitting the "monthly article limit" paywall, this can always be bypassed on The Atlantic by opening the link in a private window.
• wintermutestwin 5y ago
  I am increasingly worried about the valuable content on YouTube. There are so many old live concerts, useful how-to videos and other cultural treasures amidst all the junk. I suspect that one day, they will make their ads unblockable by embedding them in the video files. I sure hope that some people are downloading the valuable stuff and stashing it away to load onto YouTube's successor. <and please skip the tired argument that I should just pay a subscription fee to avoid their ad crap - we are already paying them with our data. My data is worth far more to me than the value they provide for it. Plus - I won't give money to a company who forced this Faustian bargain on me.>
  • robotnikman 5y ago
    I set up a server specifically for downloading Youtube videos from all my playlists on a daily basis just for this reason. At some point I got fed up at seeing all the missing videos on my playlists (and not even knowing what was removed)
    • floren 5y ago
      What do you use for automation? My first instinct is cron jobs running youtube-dl scripts against a set of playlist URLs, but I'd be interested to hear more.
      • robotnikman 5y ago
        Yep, you basically just described the setup, nothing super fancy :) Cron job which runs a batch script at midnight which feeds youtube-dl some playlists URLS's which it downloads to a HDD. I also have nextcloud running which has access to the directory the videos are saved to, so I can easily share them if I want to.
  • majormajor 5y ago
    How many how to videos, memes, concerts, etc, are really that important? In fifty years how many people will care? How many people should care because it would mean ignoring the huge volume of newer stuff? A hundred years? Two hundred? I haven't even read or seen many of the existing cultural artifacts we have from past decades and centuries, what would I do if orders of magnitudes more of them had been preserved? In fact, I'm incredibly greatful that I grew up before all the random shit that I threw out there as part of my youth was subjected to obsessive cataloging and archival efforts.
• bullen 5y ago
  The article does not scroll on an older Chrome... the internet is indeed rotten.
  • bencollier49 5y ago
    Or on Firefox.
    • justshowpost 5y ago
      Two reasons more to avoid Atlantic
• developeron29 5y ago
  "The benefits of internet far outrides its shortcomings" Stop being pessimistic
• makomk 5y ago
  It's amazing how little some trusted institutions care about this. For example, the BBC has been bragging about how many people rely on their coverage of the pandemic, but have an obnoxious habit of repeatedly overwriting old articles with new ones on similar topics and not keeping the old versions available. The history of a once-in-a-century pandemic with huge local and global impacts is literally being overwritten day by day. Sometimes this helps them whitewash their screw-ups which have lead to widespread false beliefs. For example, after the UK government targetted and hit 100,000 Covid-19 tests in a day, the BBC ran an article falsely claiming Germany had achieved this a month earlier and linked it prominently on their news front page for about a month. A large proportion of the population probably saw this and now falsely believe it, it got brought up all the time as part of the narrative that the government's big "world-leading" achievements were just playing catch up badly, but it was memory-holed from the article in a rewrite and they used that as an excuse for not publishing any correction - so unless historians dig deep in third-party archives, they'd never understand where that belief came from. (Apparently a previous version of the article also wrongly claimed France was carrying out more Covid-19 tests due to mistaking their weekly numbers for daily one, according to a correction which disappeared from the article after a few days and only exists in the Internet Archive now. I haven't been able to find the original version of that claim.)
  • tda 5y ago
    On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighting the most historically important articles.
    • throwawayboise 5y ago
      I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.
      • nitrogen 5y ago
        But the wayback machine is pretty klunky and slow, honestly. This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot. They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.
• bluedays 5y ago
  Project Xanadu was an attempt to fix this, but unfortunately it suffered from the creator wanting to capitalize on it's success. Which is, I think, why it ultimately failed.
• canadianwriter 5y ago
  I actually implemented a rule for my website: anytime I write anything and cite a link, I always also include the internet archive url as well just in case. If it's not been archived yet I submit it to be. as an example: "You don't have to trust me on this one, here's an article with [a bunch of data] | [*Archive link in case of link rot]" from: https://kolemcrae.com/notebook/virtue.html https://kolemcrae.com/notebook/virtue.html It's not perfect, but it helps reduce some of the issue. Other than that solutions are incredibly hard to come by - you need institutions to preserve urls - through tech changes and the like, when they have very little incentive to do so. Eg. making sure they implement a redirect from the http to https sounds simple enough, but not everyone did it. Also if they switch CMSs and the like.
  • a1369209993 5y ago
    > a rule for my website Note that you should also have a rule to save the link content locally, to avoid single-point-of-failure problems in the unlikely-but-catastrophic case that archive.org itself goes down. (Cf the attempts to attack them over their National Emergency Library programme last year.)
• ladyattis 5y ago
  I feel that the Internet as an archive isn't really feasible. At best, it can augment existing archival efforts such as public libraries. The fact people keep pushing off to webhosting what should be put into a library is a grave misunderstanding of the use cases for the Internet.
• debt 5y ago
  I always thought something like Ethereum could solve this type of thing; that is, if the content itself lived inside the blockchain. Obviously for larger formats that wouldn't work, but for many text based or lower resolution image formats, it wouldn't be too much overhead to just inject it all into the blockchain.
• tgbugs 5y ago
  The whole architecture of the internet is inside out. People have become numb to the insanity of encountering null pointers multiple times per day. This is understandable since the inside out structure is what allowed the web to grow quickly, but it will also be what ultimately dooms it as a real lasting store for knowledge. The problem is that the foundations are shifting sands, and we need something that has significantly more integrity at the bottom layer, we can't just bolt URNs on as an afterthought. Some organizations are able to maintain persistent data over time, but it is in spite of the technology, not because of it. I will also note that a world where it is possible to delete things is a world where individuals can be made to have written anything in the past. On the internet, at a certain point the past can be fabricated from whole cloth. edit: and ironically, the issue is that this is because the internet wasn't actually academic enough in its original design.
  • ganzuul 5y ago
    We should have kept developing Usenet. Handing control over to web browser providers was a mistake.
    • dredmorbius 5y ago
      Usenet died because reasons. It had met its effective maximum scale by the early-to-mid 1990s, at a millionfold less use than today's largest Web platforms see.
• jessehattabaugh 5y ago
  I think the publishing media that predate the web all had the same problem; posters get torn down, newspapers burn, even stone carvings weather. Hardly a new, or possibly even unnecessary phenomenon.
  • ajsnigrutin 5y ago
    The problem is, that previously, if you wrote a book about a poster, you put the photo of the poster directly in your book, and the reader could see the poster there. Now you write an article about a poster, and instead of a photo, there is an embedded instagram photo from the artist, who then removed their account, and the content is missing. Or a hotlink to the authors website, and it's missing there too. Yes, if the book was destroyed too, all info about the poster was lost, but atleast the book didn't say "go to the corner of X and Y street, and hope it wasn't removed or destroyed by the weather".
    • dredmorbius 5y ago
      Note that a chief impediment here is copyright rather than the ability to reference an extant work. A photo of an inscription is not the same as the inscription itself --- detail is lost in any translation. For digital works, it is possible to faithfully reconstruct an original with full fidelity (if necessary, embed an emulated environment of the host, server, or network originally provisioning the work). But copyright claims make this a legal suicide maneuver, at least for any entity capable of being sued, or being sued effectively. Note that numerous previous archivists have in fact been pirates or copyists, sometines under pre- or non-copyright regimes, but very often in direct rejection of copyright.
• Seattle3503 5y ago
  This reminded me of a story that hit the front page a few years ago. Even if content sticks around and isn't modified, Google will eventually forget it and you won't be able to find it without a bookmark. https://news.ycombinator.com/item?id=16153840 https://news.ycombinator.com/item?id=16153840
• neonate 5y ago
  https://archive.is/Trpkg https://archive.is/Trpkg
• dang 5y ago
  I've changed the baity title to a representative phrase from the article body [1], but it is maybe a bit too narrow now, relative to what the article is really about. Suggestions for a better title are welcome. Sometimes we use the HTML doc title but "The Rotting Internet Is a Collective Hallucination" is worse! [1] That's the best way to get a better title. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=representative%20article%20by%3Adang&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
• Zamicol 5y ago
  Hashes are the answer. How they get implemented in solving this problem is the question
  • dredmorbius 5y ago
    What specifically do you hash? How do you hash content which is programmatically determined and changes on every page load? How do you account for the same work in multiple versions, translations, or updates, strictly using hashes? (Note that a chained hash, e.g., a git history, is not a strict use of hashes, though it most definitely does use hashes.)
    • Zamicol 5y ago
      >programmatically determined and changes Ethereum's Solidity is a great real world example. The instructions are compiled into a binary, and that binary can be addressed via its cryptographic digest. Changes in state are easily represented as instructions. >multiple versions, translations, or updates, strictly using hashes? The same way a git repo does it now: Good software design. >is not a strict use of hashes Huh? I'm not sure what your trying to convey. Are you referring to the fact that git uses diffs between commits? At any commit, a repo may be re-hashed and the current state completely represented by a digest. Of all things to worry about being represented as a digest, software seems to be one of the smallest concerns.
      • dredmorbius 5y ago
        Git chain hashes. Merkle trees are useful. They include hashes. They are not simply hashes. Content-addressable storage robust against variations is another approach.
        • Zamicol 5y ago
          I'm at a loss for your criticism. Didn't the original post say, "How they [hashes] get implemented in solving this problem is the question"? Merkle trees implement cryptographic hashes. >Git chain hashes. Git chains hashes? Yes, of course, along with diffs. How the permanent web will implement digests in various applications is the question. Whatever the answer is for link rot, the guaranteed solution will implement and depend upon cryptographic hashing algorithms. There are already many examples of this class of problem being solved by utilizing hashes.
• phendrenad2 5y ago
  I think out obsession with copyright and attribution of ideas isn't helping. Many times you'll see people reference a page or PDF, which long ago became a broken link. Not one person bothers to paraphrase or copy relevant sections from it. And the wayback machine can't cover everything.
• FartyMcFarter 5y ago
  Donate money to archive.org! Some of the FAANG companies match donations to it too.
• RGamma 5y ago
  This is why I'm building/curating my personal archive with stuff that I think may be worthwhile saving (not only for myself necessarily). Perhaps there will be many personal archives like mine that one day can be shared in a similar vein to copy parties. We will need to treat the information we find online with its impermanence in mind (as authors, making things easy to copy, and consumers, copying stuff). Perhaps it is this mindset that, when sufficiently prevalent, could make the internet more like a library again; weed out the garbage und curate the nuggets. Btw I think archive.org is doing God's work but I don't believe any amount of coding and crawling will be able to save everything (nor should it). It can capture some raw data for (future AI?) historians to sift through though.
• Justin_K 5y ago
  I would simplify it as SEO rot has ruined the internet.
• abeppu 5y ago
  I feel like it's been quite a while since I saw anyone talking about IPFS, but I used to hear about it not infrequently. I don't know if this is because it was too nerdy, or too misaligned with the incentives of most organizations (it's scary for some to not be able to unpublish), or because it comes with some privacy sacrifices, or perhaps it just matters less when hardly any page is actually static any longer. But, for guarding against "published", supposedly static material disappearing, or changing silently, and for removing a short list of organizations from being responsible for preserving content, IPFS or something like it seems well-suited. Anyone who cares to preserve something can. Any change is noticeable.
  • jhunter1016 5y ago
    Yep. I find it odd that the article doesn’t even mention IPFS. Brewster Khale is a big proponent of IPFS, so that would have been a perfect opportunity to bring it up. IPFS has its flaws, but content addressing and immutability are powerful.
• samatman 5y ago
  This is why I think Twitter made a mistake in using the normal suspension mechanism to ban @realDonaldTrump. Not passing judgement on the decision to take away his posting privileges. But by suspending POTUS, everything he posted during his term in office is just... gone. Every hot link to anything he said, on any website, is broken. This is an enormous loss to any historian of the era. He was using Twitter as his main microphone to speak to the world, and all that content is, while not lost lost, thoroughly and permanently scrambled. It would have been better to just lock him out of the account, publish a statement that the @realDonaldTrump account is now permanently archived, and that any new account he tried to open will be suspended.
• fridif 5y ago
  Here's an idea, try to make things that need to survive into static downloadable content. "But I'm the consumer of a service!" --Ask the service provider to open source their work.
• overgard 5y ago
  I doubt people will ever care about this enough for it to have momentum, but there are well known technological solutions: content addressable file storage. If you do that the url is always tied to the file content itself. Of course this requires documents to actually be documents. So I don't think it works for any modern business model.
  • dredmorbius 5y ago
    Do you have any suggested reading on content-addressable file storage?
• fleddr 5y ago
  It's not a technology problem, it's an incentive problem. Had the web somehow been centralized (I have no idea what that would even look like), content still would not be archived, it would be constantly changed, and subject to censorship. Just like in a decentralized web, perhaps even more so. Archiving costs lots of money (and costs keep growing if you only add and never take away), can be highly challenging (in the case of web apps or complex dependencies), whilst providing zero immediate reward for the organization carrying this heavy load. Not only is there no incentive, many couldn't even afford to if they wanted to. And it gets worse still. Digital archiving means paying forever. Imagine paying a 100 years of electricity, hardware replacements, migrations. The entity (business, person) is long gone before that. As a ridiculous example of this: Facebook has several very large idle content data centers. Mega scale buildings full of servers storing photos of Facebook users they haven't accessed in years, and likely never will again. Yet should a user do this, they expect the photo to still be there. That's why I believe the problem should be addressed with more pragmatism. Focus on things of unquestionable long term value, and think of a good solution for this smaller scope.
• bobsmooth 5y ago
  For anyone that wants to help with this, check out the Archive Team Warrior project. You can donate bandwidth and some CPU cycles to archiving different parts of the web. There's a VM image you can download that makes it really easy. https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior You can choose to help archive reddit, pastebin, URL shorteners and other ephemeral parts of the internet https://wiki.archiveteam.org/index.php/Warrior_projects https://wiki.archiveteam.org/index.php/Warrior_projects I've also taken to updating the citations in Wikipedia articles with archive links.
• 8bitsrule 5y ago
  Libraries (like Internet Archive for one) seem like the logical 'permalink' curators of information collections. I'm thinking, for example, of a large regional history site created by a a 501(c)(3) corporation. [https://www.historylink.org/ https://www.historylink.org/] Such efforts deserve a guaranteed permanent home with permanent funding. Another example: hundreds of 'Old-time-radio' programs and early TV shows have only emerged and survived because of enthusiasts. If they disappeared from Youtube .... It'd be good to set up a universal 'permalink' library system (UPLS) like the DOI system and its 'persistent interoperable identifiers'. This could (and, at least in part, must) be publicly-funded. Anyone who wishes could apply for a unique ID. Then, subject to a set of specs from some participating library ("yes, we'll host that"), they could package that content with metadata. Backups must be ensured. If eventually the content needs to move (or can't find a home), the ID (and metadata) remains.
  • 0xbadcafebee 5y ago
    I think about used book stores, libraries, the library of congress, and just shelves in people's homes. All of that allows books to survive, when they're printed enough, and people actually want them. Lots of books, magazines, newspapers do not survive because nobody wants to read them. We don't keep a super-archive of every piece of paper ever published, because we acknowledge that it's OK for things to die. Important things we keep, useless things we throw away. It's irrational to hoard garbage. So really, all we need is to proactively decide what we do want to save, and save it as soon as it's published. Maybe also provide a "digital shelf" for people to keep their own copies, and a standard way to search for and distribute that information, like NNTP + Gnutella for e-books (along with some features to avoid being sued a la DMCA). The rest should be allowed to rot like fallen logs in the woods.
• crazygringo 5y ago
  We need to accept that link rot and content drift are part of the web. And realize that the best place to preserve history is the Internet Archive' Wayback Machine. Kind of the same way newspapers were never responsible for maintaining their archives, but librarians did on microfiche (remember that?). But I'd take it farther. First, the Internet Archive ought to have an official partnership with the Library of Congress and other national libraries across the world, that help provide funding. It shouldn't have to rely on private donations. And second, it's time browsers integrated with it -- if content no longer exists there should be a built-in option to easily check Wayback Machine with a single click, and use a heuristic to show the most recent "good" version. In other words, let the Wayback Machine be not just a, but the place for the Internet's history. Let's make it official.
  • cratermoon 5y ago
    Newspapers kept extensive archives, they called them 'the morgue', and depending on the newspapers and era, they either had microfilm/fiche or actual physical clippings. When I was studying journalism in college I would get access to all the newspapers morgues. Really anyone could call up the newspaper and ask for something from a past issue, although if they needed to actually do much searching there were other hurdles. They weren't public libraries, but they served their communities. ETA https://en.wikipedia.org/wiki/Morgue_file https://en.wikipedia.org/wiki/Morgue_file
    • cratermoon 5y ago
      And of the eight external links on that page, four are now broken.
• forgotmypw17 5y ago
  Luckily, community-driven emergent self-healing adaptations are also endemic to the Web :)
• BiteCode_dev 5y ago
  Part of it is technical. The most modern stack usually completely break deep linking: - even the best spa out there often barely shim the normal browser behavior. Yesterday my back button broke once more, in 2021. Infinite scroll don't let you pinpoint your position. User don't expect being able to copy / paste a link to take them to the content anymore. - the ecosystem of url handling is fractured. This month I worked on a Django + React app, and my clients asked that it should be able to handle being hosted behind an arbitrary URL prefix if provided in the conf. Here are the things I had to tweak: * adding the prefix to the proxy pass apache conf (yes, they are still using it); * adding the prefix the react router conf for which most tutorials were outdated; * adding the prefix in the js bundler conf as the base, and for the dev proxy; * adding the prefix in the <base> element in the main template; * making all urls and ajax calls relative to the <base>; * making all the react router Link and history.push _absolute_ (took me a while to figure this out); * serving the index.html as a template file from django, not nginx, to inject all that stuff according to the env var. * hacking the build script to replace static files URLS with template place holders because the js bundler didn't have a hook for that (thanks sed); And that's on top of the regular work of making urls in SPA works, which implies sync your backend and frontend URLS for pages and API. Who is going to do all that works? In fact, how many devs have the knowledge to do that? Pre-SPA, there would have been well documented 2 steps to do the same thing. The junior in the team could figure it out. - we had a ton of manure on top of our urls. AMP. Url shorteners. Tracking ID and redirections. Content wall. Captcha. Often several of them at the same time. If one of them break in the chain, goodbye URL. - low code mean low skill devs, that never heard the mantra "cool url don't change". They don't even know they should care. - some browsers just hide the URL. The users don't know what an url is anyway. - apps don't care about deep linking. They could handle url fine, mind you. We have the tech for it. But it's not even on the radar of most devs. You don't address the content, you consume whatever pops up, so why bother ? Plus, google is so good at finding the content you want out of the barely readable drunken mess of letters you feed it that most people don't type url anymore. People don't care about URL just like people don't care about bees dying, because it's too abstract to worry about.
• kaichanvong 5y ago
  Its brilliant. If the link is bad, find a different link. Email the webmaster if you believe its OK. Think about if you can actually help. Hashtag systematic whine.
• dredmorbius 5y ago
  A common misapprehension, even amongst those who have been engaged at foundationally-deep levels of the Web, is that the World Wide Web is not an archival mechanism but a publishing mechanism. More specifically, it's an on-demand publishing mechanism, relating to media-based publishing (books, records, physical video media) much in the same was as the electrical grid (a transmission mechanism) does to fuels (a storage function). It would be nice (in at least some regards) if publishing content at a specific addressable URL were a promise to 1) eternally provide that resource and 2) never change it. But there's no way to guarantee that this will be the case. In the past, the means for achieving archival of information was: - To specifically record that information in some form. As with, say, Plato's memorialising of Socrates's dialgogues. - To create multiple copies of those recordings, so that loss of any one instance doesn't mean total loss. - To define a refererencing or indexing system such that individual works can be identified unambiguously. (Or more specifically, with an acceptable level of ambiguity.) - To develop a means of agreement as to what the canonical or recongised version of a work is, or absent that, of identifying more canonical forms or lineages. (The historiography of the history of philosophy is an interesting sub-field, with a good introductory treatment in Peter Adamson's History of Philosophy Without Any Gaps podcast.) URLs of and by themselves address these needs poorly. At best they point to a location at which a document may have been available at a point in time. A URL plus a time-range (which is what the Internet Archive's Wayback Machine effectively delivers) is a much better approximation to what is needed. (The notion of time-bounded identifiers generally seems a useful one, as might be applied also to domain and user names, for example.) As I see it, the goal of the archival web needs to address a number of points which Zittrain addresses only very indirectly: - Identification of what really should be archived. Right To Be Forgotten exists for very well-founded reasons, and a world without forgetting (or with very capricious forgetting) is one form of hell. - True document-centric identification. I've been thinking about this for a while (recent discussion here: https://news.ycombinator.com/item?id=27455520 https://news.ycombinator.com/item?id=27455520), and something that is based on the actual contents while being resilient to mild changes seems most optimal. Checksums, not so much, tuples or ngrams, possibly warmer. - A number of archival institutions. The Internet Archive is certainly amongst these. Other libraries, if at all possible spread across multiple institutions and jurisdictions would be preferable. IA have been working with numerous academic institutions and the US Library of Congress, though IA itself still carries most of the burden. This isn't a new problem. In particular, each time there's been an explosion in some new form of publishing, there's been a scramble by archivists to keep up. Denis Diderot, 18th century encyclopaedist, has an awesome quote about the information explosion drowning his own generation (the encyclopaedia was his technical solution to that problem).[1] Numerous elements of what we now accept as standard bibliographic elements (titles, authors, tables of contents, indices, references, citations, page numbers, paragraphs, inter-word spaces, ...) were invented to answer specific needs, not always of archivists as a principle focus, though often providing benefits to them. Cataloguing and classification systems likewise. I appreciate Zittrain's message. He's crying over a lost cause and looking backwards, not to the future. ________________________________ Notes: 1. Diderot: https://www.historyofinformation.com/detail.php?entryid=2877 https://www.historyofinformation.com/detail.php?entryid=2877
• jefurii 5y ago
  Yes, and paper books go out of print.
• tegiddrone 5y ago
  I would love to work on micro format/protocol which allows me to link to the original content but also include an IPFS node to a WARC. Have a little cli/desktop tool which produces and pins a WARC. <a href="https://example.com/some-article" data-ipfs-warc="0xB45165ED3CD4 ..."> Some Article </a>
• raxi 5y ago
  I'd highlight two arguments which are mentioned in the article but shadowed by main topic of prevention-linkrot-by-archiving (no wonder, as it is written by Perma.cc co-founder): 1. Failure of Lumen/ChillingEffect initiative to prevent bogus takedown requests. Currently, anybody is able to takedown any page on the Internet by sending bogus requests and then takedown any mention about who did it. 2. Google's failure to “organize the world’s information and make it universally accessible and useful”. As the author said: “no such transparent, academic competitive search engine exists in 2021“. I see some relation between those two.