9 ms·

38% of webpages that existed in 2013 are no longer accessible a decade later

231 points by Kye 2y ago

Bostonian 2y ago
If you see useful computer code on someone's site, do you need permission to mirror it publicly on GitHub with attribution and the same license?
- tuwtuwtuwtuw 2y ago
  Depends on license.
  - williamcotton 2y ago
    It doesn’t, actually: https://www.copyright.gov/fair-use/ https://www.copyright.gov/fair-use/ more likely to find that nonprofit educational and noncommercial uses are fair. less likely to support a claim of a fair use than using a factual work (such as a technical article or news item). if the use employs only a small amount of copyrighted material, fair use is more likely. And since code tends to be more idea than expression most of it can be considered to not fall under copyright after the application of the Abstraction, Filtration, Comparison doctrine. Of course if you piss off an entity with a bunch of money to throw at lawyers it could be a bigger issue, regardless if you’re in the right, because defending yourself can rack up legal fees. I do copyright/patent/trade secret inspections of source code for a living. EDIT: Yet again, downvoted for just stating the truth... The irony is that 10 years ago and before the controversies around LLMs this comment would not have garnered negative attention because the forum was all for weaker copyrights... when copyright affected musicians instead of programmers' bottom lines... Sigh...
    - nicbou 2y ago
      There are other countries with different laws. In Germany it would definitely be copyright infringement.
      - williamcotton 2y ago
        The entire world benefits from sites like the Internet Archive and the commons-friendly approach of fair use. I recommend changing the laws in your country! EDIT: I did a little bit of investigation and there are similar limits on copyrights in Germany known as "limitations on protected rights" that seem to carve out things like educational use and archiving, but I don't know anything about German law. I would find it surprising if most Western nations didn't have something similar to fair use unless there was an active interest in damaging the public's access to information.
    - chuckadams 2y ago
      Fair use is an affirmative defense: you only get to pull it out after they sue you, and your claim is judged on a case by case basis. It’s not the magic spell of protection people seem to think it is.
      - williamcotton 2y ago
        Yup, I know that, but I encourage people to not be afraid of republishing little bits of code for educational or archival purposes. They should know they do have fair use to lean on if they do end up in the incredibly unlikely situation of ending up in court.. unless of course they're painting a very large target on their backs and making quite a bit of money from the IP of a large corporation!
- z3t4 2y ago
  You can take small snippets, but not the whole page. But why do you want to release someone else work publicly!? Just save it on your own device if you find it useful.
  - williamcotton 2y ago
    In the context of the article, I assume because they want to archive the work for a future audience.
- arlcode 2y ago
  Im am not a lawyer but as far as I know ,it depends on you jurisdiction and the exact code in question. Many legal systems don't know "fair use" and by default you have effectively zero rights to do anything with copyrighted materials without explicit permission. The license will tell you what is allowed (and If its a standard one you can assume it is in accordance with the law)
- deleted 2y ago
  [deleted]
dotcoma 2y ago
including twitter.com ! ;)
ivan_gammel 2y ago
This is a feature, not a bug. It would be a terrible life in a world that does not forget or forgive. It’s also good that some preservation effort is necessary for worthy content: the value of it gets more appreciated.
- ants_everywhere 2y ago
  > It would be a terrible life in a world that does not forget or forgive This is an orthogonal concern, and arguably is mainly about privacy > It’s also good that some preservation effort is necessary for worthy content: the value of it gets more appreciated. This same argument seems to imply that virtually everything should be expensive. Cheap storage is bad because we don't appreciate the value of the files we store. Expensive healthcare is good because it really makes us appreciate our organs. > worthy content The hard part is looking into the future to determine which content will be considered worthy then. So far no human civilization has managed to figure that out. They mostly seemed to focus on preserving the image of how amazing their kings were.
  - ivan_gammel 2y ago
    >This same argument seems to imply that virtually everything should be expensive Non-zero doesn’t mean “unaffordable” or “expensive”. > The hard part is looking into the future… The hardest part is to understand that the content we want to preserve carries more valuable information about us than about itself. Scientific knowledge can be discovered again, it’s not something to worry about. The preservation shapes the future views of us, leaving the trace in the history of those who preserve, their life and their experiences. Maybe they just needed to accept their mortality and irreversible flow of time?
    - ants_everywhere 2y ago
      I don't know. Scientific knowledge is expensive to rediscover. It requires a lot of false starts and often involves a great deal of luck/randomness. Historically, periods of flourishing are often associated with the rediscovery or importing of large collections of wisdom. For example revivals of ancient works or the introduction of works from afar due to newly developed trade routes. If ideas were easy to rediscover, we shouldn't expect those events to have much impact. There is certainly a cost of storing data, and cost should enter the equation. But we're losing a lot of data for reasons other than cost and we don't have a reasonable way of assigning a value to the lost data.
badgersnake 2y ago
I’m surprised it’s that low to be honest. Most of the web seems to by SEO crap these days.
- brabel 2y ago
  Agree. Sometimes you just experiment with something, put up a tiny website somewhere... forget about it until you decide it's no longer relevant for whatever reason and you pull the plug on it... it's not a bad thing. But it's great to have stuff like web archives though, to keep our collective memory for worthwhile content. I specially hope that accurate accounts of events gets preserved, as it was originally written, somewhere it can't be changed. That's because rewriting history seems to be a favourite these days and preserving the original accounts as things were happening can help combat this, and even if the account were not completely accurate, it can help understand the actions of contemporary actors - i.e. you may be able to understand what they thought was true at the time, even if that was later revealed to be incorrect.
- nicbou 2y ago
  Some things still exist but are just no longer surfaced by Google.
theginger 2y ago
This is very glass is half empty. Why isn't this, 62% of websites from 2013 are still available 11 years later!
- wiseowise 2y ago
  Because this isn’t good news at all.
- Zambyte 2y ago
  That still doesn't sound very good to me. My mind quite effortlessly inserts "only" before 62%.
- psychoslave 2y ago
  I would be interested to know how it compares with printed books for example.
bryanrasmussen 2y ago
Yeah, I've noticed this - I made a funny little art project in 2013 wordclouding parodies of This is Just to Say (and later other famous poems and literary quotes that often get parodied) because wordclouding is the least interesting kind of visualization there is: https://medium.com/luminasticity/fruit-stealing-scoundrel-has-returned-aca2bf72dd7b https://medium.com/luminasticity/fruit-stealing-scoundrel-ha... and quite a lot of the content I used is no longer found, so all that remains is the word cloud but not the original poem. It's a bit worse than the article details though because often the web page is there, or if not is available from IA, but much of the content was actually inside of comments and so forth. (project was originally inspired because back then you couldn't go anywhere on the web without someone hitting you over the head with their This is Just to Say parodies)
squarefoot 2y ago
Be sure to donate some quid to the Internet Archive (archive.org) to support their efforts to preserve (not just) old content, then do your best to make local copies of anything you find of value, just in case they disappear one day. A good number of mostly technical pages I have in my bookmarks file, that grew steadily and has been moved during installations for over 20 years, now point to their latest complete backup before the said page went silent. The Internet Archive is a huge boon to everyone.
- massysett 2y ago
  I realized I was overusing bookmarks. I now save webpages (perhaps as PDF) if it contains information I want to refer to later, such as an insightful article, technical information, a humorous bit, or the like. Bookmarks are good only for links to things for which only the most current version is worth accessing. That’s my banking websites, a shopping site, my employer’s remote desktop system, etc.
  - dewey 2y ago
    There's also https://archivebox.io https://archivebox.io which can take your bookmarks and archive them in many ways. Unfortunately back when I tried it last time it was a big buggy, I wish there was a better solution to build a nice archive of the sites I visit more often just in case.
    - zimpenfish 2y ago
      I found the Chrome it uses for fetching needs a kick every couple of days. That definitely improved my experience.
  - dotancohen 2y ago
    How much disk space did that consume? I like the idea that in addition to saving the page, you can annotate it as well.
    - massysett 2y ago
KronisLV 2y ago
From the point of actually trying to keep something online, it's basically constantly having to fight against code rot. You don't update your server or database or runtime/framework/library? You'll get hacked and will drown in CVEs. You do try to do these updates? Have fun rewriting bits of your code, because the old version of a framework/library is no longer supported and there are breaking changes, which mean needing a partial rewrite. Your best bet around that might be one of the relatively stable databases like SQLite, a micro-framework on the back end for a RESTful API, a simple solution for auth like basicauth/mTLS/... at the web server level in front of your API and then something without a toolchain on the front end, like jQuery. I mean this unironically, unless you want to maintain very few sites, then you probably have a bit more time on your hands. Feels to me like the only content that can have any sort of longevity without constant investment of time is static sites - where updating your web server or moving to a different one is trivial and there are no write operations involved in most of the processes (maybe setup logrotate or just delete the logs occasionally).
- jonnycomputer 2y ago
  One of the reasons I moved my personal site to static. Don't have to think about it anymore.
  - TaylorAlexander 2y ago
    Yah I’m thinking about the Space Jam website. HTML doesn’t seem to rot like some hot new JS library does.
    - bilalq 2y ago
      It's an issue of servers, not JS. A BackboneJS app written in CoffeeScript back in 2010 would still run just fine today if it was hosted on S3/CloudFront. Replace the framework/language/year with anything, and it'd still hold true. But if the page needed to fetch data from an unmaintained API server that ran out of disk space, lost its DB network connection, got rebooted by a VPS provider, or any other issue, that site will probably never work again.
deleted 2y ago
[deleted]
agumonkey 2y ago
How much of pages today are not worth accessing ?
daniel31x13 2y ago
This is one of the main reasons I created Linkwarden - to combat Link-Rot. Linkwarden is an open-source collaborative bookmark manager to collect, organize and preserve webpages: https://linkwarden.app https://linkwarden.app
vaylian 2y ago
Something I've been wondering about for some time: Aaron Swartz's website is still online more than 10 years after his suicide: http://www.aaronsw.com/ http://www.aaronsw.com/ Someone has to keep paying the bills and renewing the domain. Still no HTTPS though. Does anyone know how this is handled?
- Kye 2y ago
  >> "Server information sidebar: This site is being served from an Ubuntu box with 2GB of RAM. The server is currently provided by several people" This is either added since his death, and it's maintained by supporters, or it was there to begin with and one of the several people took over maintenance and funding. But that still leaves the question of who. Maybe they want to remain anonymous.
k__ 2y ago
Arweave tries to tackle this issue. https://arwiki.wiki/#/en/the-permaweb https://arwiki.wiki/#/en/the-permaweb
- bugmenot1k 2y ago
  [dead]
martindbp 2y ago
I have one of those catalogs from 1996 with an index of web pages for different topics. Maybe I should scan it and do the same experiment
xbmcuser 2y ago
I think a bigger problem than 38% of webpages being dead is a lot of it is entities/groups/businesses now use facebook pages almost exclusively and have no other web presence outside of Facebook. In other words a Facebook account becomes a requirement to interact with them.
- dzhiurgis 2y ago
  I get your sentiment, but facebook also acts as a spam filter - not entirely bad thing for business owner
  - Zambyte 2y ago
    Funnily enough I specifically don't use Facebook or other Facebook owned services because of all of the spam.
  - soulofmischief 2y ago
    Works both ways, too! If a business is only on Facebook, I don't do business with them as I don't use Facebook. A win-win in my book, as I prefer doing business with people whose ethics overlap with my own.
    - vundercind 2y ago
      Maybe it’s different where you are, but around here that filter would mean I could almost only patronize large chains. Small businesses have Facebook or maybe insta (which is much worse, Facebook business pages grant far more access to a non-logged-in user) and no website. Restaurants might have a barely-updated website (the updates are on Facebook) that links to some third party ordering service, maybe.
      - soulofmischief 2y ago
        I definitely get your point. But it's clear that continuing to use Facebook in that manner will only strengthen the isolation effect. Voting with your wallet and going against the machine invariably involves some level of personal sacrifice. For me, sacrificing patronage is incredibly easy to do. There is more to life than commercialization. My girlfriend says she only uses Facebook to interface with small stores, who use it as a sole point of contact or distribution. Let that sink in for a moment. Breaking this cycle will require hard work.
pif 2y ago
What has humankind lost with those pages vanishing?
- nicbou 2y ago
  In some cases, a lot of valuable information that doesn't exist anywhere else. A big German immigration forum vanished this winter. There was a lot of valuable information for people navigating tricky bureaucratic processes.
gregoriol 2y ago
How many % of brick and mortar shops from 2013 are still in business today? How many % of people who were alive in 2013 are still alive today?
- Kye 2y ago
  Let's preserve those, too. There was a really nice music school/venue/coffee shop in the town I used to live in. It shut down, and they took their Facebook and Instagram pages down. The only evidence it ever existed is in posts announcing events on other Facebook pages, in memories of people who went there, and on still surviving business listings that are likely to go away.
deleted 2y ago
[deleted]
frankthepickle 2y ago
are we saying we're going to keep making hard drives so we can save everything ever produced? I see the value in many things but I worry about the load of expecting everything to be saved forever.
Ferret7446 2y ago
A natural consequence of federation. You could attempt to archive everything and host it. But ultimately, each host is responsible for what it hosts and keeping itself available.
- detourdog 2y ago
  That is what I like about federation. All the incentives are individualized. I think the scale that was needed for business in the last century has disappeared. Businesses previous needed scale and reach but for an individual to make a decent living in a niche is very easy today.
robertlagrant 2y ago
> It should be a standard for the open internet, that any respectable page has an "export complete archives" as a clickable button. (But it's mainly the opposite, today: adtech corporations don't want your written works to be free, they want them incarcerated in their revenue-generating walled gardens—and if ten billion human's worth of written history gets erased at the end, well, too bad for them!) [0] Can't you export your data from social media platforms? I haven't tried it, but did Google this for Facebook[1] and it looks as though you can? [0] a now-deleted comment [1] https://www.techlicious.com/how-to/how-to-download-all-your-facebook-data https://www.techlicious.com/how-to/how-to-download-all-your-...
- MattGaiser 2y ago
  I’ve exported my data on Facebook before.
throwaway290 2y ago
This will get only worse as people realize that almost anything published is fed into ML products and used against them.
barfbagginus 2y ago
I saw meme beams glitter past Ebaums Gate on the arm of Orion. These moments will be forgotten, like Cheezburgers in the rain. Prepare to die.
Brosper 2y ago
It's like with restaurants. A lot of them disappeared after the pandemic. Web hosting is not free.
- rchaud 2y ago
  Majority of Wordpress.com and Blogger.com sites are on the free tier.
  - Brosper 2y ago
    OK, so domains are not free. But many businesses had their own hosting and domain.
onion2k 2y ago
I'm surprised it's not more. 2013 was long after the days of hobbyist websites of the early net, and into the time when most new sites were business driven. Given how long businesses last I'd expect many more sites to be long gone 11 years later. I guess maybe the death of a lot of community-building spaces (angelfire, Geocities, etc) probably counts for a lot of them going. What would be particularly interesting would be to graph how long websites last for. I suspect quite a lot of the content from the early days is still around, and this period (2008 - 2018) is the peak of sites vanishing.
- rchaud 2y ago
  A lot of the content from the early days was on platforms which are long dead: - Geocities - University-provided FTP folder (deleted after you graduate) - ISP-provided FTP folder (all those Earthlink, Juno, Comcast sites: probably deleted)
Springtime 2y ago
The predominant sentiment I'm seeing in comments here is a 'what does it matter?' While at the same time in discussions here about search engine result quality inevitably the most popular comments express a decline in quality (influenced by SEO, AI, spam, among other things), along with a desire for surfacing interesting, human made content that gets buried by modern search algorithms. On HN we see every day interesting, first page content marked with a title representing its year, sometimes dating back a decade or more. Its age isn't apparently a detracting aspect to this audience if the content is still worth sharing. And from the article the headline figure also doesn't represent irrelevant/undesirable content either but Wikipedia references, news articles, government pages, along with less unexpectedly ephemeral things like Twitter posts. We're lucky there is archive.org but since it's not indexed like a regular search engine the only tether to old pages are still-live links found via regular search engines/sites (including HN). Essentially unless sites continue to exist that contain links to archived content the chances of future discovery becomes slim. My stance is if you find something interesting that you expect is worthwhile sharing try saving it in the most convenient single file page format available to you (MHTML, SingleFile, PDF), to have your own copy. For MHTML at least it also saves the original page URL in its metadata. Saving to online archives is also great but admittedly higher friction (and can sometimes result in things like IP restrictions on archive.org even when saving just a handful of pages in a row, ime).
- Kye 2y ago
  The neat thing about mhtml is it's plain old MIME. You can change the extension to eml and open it in email clients.
amanzi 2y ago
Some of the better websites at least make an effort to archive old content. e.g. here's CNN and BBC websites with coverage from the 9/11 attacks: http://news.bbc.co.uk/hi/english/static/in_depth/americas/2001/day_of_terror/ http://news.bbc.co.uk/hi/english/static/in_depth/americas/20... http://edition.cnn.com/SPECIALS/2001/trade.center/index.html http://edition.cnn.com/SPECIALS/2001/trade.center/index.html Don't expect many of the links to work properly, but it's still interesting to see what the web used to look like.
- mhh__ 2y ago
  Some of the interactive stuff on old BBC election coverages still almost work to this day. Hard to imagine that with many sites now 20 years on. It's not even that it;s impossible with the technology, it's probably closer to how writing got worse after the invention of the word processor. Every thing is managed and structured now so the freedom / bubble needed to make things good in a way that can't be easily explained is gone.
eigenvalue 2y ago
I view this as a serious failing of the internet that we collectively should have done a better job of avoiding. In most cases I believe the content itself is in fact still available somewhere and it’s simply the link that broke. Some kind of two layer system like the DOI system used for libraries would be helpful for cases like that: https://nuim.libguides.com/referencing/DigitalObjectIdentifiers https://nuim.libguides.com/referencing/DigitalObjectIdentifi... But it would need to handled automatically to maintain the full usefulness and convenience of URLs. Not sure how that could work though.
shrubble 2y ago
Not surprising - even HP's support site is full of dead /broken links, you can't even locate information on 2013 era desktops and servers.
zargath 2y ago
As many have stated, I would assume more than 38%. But good quality content is rare and the dynamic content made page combinations go infinite 20 years ago. We are maintaining 25y old urls is a bit ehh. cumbersome, and I sometime wonder if its worth it. Most of the traffic seem to come from bots and they do seem to learn some of the 301's. It seems to be good for SEO, etc. Some users also gets redirected to the content on the new urls. It feels a bit like helping an elderly person over the street to where the shop is. Anyway, I hope that bots and humans trust our services more.
ekianjo 2y ago
> Nearly one-in-five tweets are no longer publicly visible on the site just months after being posted. In 60% of these cases, the account that originally posted the tweet was made private, suspended or deleted entirely. In the other 40%, the account holder deleted the individual tweet, but the account itself still existed this seems extraordinarily high. since they cant do that estimation on all tweets ever sent it must be a bias of their sample
ksec 2y ago
One of the reasons why Bookmarks are useless. Most of them are dead, and those that are not can not easily be searched within Google. The web we have today is just worst than pre-Google dominant era.
axegon_ 2y ago
Honestly, seeing startups die and people abandoning their blogs(of which I'm also guilty, despite my best intentions, I just happen to pay my domains for 10 years at a time), I would have expected the number to be much higher.
earthboundkid 2y ago
I have run a news website since 2019. Every hour, I have a crawler look for dead links. I replace about one link a day with a link to archive.org. The funniest ones are the day after an election when all the candidate websites go blank. The saddest are the government websites that go offline from 3am to 5am every week.
- notRobot 2y ago
  Interesting, does your crawler check every link every hour, or does it go through them a batch at a time?
  - earthboundkid 2y ago
    Every link every hour. https://github.com/spotlightpa/linkrot https://github.com/spotlightpa/linkrot
zokier 2y ago
Everything on internet is intrisically ephemeral. Embrace that instead of fighting against it. If you want to archive stuff then make offline copies. PDF/A (especially the -1 and -2 versions) is format explicitly designed for archiving and works well for static content. I think it is bit of a shame that mirroring is not more readily built into web stack (=http/html); if you could trivially make links that included local copy (as fallback?) this linkrot would be far lesser concern. The way how for example wikipedia links everything through archive.org is bit of a hack imho
PaulStatezny 2y ago
This is a good thing. In an age of information overload, an Internet that never contracts sounds overwhelming.
iAkashPaul 2y ago
Not being able to renew domains beyond 10yrs could also be a factor
runamuck 2y ago
I post my site content Markdown to an open Git repo for this reason. Anyone can pull and build my pages. I think Git should stay for at least another 100 years. https://github.com/hatdropper1977/john.soban.ski https://github.com/hatdropper1977/john.soban.ski
lagniappe 2y ago
I hope not all things last forever. A while back I stumbled upon my first .com, from the 90s, which was hosted on Angelfire and dutifully rehosted by archive.org and it went about how you'd imagine. Despite being in 4th grade when my little friend and I made the webpage, things on there (while fine for the era) are just not okay by today's standards even if I understand the context for what led to it being there. It was nothing terrible, but just distasteful in a blissfully unaware way a 4th grader in the 90's would be. I realize that stuff will probably never be off my conscience and I just have to deal with it and hope nobody sees it.
- otachack 2y ago
  I have similar material. If it's reassuring, we all were just kids/teens and learning of the world. I feel a lot for the youth after us that made the Internet more accessible and, at times, more permanent.
- BirAdam 2y ago
  I feel your pain. Thankfully, even the archive occasionally takes stuff off.
superkuh 2y ago
This is because the majority of webpages are on commercial (for profit) sites now and for-profit companies do not build anything that lasts. Part of this is their use case requirements: CA TLS has to be the only way to access the sites. And since CA TLS is extremely fragile and short lived, so are the sites. But additionally, any dynamic site also has a short lifetime. HTML files-in-folders sites will last till the heat death of the universe unmaintained. A php or nodejs/etc site will last a year or two after it stops being mantained. The core of the web pages on the internet are still there. It's just that the thick layer of commercial cr'app' websites built up on top are transient.
nimajneb 2y ago
I've been thinking about this for a couple years. With our (recent) history becoming more and more digital we are losing more of our history. There's lot of creativity of all forms being digital and online only lasting only as long as the creator supports it. No longer are we viewing physical pieces art, but ephemeral pieces art. (I'm including websites made for personal use, by hobbyists, etc). I know Archive.org archives a lot, but they can't archive everything, especially all the small personal websites.
- BirAdam 2y ago
  Not just personal sites, but even news sites. Storage is expensive. Site owners don’t want to hold stuff forever, and the Archive cannot afford to hold everything forever either.
b112 2y ago
If one looks at the graph, 20% of sites were dead in 3 years. This isn't a facebook thing, most facebook migration happened a decade ago. Instead this is companies closing, campaign websites shuttering, urls being changed, community events being over, and so on.
JoeAltmaier 2y ago
In 2013 there were maybe 630M websites. Today there are over a billion. So those 38% amount to 22% of today's internet Seems like a pretty good half-life!
bdcravens 2y ago
Not a problem. The web is a communication medium first and foremost, and part of the freedom of communication is the freedom to de-communicate.
epolanski 2y ago
I often look previous years posts when there is nothing I can find interesting for the day on HN. Go back few years already and most links are dead.
furyofantares 2y ago
While I am aware that nothing is permanent and everything takes maintenance, I sure feel like there ought to be a bit _more_ permanent way to publish things. I run a few daily word games. They're static sites that could continue forever as long as they have players, but sometimes I think about what would happen if I died or stopped paying attention. Domains would expire. Maybe some people could still play cached versions, and archive site versions would still exist. I host them on github pages so if I exposed that url then that could probably exist for as long as github does - or breaks something that requires the owner to click a button to fix or something. I could probably publish a free desktop version on various stores, and that might last as long as the store, or OS upgrades break it. It would be great if there was a way to publish something such that it exists as long as anyone cares about it.
- mikepurvis 2y ago
  I bet GitHub pages is more permanent than the iOS store ecosystem.
deleted 2y ago
[deleted]
markx2 2y ago
I blogged from 2000 onward but hit my stride with Wordpress in 2004. I posted thousands of times. I had a Google PR on 8 at one point but it tended to be 6/7. In 2020 I decided to shutter the site. The database with the content is still there, it's just inaccessible. Google, Bing, Archive.org and others I contacted and had them remove my content. Then I removed everything visible. Very very happy I did, especially with the rise of AI.
renegat0x0 2y ago
Attention is limited. We cannot see everything on the Internet. We do not have enough time for that. There is a lot of valuable and interesting data on the Internet, but it is not visible. Certainly high quality, low profile blog that ended its development in 2015 will not be ranked high in Google. Media platform, search engines monetize content. YouTube channels need to churn new content every week or so to stay relevant and to stay watchable. Our society produces content, not quality, not products. SEO can be gamed, it is impossible to create objective index of valuable content. Bad actors will hack the game, spam results, destroy quality to gain profit. Google search engine most often connects users with media sites, with news sites, with the middle men. The more often not connect users with product directly. Write "search engine" in search query, you may not only find search a "search engines" but articles about "Best search engines in 2024", or "best SEO tricks to boost your page". Google does not have any incentive to fix this. Search engines are dead tech. It will be replaced by chatbots in a few years. People will not search for content, content will be generated at wish. Some time ago I have created my own domain repository with domain names: https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database I wanted to find "wargames" related pages. It is quite impossible to find anything interesting concerning warhammer on the normie internet (not Facebook). The second thing is I cannot find anything "amiga" related. This solved this my initial problem. I have also found out that many interesting pages are gone. I think that Google directing our attention toward "content" broke good quality pages. Right now I am using less and less google, because I use more and more my bookmark manager. https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive My solutions may not be as complex as common crawl, but they are enough for me. For now. I am still working on my program. It has been fun and interesting experience for me, and I learned a lot. About open graph protocol, about schema, about web scraping, etc. etc. Maybe this will inspire people to be more self sufficient, and more self-hostable. In times of walled gardens we need more standard, and more open data to keep what remains of the old wild west of the Internet.
EasyMark 2y ago
That's a much lower number than I would have suspected after 10+ years.
benwillies 2y ago
I remember people saying that once it is on the web it is there forever. But my experience is how fast web sites disappear. Or get redirected to a nonsensical site in the orient. I think maintaining the domain name is the big problem. However I am proud to say that you can still see my very first published web page from 1995 if you know the rather obscure url... http://admin.benwillies.com/ticker/ http://admin.benwillies.com/ticker/ I wrote this page as a proof of concept for a friend of mine who was a financial consultant but unfortunately the humor was a turn-off instead of getting him excited about this new fangled thing called the Web. I never made that mistake again.
mediumsmart 2y ago
This is about single pages that have been deleted in functional websites. How dare you?

231 points by Kye 2y ago

If you see useful computer code on someone's site, do you need permission to mirror it publicly on GitHub with attribution and the same license?

tuwtuwtuwtuw 2y ago
Depends on license.
- williamcotton 2y ago
  It doesn’t, actually: https://www.copyright.gov/fair-use/ https://www.copyright.gov/fair-use/ more likely to find that nonprofit educational and noncommercial uses are fair. less likely to support a claim of a fair use than using a factual work (such as a technical article or news item). if the use employs only a small amount of copyrighted material, fair use is more likely. And since code tends to be more idea than expression most of it can be considered to not fall under copyright after the application of the Abstraction, Filtration, Comparison doctrine. Of course if you piss off an entity with a bunch of money to throw at lawyers it could be a bigger issue, regardless if you’re in the right, because defending yourself can rack up legal fees. I do copyright/patent/trade secret inspections of source code for a living. EDIT: Yet again, downvoted for just stating the truth... The irony is that 10 years ago and before the controversies around LLMs this comment would not have garnered negative attention because the forum was all for weaker copyrights... when copyright affected musicians instead of programmers' bottom lines... Sigh...
  - nicbou 2y ago
    There are other countries with different laws. In Germany it would definitely be copyright infringement.
    - williamcotton 2y ago
      The entire world benefits from sites like the Internet Archive and the commons-friendly approach of fair use. I recommend changing the laws in your country! EDIT: I did a little bit of investigation and there are similar limits on copyrights in Germany known as "limitations on protected rights" that seem to carve out things like educational use and archiving, but I don't know anything about German law. I would find it surprising if most Western nations didn't have something similar to fair use unless there was an active interest in damaging the public's access to information.
  - chuckadams 2y ago
    Fair use is an affirmative defense: you only get to pull it out after they sue you, and your claim is judged on a case by case basis. It’s not the magic spell of protection people seem to think it is.
    - williamcotton 2y ago
      Yup, I know that, but I encourage people to not be afraid of republishing little bits of code for educational or archival purposes. They should know they do have fair use to lean on if they do end up in the incredibly unlikely situation of ending up in court.. unless of course they're painting a very large target on their backs and making quite a bit of money from the IP of a large corporation!
z3t4 2y ago
You can take small snippets, but not the whole page. But why do you want to release someone else work publicly!? Just save it on your own device if you find it useful.
- williamcotton 2y ago
  In the context of the article, I assume because they want to archive the work for a future audience.
arlcode 2y ago
Im am not a lawyer but as far as I know ,it depends on you jurisdiction and the exact code in question. Many legal systems don't know "fair use" and by default you have effectively zero rights to do anything with copyrighted materials without explicit permission. The license will tell you what is allowed (and If its a standard one you can assume it is in accordance with the law)
deleted 2y ago
[deleted]

dotcoma 2y ago

including twitter.com ! ;)

ivan_gammel 2y ago

This is a feature, not a bug. It would be a terrible life in a world that does not forget or forgive. It’s also good that some preservation effort is necessary for worthy content: the value of it gets more appreciated.

ants_everywhere 2y ago
> It would be a terrible life in a world that does not forget or forgive This is an orthogonal concern, and arguably is mainly about privacy > It’s also good that some preservation effort is necessary for worthy content: the value of it gets more appreciated. This same argument seems to imply that virtually everything should be expensive. Cheap storage is bad because we don't appreciate the value of the files we store. Expensive healthcare is good because it really makes us appreciate our organs. > worthy content The hard part is looking into the future to determine which content will be considered worthy then. So far no human civilization has managed to figure that out. They mostly seemed to focus on preserving the image of how amazing their kings were.
- ivan_gammel 2y ago
  >This same argument seems to imply that virtually everything should be expensive Non-zero doesn’t mean “unaffordable” or “expensive”. > The hard part is looking into the future… The hardest part is to understand that the content we want to preserve carries more valuable information about us than about itself. Scientific knowledge can be discovered again, it’s not something to worry about. The preservation shapes the future views of us, leaving the trace in the history of those who preserve, their life and their experiences. Maybe they just needed to accept their mortality and irreversible flow of time?
  - ants_everywhere 2y ago
    I don't know. Scientific knowledge is expensive to rediscover. It requires a lot of false starts and often involves a great deal of luck/randomness. Historically, periods of flourishing are often associated with the rediscovery or importing of large collections of wisdom. For example revivals of ancient works or the introduction of works from afar due to newly developed trade routes. If ideas were easy to rediscover, we shouldn't expect those events to have much impact. There is certainly a cost of storing data, and cost should enter the equation. But we're losing a lot of data for reasons other than cost and we don't have a reasonable way of assigning a value to the lost data.

badgersnake 2y ago

I’m surprised it’s that low to be honest. Most of the web seems to by SEO crap these days.

brabel 2y ago
Agree. Sometimes you just experiment with something, put up a tiny website somewhere... forget about it until you decide it's no longer relevant for whatever reason and you pull the plug on it... it's not a bad thing. But it's great to have stuff like web archives though, to keep our collective memory for worthwhile content. I specially hope that accurate accounts of events gets preserved, as it was originally written, somewhere it can't be changed. That's because rewriting history seems to be a favourite these days and preserving the original accounts as things were happening can help combat this, and even if the account were not completely accurate, it can help understand the actions of contemporary actors - i.e. you may be able to understand what they thought was true at the time, even if that was later revealed to be incorrect.
nicbou 2y ago
Some things still exist but are just no longer surfaced by Google.

theginger 2y ago

This is very glass is half empty. Why isn't this, 62% of websites from 2013 are still available 11 years later!

wiseowise 2y ago
Because this isn’t good news at all.
Zambyte 2y ago
That still doesn't sound very good to me. My mind quite effortlessly inserts "only" before 62%.
psychoslave 2y ago
I would be interested to know how it compares with printed books for example.

bryanrasmussen 2y ago

Yeah, I've noticed this - I made a funny little art project in 2013 wordclouding parodies of This is Just to Say (and later other famous poems and literary quotes that often get parodied) because wordclouding is the least interesting kind of visualization there is: https://medium.com/luminasticity/fruit-stealing-scoundrel-has-returned-aca2bf72dd7b https://medium.com/luminasticity/fruit-stealing-scoundrel-ha... and quite a lot of the content I used is no longer found, so all that remains is the word cloud but not the original poem. It's a bit worse than the article details though because often the web page is there, or if not is available from IA, but much of the content was actually inside of comments and so forth. (project was originally inspired because back then you couldn't go anywhere on the web without someone hitting you over the head with their This is Just to Say parodies)

squarefoot 2y ago

Be sure to donate some quid to the Internet Archive (archive.org) to support their efforts to preserve (not just) old content, then do your best to make local copies of anything you find of value, just in case they disappear one day. A good number of mostly technical pages I have in my bookmarks file, that grew steadily and has been moved during installations for over 20 years, now point to their latest complete backup before the said page went silent. The Internet Archive is a huge boon to everyone.

massysett 2y ago
I realized I was overusing bookmarks. I now save webpages (perhaps as PDF) if it contains information I want to refer to later, such as an insightful article, technical information, a humorous bit, or the like. Bookmarks are good only for links to things for which only the most current version is worth accessing. That’s my banking websites, a shopping site, my employer’s remote desktop system, etc.
- dewey 2y ago
  There's also https://archivebox.io https://archivebox.io which can take your bookmarks and archive them in many ways. Unfortunately back when I tried it last time it was a big buggy, I wish there was a better solution to build a nice archive of the sites I visit more often just in case.
  - zimpenfish 2y ago
    I found the Chrome it uses for fetching needs a kick every couple of days. That definitely improved my experience.
- dotancohen 2y ago
  How much disk space did that consume? I like the idea that in addition to saving the page, you can annotate it as well.
  - massysett 2y ago

KronisLV 2y ago

From the point of actually trying to keep something online, it's basically constantly having to fight against code rot. You don't update your server or database or runtime/framework/library? You'll get hacked and will drown in CVEs. You do try to do these updates? Have fun rewriting bits of your code, because the old version of a framework/library is no longer supported and there are breaking changes, which mean needing a partial rewrite. Your best bet around that might be one of the relatively stable databases like SQLite, a micro-framework on the back end for a RESTful API, a simple solution for auth like basicauth/mTLS/... at the web server level in front of your API and then something without a toolchain on the front end, like jQuery. I mean this unironically, unless you want to maintain very few sites, then you probably have a bit more time on your hands. Feels to me like the only content that can have any sort of longevity without constant investment of time is static sites - where updating your web server or moving to a different one is trivial and there are no write operations involved in most of the processes (maybe setup logrotate or just delete the logs occasionally).

jonnycomputer 2y ago
One of the reasons I moved my personal site to static. Don't have to think about it anymore.
- TaylorAlexander 2y ago
  Yah I’m thinking about the Space Jam website. HTML doesn’t seem to rot like some hot new JS library does.
  - bilalq 2y ago
    It's an issue of servers, not JS. A BackboneJS app written in CoffeeScript back in 2010 would still run just fine today if it was hosted on S3/CloudFront. Replace the framework/language/year with anything, and it'd still hold true. But if the page needed to fetch data from an unmaintained API server that ran out of disk space, lost its DB network connection, got rebooted by a VPS provider, or any other issue, that site will probably never work again.

deleted 2y ago

[deleted]

agumonkey 2y ago

How much of pages today are not worth accessing ?

daniel31x13 2y ago

This is one of the main reasons I created Linkwarden - to combat Link-Rot. Linkwarden is an open-source collaborative bookmark manager to collect, organize and preserve webpages: https://linkwarden.app https://linkwarden.app

vaylian 2y ago

Something I've been wondering about for some time: Aaron Swartz's website is still online more than 10 years after his suicide: http://www.aaronsw.com/ http://www.aaronsw.com/ Someone has to keep paying the bills and renewing the domain. Still no HTTPS though. Does anyone know how this is handled?

Kye 2y ago
>> "Server information sidebar: This site is being served from an Ubuntu box with 2GB of RAM. The server is currently provided by several people" This is either added since his death, and it's maintained by supporters, or it was there to begin with and one of the several people took over maintenance and funding. But that still leaves the question of who. Maybe they want to remain anonymous.

k__ 2y ago

Arweave tries to tackle this issue. https://arwiki.wiki/#/en/the-permaweb https://arwiki.wiki/#/en/the-permaweb

bugmenot1k 2y ago
[dead]

martindbp 2y ago

I have one of those catalogs from 1996 with an index of web pages for different topics. Maybe I should scan it and do the same experiment

xbmcuser 2y ago

I think a bigger problem than 38% of webpages being dead is a lot of it is entities/groups/businesses now use facebook pages almost exclusively and have no other web presence outside of Facebook. In other words a Facebook account becomes a requirement to interact with them.

dzhiurgis 2y ago
I get your sentiment, but facebook also acts as a spam filter - not entirely bad thing for business owner
- Zambyte 2y ago
  Funnily enough I specifically don't use Facebook or other Facebook owned services because of all of the spam.
- soulofmischief 2y ago
  Works both ways, too! If a business is only on Facebook, I don't do business with them as I don't use Facebook. A win-win in my book, as I prefer doing business with people whose ethics overlap with my own.
  - vundercind 2y ago
    Maybe it’s different where you are, but around here that filter would mean I could almost only patronize large chains. Small businesses have Facebook or maybe insta (which is much worse, Facebook business pages grant far more access to a non-logged-in user) and no website. Restaurants might have a barely-updated website (the updates are on Facebook) that links to some third party ordering service, maybe.
    - soulofmischief 2y ago
      I definitely get your point. But it's clear that continuing to use Facebook in that manner will only strengthen the isolation effect. Voting with your wallet and going against the machine invariably involves some level of personal sacrifice. For me, sacrificing patronage is incredibly easy to do. There is more to life than commercialization. My girlfriend says she only uses Facebook to interface with small stores, who use it as a sole point of contact or distribution. Let that sink in for a moment. Breaking this cycle will require hard work.

pif 2y ago

What has humankind lost with those pages vanishing?

nicbou 2y ago
In some cases, a lot of valuable information that doesn't exist anywhere else. A big German immigration forum vanished this winter. There was a lot of valuable information for people navigating tricky bureaucratic processes.

gregoriol 2y ago

How many % of brick and mortar shops from 2013 are still in business today? How many % of people who were alive in 2013 are still alive today?

Kye 2y ago
Let's preserve those, too. There was a really nice music school/venue/coffee shop in the town I used to live in. It shut down, and they took their Facebook and Instagram pages down. The only evidence it ever existed is in posts announcing events on other Facebook pages, in memories of people who went there, and on still surviving business listings that are likely to go away.

deleted 2y ago

[deleted]

frankthepickle 2y ago

are we saying we're going to keep making hard drives so we can save everything ever produced? I see the value in many things but I worry about the load of expecting everything to be saved forever.

Ferret7446 2y ago

A natural consequence of federation. You could attempt to archive everything and host it. But ultimately, each host is responsible for what it hosts and keeping itself available.

detourdog 2y ago
That is what I like about federation. All the incentives are individualized. I think the scale that was needed for business in the last century has disappeared. Businesses previous needed scale and reach but for an individual to make a decent living in a niche is very easy today.

robertlagrant 2y ago

> It should be a standard for the open internet, that any respectable page has an "export complete archives" as a clickable button. (But it's mainly the opposite, today: adtech corporations don't want your written works to be free, they want them incarcerated in their revenue-generating walled gardens—and if ten billion human's worth of written history gets erased at the end, well, too bad for them!) [0] Can't you export your data from social media platforms? I haven't tried it, but did Google this for Facebook[1] and it looks as though you can? [0] a now-deleted comment [1] https://www.techlicious.com/how-to/how-to-download-all-your-facebook-data https://www.techlicious.com/how-to/how-to-download-all-your-...

MattGaiser 2y ago
I’ve exported my data on Facebook before.

throwaway290 2y ago

This will get only worse as people realize that almost anything published is fed into ML products and used against them.

barfbagginus 2y ago

I saw meme beams glitter past Ebaums Gate on the arm of Orion. These moments will be forgotten, like Cheezburgers in the rain. Prepare to die.

Brosper 2y ago

It's like with restaurants. A lot of them disappeared after the pandemic. Web hosting is not free.

rchaud 2y ago
Majority of Wordpress.com and Blogger.com sites are on the free tier.
- Brosper 2y ago
  OK, so domains are not free. But many businesses had their own hosting and domain.

onion2k 2y ago

I'm surprised it's not more. 2013 was long after the days of hobbyist websites of the early net, and into the time when most new sites were business driven. Given how long businesses last I'd expect many more sites to be long gone 11 years later. I guess maybe the death of a lot of community-building spaces (angelfire, Geocities, etc) probably counts for a lot of them going. What would be particularly interesting would be to graph how long websites last for. I suspect quite a lot of the content from the early days is still around, and this period (2008 - 2018) is the peak of sites vanishing.

rchaud 2y ago
A lot of the content from the early days was on platforms which are long dead: - Geocities - University-provided FTP folder (deleted after you graduate) - ISP-provided FTP folder (all those Earthlink, Juno, Comcast sites: probably deleted)

Springtime 2y ago

The predominant sentiment I'm seeing in comments here is a 'what does it matter?' While at the same time in discussions here about search engine result quality inevitably the most popular comments express a decline in quality (influenced by SEO, AI, spam, among other things), along with a desire for surfacing interesting, human made content that gets buried by modern search algorithms. On HN we see every day interesting, first page content marked with a title representing its year, sometimes dating back a decade or more. Its age isn't apparently a detracting aspect to this audience if the content is still worth sharing. And from the article the headline figure also doesn't represent irrelevant/undesirable content either but Wikipedia references, news articles, government pages, along with less unexpectedly ephemeral things like Twitter posts. We're lucky there is archive.org but since it's not indexed like a regular search engine the only tether to old pages are still-live links found via regular search engines/sites (including HN). Essentially unless sites continue to exist that contain links to archived content the chances of future discovery becomes slim. My stance is if you find something interesting that you expect is worthwhile sharing try saving it in the most convenient single file page format available to you (MHTML, SingleFile, PDF), to have your own copy. For MHTML at least it also saves the original page URL in its metadata. Saving to online archives is also great but admittedly higher friction (and can sometimes result in things like IP restrictions on archive.org even when saving just a handful of pages in a row, ime).

Kye 2y ago
The neat thing about mhtml is it's plain old MIME. You can change the extension to eml and open it in email clients.

amanzi 2y ago

Some of the better websites at least make an effort to archive old content. e.g. here's CNN and BBC websites with coverage from the 9/11 attacks: http://news.bbc.co.uk/hi/english/static/in_depth/americas/2001/day_of_terror/ http://news.bbc.co.uk/hi/english/static/in_depth/americas/20... http://edition.cnn.com/SPECIALS/2001/trade.center/index.html http://edition.cnn.com/SPECIALS/2001/trade.center/index.html Don't expect many of the links to work properly, but it's still interesting to see what the web used to look like.

mhh__ 2y ago
Some of the interactive stuff on old BBC election coverages still almost work to this day. Hard to imagine that with many sites now 20 years on. It's not even that it;s impossible with the technology, it's probably closer to how writing got worse after the invention of the word processor. Every thing is managed and structured now so the freedom / bubble needed to make things good in a way that can't be easily explained is gone.

eigenvalue 2y ago

I view this as a serious failing of the internet that we collectively should have done a better job of avoiding. In most cases I believe the content itself is in fact still available somewhere and it’s simply the link that broke. Some kind of two layer system like the DOI system used for libraries would be helpful for cases like that: https://nuim.libguides.com/referencing/DigitalObjectIdentifiers https://nuim.libguides.com/referencing/DigitalObjectIdentifi... But it would need to handled automatically to maintain the full usefulness and convenience of URLs. Not sure how that could work though.

shrubble 2y ago

Not surprising - even HP's support site is full of dead /broken links, you can't even locate information on 2013 era desktops and servers.

zargath 2y ago

As many have stated, I would assume more than 38%. But good quality content is rare and the dynamic content made page combinations go infinite 20 years ago. We are maintaining 25y old urls is a bit ehh. cumbersome, and I sometime wonder if its worth it. Most of the traffic seem to come from bots and they do seem to learn some of the 301's. It seems to be good for SEO, etc. Some users also gets redirected to the content on the new urls. It feels a bit like helping an elderly person over the street to where the shop is. Anyway, I hope that bots and humans trust our services more.

ekianjo 2y ago

> Nearly one-in-five tweets are no longer publicly visible on the site just months after being posted. In 60% of these cases, the account that originally posted the tweet was made private, suspended or deleted entirely. In the other 40%, the account holder deleted the individual tweet, but the account itself still existed this seems extraordinarily high. since they cant do that estimation on all tweets ever sent it must be a bias of their sample

ksec 2y ago

One of the reasons why Bookmarks are useless. Most of them are dead, and those that are not can not easily be searched within Google. The web we have today is just worst than pre-Google dominant era.

axegon_ 2y ago

Honestly, seeing startups die and people abandoning their blogs(of which I'm also guilty, despite my best intentions, I just happen to pay my domains for 10 years at a time), I would have expected the number to be much higher.

earthboundkid 2y ago

I have run a news website since 2019. Every hour, I have a crawler look for dead links. I replace about one link a day with a link to archive.org. The funniest ones are the day after an election when all the candidate websites go blank. The saddest are the government websites that go offline from 3am to 5am every week.

notRobot 2y ago
Interesting, does your crawler check every link every hour, or does it go through them a batch at a time?
- earthboundkid 2y ago
  Every link every hour. https://github.com/spotlightpa/linkrot https://github.com/spotlightpa/linkrot

zokier 2y ago

Everything on internet is intrisically ephemeral. Embrace that instead of fighting against it. If you want to archive stuff then make offline copies. PDF/A (especially the -1 and -2 versions) is format explicitly designed for archiving and works well for static content. I think it is bit of a shame that mirroring is not more readily built into web stack (=http/html); if you could trivially make links that included local copy (as fallback?) this linkrot would be far lesser concern. The way how for example wikipedia links everything through archive.org is bit of a hack imho

PaulStatezny 2y ago

This is a good thing. In an age of information overload, an Internet that never contracts sounds overwhelming.

iAkashPaul 2y ago

Not being able to renew domains beyond 10yrs could also be a factor

runamuck 2y ago

I post my site content Markdown to an open Git repo for this reason. Anyone can pull and build my pages. I think Git should stay for at least another 100 years. https://github.com/hatdropper1977/john.soban.ski https://github.com/hatdropper1977/john.soban.ski

lagniappe 2y ago

I hope not all things last forever. A while back I stumbled upon my first .com, from the 90s, which was hosted on Angelfire and dutifully rehosted by archive.org and it went about how you'd imagine. Despite being in 4th grade when my little friend and I made the webpage, things on there (while fine for the era) are just not okay by today's standards even if I understand the context for what led to it being there. It was nothing terrible, but just distasteful in a blissfully unaware way a 4th grader in the 90's would be. I realize that stuff will probably never be off my conscience and I just have to deal with it and hope nobody sees it.

otachack 2y ago
I have similar material. If it's reassuring, we all were just kids/teens and learning of the world. I feel a lot for the youth after us that made the Internet more accessible and, at times, more permanent.
BirAdam 2y ago
I feel your pain. Thankfully, even the archive occasionally takes stuff off.

superkuh 2y ago

This is because the majority of webpages are on commercial (for profit) sites now and for-profit companies do not build anything that lasts. Part of this is their use case requirements: CA TLS has to be the only way to access the sites. And since CA TLS is extremely fragile and short lived, so are the sites. But additionally, any dynamic site also has a short lifetime. HTML files-in-folders sites will last till the heat death of the universe unmaintained. A php or nodejs/etc site will last a year or two after it stops being mantained. The core of the web pages on the internet are still there. It's just that the thick layer of commercial cr'app' websites built up on top are transient.

nimajneb 2y ago

I've been thinking about this for a couple years. With our (recent) history becoming more and more digital we are losing more of our history. There's lot of creativity of all forms being digital and online only lasting only as long as the creator supports it. No longer are we viewing physical pieces art, but ephemeral pieces art. (I'm including websites made for personal use, by hobbyists, etc). I know Archive.org archives a lot, but they can't archive everything, especially all the small personal websites.

BirAdam 2y ago
Not just personal sites, but even news sites. Storage is expensive. Site owners don’t want to hold stuff forever, and the Archive cannot afford to hold everything forever either.

b112 2y ago

If one looks at the graph, 20% of sites were dead in 3 years. This isn't a facebook thing, most facebook migration happened a decade ago. Instead this is companies closing, campaign websites shuttering, urls being changed, community events being over, and so on.

JoeAltmaier 2y ago

In 2013 there were maybe 630M websites. Today there are over a billion. So those 38% amount to 22% of today's internet Seems like a pretty good half-life!

bdcravens 2y ago

Not a problem. The web is a communication medium first and foremost, and part of the freedom of communication is the freedom to de-communicate.

epolanski 2y ago

I often look previous years posts when there is nothing I can find interesting for the day on HN. Go back few years already and most links are dead.

furyofantares 2y ago

While I am aware that nothing is permanent and everything takes maintenance, I sure feel like there ought to be a bit _more_ permanent way to publish things. I run a few daily word games. They're static sites that could continue forever as long as they have players, but sometimes I think about what would happen if I died or stopped paying attention. Domains would expire. Maybe some people could still play cached versions, and archive site versions would still exist. I host them on github pages so if I exposed that url then that could probably exist for as long as github does - or breaks something that requires the owner to click a button to fix or something. I could probably publish a free desktop version on various stores, and that might last as long as the store, or OS upgrades break it. It would be great if there was a way to publish something such that it exists as long as anyone cares about it.

mikepurvis 2y ago
I bet GitHub pages is more permanent than the iOS store ecosystem.

deleted 2y ago

[deleted]

markx2 2y ago

I blogged from 2000 onward but hit my stride with Wordpress in 2004. I posted thousands of times. I had a Google PR on 8 at one point but it tended to be 6/7. In 2020 I decided to shutter the site. The database with the content is still there, it's just inaccessible. Google, Bing, Archive.org and others I contacted and had them remove my content. Then I removed everything visible. Very very happy I did, especially with the rise of AI.

renegat0x0 2y ago

Attention is limited. We cannot see everything on the Internet. We do not have enough time for that. There is a lot of valuable and interesting data on the Internet, but it is not visible. Certainly high quality, low profile blog that ended its development in 2015 will not be ranked high in Google. Media platform, search engines monetize content. YouTube channels need to churn new content every week or so to stay relevant and to stay watchable. Our society produces content, not quality, not products. SEO can be gamed, it is impossible to create objective index of valuable content. Bad actors will hack the game, spam results, destroy quality to gain profit. Google search engine most often connects users with media sites, with news sites, with the middle men. The more often not connect users with product directly. Write "search engine" in search query, you may not only find search a "search engines" but articles about "Best search engines in 2024", or "best SEO tricks to boost your page". Google does not have any incentive to fix this. Search engines are dead tech. It will be replaced by chatbots in a few years. People will not search for content, content will be generated at wish. Some time ago I have created my own domain repository with domain names: https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database I wanted to find "wargames" related pages. It is quite impossible to find anything interesting concerning warhammer on the normie internet (not Facebook). The second thing is I cannot find anything "amiga" related. This solved this my initial problem. I have also found out that many interesting pages are gone. I think that Google directing our attention toward "content" broke good quality pages. Right now I am using less and less google, because I use more and more my bookmark manager. https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive My solutions may not be as complex as common crawl, but they are enough for me. For now. I am still working on my program. It has been fun and interesting experience for me, and I learned a lot. About open graph protocol, about schema, about web scraping, etc. etc. Maybe this will inspire people to be more self sufficient, and more self-hostable. In times of walled gardens we need more standard, and more open data to keep what remains of the old wild west of the Internet.

EasyMark 2y ago

That's a much lower number than I would have suspected after 10+ years.

benwillies 2y ago

I remember people saying that once it is on the web it is there forever. But my experience is how fast web sites disappear. Or get redirected to a nonsensical site in the orient. I think maintaining the domain name is the big problem. However I am proud to say that you can still see my very first published web page from 1995 if you know the rather obscure url... http://admin.benwillies.com/ticker/ http://admin.benwillies.com/ticker/ I wrote this page as a proof of concept for a friend of mine who was a financial consultant but unfortunately the humor was a turn-off instead of getting him excited about this new fangled thing called the Web. I never made that mistake again.

mediumsmart 2y ago

This is about single pages that have been deleted in functional websites. How dare you?