6 ms·

Before 2010 I had this unquestioned assumption that links are supposed to last forever. I used the bookmark feature of my browser extensively. Some time afterwa

by kccqzy 1y ago

Before 2010 I had this unquestioned assumption that links are supposed to last forever. I used the bookmark feature of my browser extensively. Some time afterwards, I discovered that a large fraction of my bookmarks were essentially unusable due to linkrot. My modus operandi after that was to print the webpage as a PDF. A bit afterwards when reader views became popular reliable, I just copy-pasted the content from the reader view into an RTF file.

flexagoon 1y ago
By the way, if you install the official Web Archive browser extension, you can configure it to automatically archive every page you visit
- petethomas 1y ago
  This a good suggestion with the caveat that entire domains can and do disappear: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/ https://help.archive.org/help/how-do-i-request-to-remove-som...
  - Akronymus 1y ago
    That's especially annoying when a formerly useful site gets abandoned, a new owner picks up the domain, then gets IA to delete the old archives as well. Or even worse, when a domain parking company does that: https://archive.org/post/423432/domainsponsorcom-erasing-prior-archived-copies-of-135000-domains https://archive.org/post/423432/domainsponsorcom-erasing-pri...
- vitorsr 1y ago
  > you can configure it to automatically archive every page you visit What?? I am a heavy user of the Internet Archive services, not just the Wayback Machine, including official and "unofficial" clients and endpoints, and I had absolutely no idea the extension could do this. To bulk archive I would manually do it via the web interface or batch automate it. The limitations of manually doing it one by one are obvious, and the limitations of doing it in batches requires, well, keeping batches (lists).
- internetter 1y ago
  recently I've come to believe even IA and especially archive.is are ephermal. I've watched sites I've saved disappear without a trace, except in my selfhosted archives. A technological conundrum, however, is the fact that I have no way to prove that my archive is an accurate representation of a site at a point in time. Hmmm, or maybe I do? Maybe something funky with cert chains could be done.
  - akoboldfrying 1y ago
    There are timestamping services out there, some of which may be free. It should (I think) be possible to basically submit the target site's URL to the timestamping service, and get back a certificate saying "I, Timestamps-R-US, assert that the contents of https://targetsite.com/foo/bar https://targetsite.com/foo/bar downloaded at 12:34pm on 29/5/2025 hashes to abc12345 with SHA-1", signed with their private key and verifiable (by anyone) with their public key. Then you download the same URL, and check that the hashes match. IIUC the timestamping service needs to independently download the contents itself in order to hash it, so if you need to be logged in to see the content there might be complications, and if there's a lot of content they'll probably want to charge you.
    - XorNot 1y ago
      Websites don't really produce consistent content even from identical requests though. But you also don't need to do this: all you need is a service which will attest that it saw a particular hashsum at a particular time. It's up to other mechanisms to prove what that means.
      - akoboldfrying 1y ago
        > Websites don't really produce consistent content even from identical requests though. Often true in practice unfortunately, but to the extent that it is true, any approach that tries to use hashes to prove things to a third party is sunk. (We could imagine a timestamping service that allows some kind of post-download "normalisation" step to strip out content that varies between queries and then hash the results of that, but that doesn't seem practical to offer as a free service.) > all you need is a service which will attest that it saw a particular hashsum at a particular time Isn't that what I'm proposing?
90s_dev 1y ago
My solution has been to just remember the important stuff, or at least where to find it. I'm not dead yet so I guess it works.
- TeMPOraL 1y ago
  It was my solution too, and I liked it, but over the past decade or so, I noticed that even when I remember where to find some stuff, hell, even if I just remember how to find it, when I actually try and find it, it often isn't there anymore. "Search rot" is just as big a problem as link rot. As for being still alive, by that measure hardly anything anyone does is important in the modern world. It's pretty hard to fail at thinking or remembering so badly that it becomes a life-or-death thing.
  - 90s_dev 1y ago
    > hardly anything anyone does is important Agreed.
- mock-possum 1y ago
  I’ve found that whenever I think “why don’t other people just do X” it’s because I’m misunderstanding what’s involved in X for them, and that generally if they could ‘just’ do X then they would. “Why don’t you just” is a red flag now for me.
  - chii 1y ago
    this applies to basically any suggested solution to any problem. "Why don't you just ..." is just lazy idea suggestion from armchair internet warriors.
  - 90s_dev 1y ago
    Not always. I love it when people offer me a much simpler solution to a problem I overengineered, so I can throw away my solution and use the simpler one. Half the time people are suggested a better way, it's because they're actually doing it wrong, they've gotten the solution's requirements all wrong in the first place, and this perspective helps.
nonethewiser 1y ago
A reference is a bet on continuity. At a fundamental level, broken website links and dangling pointers in C are the same.
lappa 1y ago
I use the SingleFile extension to archive every page I visit. It's easy to set up, but be warned, it takes up a lot of disk space. $ du -h ~/archive/webpages 1.1T /home/andrew/archive/webpages https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile
- davidcollantes 1y ago
  How do you manage those? Do you have a way to search them, or a specific way to catalogue them, which will make it easy to find exactly what you need from them?
  - nirav72 1y ago
    KaraKeep is a decent self hostable app that has support for receiving singlefile pages via singlefile browser extension and pointing to karakeep API. This allows me to search for archived pages. (Plus auto summarization and tagging via LLM).
    - dotancohen 1y ago
      Very naive question, surely. What does KaraKeep provide that grep doesn't?
      - nirav72 1y ago
        jokes aside. It has a mobile app
        dotancohen 1y ago
        I don't get it aside. How does that help him search files on his local file system? Or is he syncing an index of his entire web history to his mobile device?
        nirav72 1y ago
blurbleblurble 1y ago
Use WARC: https://en.wikipedia.org/wiki/WARC_(file_format) https://en.wikipedia.org/wiki/WARC_(file_format) with WebRecorder: https://webrecorder.net/ https://webrecorder.net/
- shwouchk 1y ago
  warc is not a panacea; for example, gemini makes it super annoying to get a transcript of your conversation, so i started saving those as pdf and warc. turns out that unlike most webpages, the pdf version is only a single page of what is visible on screen. turns out also that opening the warc immediately triggers a js redirect that is planted in the page. i can still extract the text manually - it’s embedded there - but i cannot “just open” the warc in my browser and expect an offline “archive” version - im interacting with a live webpage! this sucks from all sides - usability, privacy, security. Admittedly, i don’t use webrecorder - does it solve this problem? did you verify?
  - weinzierl 1y ago
    Not sure if you tried that. Chrome has a take full page screenshot command. Just open the command bar in dev tools and search for "full" and you will fund it. Firefox has it right in the context menu, no need for dev tools. Unfortunately there are sites where it does not work.
    - eMPee584 1y ago
      Apart from small UX nits, FF's screenshot feature is great - it's just that storing a 2-15MiB bitmap copy of a text medium still feels dirty to me every time.. would much prefer a PDF export, page size matching the scroll port, with embedded fonts and vectors and without print CSS..
andai 1y ago
Is there some kind of thing that turns a web page into a text file? I know you can do it with beautiful soup (or like 4 lines of python stdlib), but I usually need it on my phone, where I don't know a good option. My phone browser has a "reader view" popup but it only appears sometimes, and usually not on pages that need it! Edit: Just installed w3m in Termux... the things we can do nowadays!
- XorNot 1y ago
  You want Zotero. It's for bibliographies, but it also archives and stores web pages locally with a browser integration.
  - _huayra_ 1y ago
    I frankly don't know how I'd collect any useful info without it. I'm sure there are bookmark services that also allow notes, but the tagging, linking related things, etc, all in the app is awesome, plus the ability to export bib tex for writing a paper!
taeric 1y ago
That assumption isn't true of any sources? Things flat out change. Some literally, others more in meaning. Some because they are corrected, but there are other reasons. Not that I don't think there is some benefit in what you are attempting, of course. A similar thing I still wish I could do is to "archive" someone's phone number from my contact list. Be it a number that used to be ours, or family/friends that have passed.
rubit_xxx16 1y ago
> Before 2010 I had this unquestioned assumption that links are supposed to last forever Any site/company whatsoever of this world (and most) that promises that anything will last forever is seriously deluded or intentionally lying, unless their theory of time is different than that of the majority.
mycall 1y ago
Is there some browser extension that automatically goes to web.archive.org if the link timesout?
- theblazehen 1y ago
  I use the Resurrect Pages addon
jwe 1y ago
I can recommend to use Pinboard with the archive option
account42 1y ago
I really is a travesty that Browsers still haven't updated their bookmark feature based on this realization - all bookmarks should store not only the link but a full copy of the rendered page (not just the source which could rely on dynamic content that will no longer be available). Also, open tabs should work the same way: I never want to see a network error while going back to a tab while not having an internet connection because the browser has helpfully evicted that tab from memory. It should just reload the state from disk instead of the network in this case until I manually refresh the page.
m-p-3 1y ago
I export text-based content I want to retain into Markdown files, and when I find something useful for work I also send the URL to the Wayback Machine.

by kccqzy 1y ago

flexagoon 1y ago

By the way, if you install the official Web Archive browser extension, you can configure it to automatically archive every page you visit

petethomas 1y ago
This a good suggestion with the caveat that entire domains can and do disappear: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/ https://help.archive.org/help/how-do-i-request-to-remove-som...
- Akronymus 1y ago
  That's especially annoying when a formerly useful site gets abandoned, a new owner picks up the domain, then gets IA to delete the old archives as well. Or even worse, when a domain parking company does that: https://archive.org/post/423432/domainsponsorcom-erasing-prior-archived-copies-of-135000-domains https://archive.org/post/423432/domainsponsorcom-erasing-pri...
vitorsr 1y ago
> you can configure it to automatically archive every page you visit What?? I am a heavy user of the Internet Archive services, not just the Wayback Machine, including official and "unofficial" clients and endpoints, and I had absolutely no idea the extension could do this. To bulk archive I would manually do it via the web interface or batch automate it. The limitations of manually doing it one by one are obvious, and the limitations of doing it in batches requires, well, keeping batches (lists).
internetter 1y ago
recently I've come to believe even IA and especially archive.is are ephermal. I've watched sites I've saved disappear without a trace, except in my selfhosted archives. A technological conundrum, however, is the fact that I have no way to prove that my archive is an accurate representation of a site at a point in time. Hmmm, or maybe I do? Maybe something funky with cert chains could be done.
- akoboldfrying 1y ago
  There are timestamping services out there, some of which may be free. It should (I think) be possible to basically submit the target site's URL to the timestamping service, and get back a certificate saying "I, Timestamps-R-US, assert that the contents of https://targetsite.com/foo/bar https://targetsite.com/foo/bar downloaded at 12:34pm on 29/5/2025 hashes to abc12345 with SHA-1", signed with their private key and verifiable (by anyone) with their public key. Then you download the same URL, and check that the hashes match. IIUC the timestamping service needs to independently download the contents itself in order to hash it, so if you need to be logged in to see the content there might be complications, and if there's a lot of content they'll probably want to charge you.
  - XorNot 1y ago
    Websites don't really produce consistent content even from identical requests though. But you also don't need to do this: all you need is a service which will attest that it saw a particular hashsum at a particular time. It's up to other mechanisms to prove what that means.
    - akoboldfrying 1y ago
      > Websites don't really produce consistent content even from identical requests though. Often true in practice unfortunately, but to the extent that it is true, any approach that tries to use hashes to prove things to a third party is sunk. (We could imagine a timestamping service that allows some kind of post-download "normalisation" step to strip out content that varies between queries and then hash the results of that, but that doesn't seem practical to offer as a free service.) > all you need is a service which will attest that it saw a particular hashsum at a particular time Isn't that what I'm proposing?

90s_dev 1y ago

My solution has been to just remember the important stuff, or at least where to find it. I'm not dead yet so I guess it works.

TeMPOraL 1y ago
It was my solution too, and I liked it, but over the past decade or so, I noticed that even when I remember where to find some stuff, hell, even if I just remember how to find it, when I actually try and find it, it often isn't there anymore. "Search rot" is just as big a problem as link rot. As for being still alive, by that measure hardly anything anyone does is important in the modern world. It's pretty hard to fail at thinking or remembering so badly that it becomes a life-or-death thing.
- 90s_dev 1y ago
  > hardly anything anyone does is important Agreed.
mock-possum 1y ago
I’ve found that whenever I think “why don’t other people just do X” it’s because I’m misunderstanding what’s involved in X for them, and that generally if they could ‘just’ do X then they would. “Why don’t you just” is a red flag now for me.
- chii 1y ago
  this applies to basically any suggested solution to any problem. "Why don't you just ..." is just lazy idea suggestion from armchair internet warriors.
- 90s_dev 1y ago
  Not always. I love it when people offer me a much simpler solution to a problem I overengineered, so I can throw away my solution and use the simpler one. Half the time people are suggested a better way, it's because they're actually doing it wrong, they've gotten the solution's requirements all wrong in the first place, and this perspective helps.

nonethewiser 1y ago

A reference is a bet on continuity. At a fundamental level, broken website links and dangling pointers in C are the same.

lappa 1y ago

I use the SingleFile extension to archive every page I visit. It's easy to set up, but be warned, it takes up a lot of disk space. $ du -h ~/archive/webpages 1.1T /home/andrew/archive/webpages https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile

davidcollantes 1y ago
How do you manage those? Do you have a way to search them, or a specific way to catalogue them, which will make it easy to find exactly what you need from them?
- nirav72 1y ago
  KaraKeep is a decent self hostable app that has support for receiving singlefile pages via singlefile browser extension and pointing to karakeep API. This allows me to search for archived pages. (Plus auto summarization and tagging via LLM).
  - dotancohen 1y ago
    Very naive question, surely. What does KaraKeep provide that grep doesn't?
    - nirav72 1y ago
      jokes aside. It has a mobile app
      - dotancohen 1y ago
        I don't get it aside. How does that help him search files on his local file system? Or is he syncing an index of his entire web history to his mobile device?
        nirav72 1y ago

blurbleblurble 1y ago

Use WARC: https://en.wikipedia.org/wiki/WARC_(file_format) https://en.wikipedia.org/wiki/WARC_(file_format) with WebRecorder: https://webrecorder.net/ https://webrecorder.net/

shwouchk 1y ago
warc is not a panacea; for example, gemini makes it super annoying to get a transcript of your conversation, so i started saving those as pdf and warc. turns out that unlike most webpages, the pdf version is only a single page of what is visible on screen. turns out also that opening the warc immediately triggers a js redirect that is planted in the page. i can still extract the text manually - it’s embedded there - but i cannot “just open” the warc in my browser and expect an offline “archive” version - im interacting with a live webpage! this sucks from all sides - usability, privacy, security. Admittedly, i don’t use webrecorder - does it solve this problem? did you verify?
- weinzierl 1y ago
  Not sure if you tried that. Chrome has a take full page screenshot command. Just open the command bar in dev tools and search for "full" and you will fund it. Firefox has it right in the context menu, no need for dev tools. Unfortunately there are sites where it does not work.
  - eMPee584 1y ago
    Apart from small UX nits, FF's screenshot feature is great - it's just that storing a 2-15MiB bitmap copy of a text medium still feels dirty to me every time.. would much prefer a PDF export, page size matching the scroll port, with embedded fonts and vectors and without print CSS..

andai 1y ago

Is there some kind of thing that turns a web page into a text file? I know you can do it with beautiful soup (or like 4 lines of python stdlib), but I usually need it on my phone, where I don't know a good option. My phone browser has a "reader view" popup but it only appears sometimes, and usually not on pages that need it! Edit: Just installed w3m in Termux... the things we can do nowadays!

XorNot 1y ago
You want Zotero. It's for bibliographies, but it also archives and stores web pages locally with a browser integration.
- _huayra_ 1y ago
  I frankly don't know how I'd collect any useful info without it. I'm sure there are bookmark services that also allow notes, but the tagging, linking related things, etc, all in the app is awesome, plus the ability to export bib tex for writing a paper!

taeric 1y ago

That assumption isn't true of any sources? Things flat out change. Some literally, others more in meaning. Some because they are corrected, but there are other reasons. Not that I don't think there is some benefit in what you are attempting, of course. A similar thing I still wish I could do is to "archive" someone's phone number from my contact list. Be it a number that used to be ours, or family/friends that have passed.

rubit_xxx16 1y ago

> Before 2010 I had this unquestioned assumption that links are supposed to last forever Any site/company whatsoever of this world (and most) that promises that anything will last forever is seriously deluded or intentionally lying, unless their theory of time is different than that of the majority.

mycall 1y ago

Is there some browser extension that automatically goes to web.archive.org if the link timesout?

theblazehen 1y ago
I use the Resurrect Pages addon

jwe 1y ago

I can recommend to use Pinboard with the archive option

account42 1y ago

I really is a travesty that Browsers still haven't updated their bookmark feature based on this realization - all bookmarks should store not only the link but a full copy of the rendered page (not just the source which could rely on dynamic content that will no longer be available). Also, open tabs should work the same way: I never want to see a network error while going back to a tab while not having an internet connection because the browser has helpfully evicted that tab from memory. It should just reload the state from disk instead of the network in this case until I manually refresh the page.

m-p-3 1y ago

I export text-based content I want to retain into Markdown files, and when I find something useful for work I also send the URL to the Wayback Machine.