6 ms·

Some days, nearly half the links I click are dead so I've found myself relying on the waybackmachine more and more over the past few months. It's really shockin

by veqq 4y ago

Some days, nearly half the links I click are dead so I've found myself relying on the waybackmachine more and more over the past few months. It's really shocking just how fast digital obsolescence reared its ugly head. Of course angelcites etc. were a clear early blow, but nowadays... I've started saving the html (including the css seems like too much overhead, and often it's incomplete or relies on downloads still - and screenshots are not searchable) of every interesting article I find online, downloading quite a few videos too with yt-dlp. I'd long copypasted all interesting comments into a txt file, but now it seems like data hoarding's the way to go - at least in moderation, focusing on things I'll actually refer back to. I remember 15 years ago, discovering pdf dumps on random sites like a kid in a candy store. Perhaps it'll be like that again, with people presenting museums of their favorite old pages.

dspillett 4y ago
> I've started saving I do similar. I've had https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox bookmarked for a while as something to try better organise all that, but like a great many things I haven't go around to it yet.
- circustaco 4y ago
  I use Raindrop for this. It’s a pretty great bookmark manager made by an indie dev, but it also can create archives of bookmarked pages. https://help.raindrop.io/backups#permanent-library https://help.raindrop.io/backups#permanent-library
  - patrickdward 4y ago
    I use Raindrop but didn't know abou that feature. Thanks.
  - dspillett 4y ago
    Thanks, I'll add that to the list of things to try out.
  - dspillett 4y ago
    “Only available in Pro plan”. No information on whether that is a general limitation or if these features are present if self-hosted, so I assume the former. And there seems to be little obvious information for self-hosting. So probably not one for my use case.
082349872349872 4y ago
While the author(s) are still alive, they are often a productive contact. (In one case I was able to give back: bundling up the several scans an author had of a half-century old paper from their student days into a single, hopefully cromulent, PDF) Edit: recall also that accepting that links are one-way and might be dead was the key simplification that allowed HTTP to take off after prior attempts at hypermedia had failed.
- mrybczyn 4y ago
  Good (Edit) point! It's good that the web accepts dead links by design, we can't expect perfection from our distributed information, but it seems that the bitrot of information is too high compared to the information storage technologies available. 2 spinning rust drive can store the library of congress. ~ 2,000 drives would store the web (1). How many millions of these drives get manufactured per year? Our technology systems are failing us - all those words are being lost, like tears in rain. (1)Back of the envelope estimation: https://www.worldwidewebsize.com/ https://www.worldwidewebsize.com/ ~ estimates 50 billion websites, with some estimates ~ 6 pages of information per website. Let's say 1mb per page on average. So ~2,000 drives would store the entire web.
  - 8bitsrule 4y ago
    2000 drives x $100 = $200,000. Double that for backup, $400,000. Admin, maintenance, let's say total, $1M/year. So, Wikipedia could end its own deadlink problem (IF the reference sources would agree. But stuff goes missing at Wayback because people don't agree to their pages being backed-up. Copyright, whatever. So it's like Global Heating, the tech is there, but people just can't agree. So 'pirate' backer-uppers go to jail. And island-nations and expensive ocean-side properties are being submerged. So it goes.
    - toomuchtodo 4y ago
      The Internet Archive has a bot that updates dead Wikipedia references to point to archived content. https://meta.wikimedia.org/wiki/InternetArchiveBot https://meta.wikimedia.org/wiki/InternetArchiveBot
msravi 4y ago
I use the markdownload extension[1] on firefox and move the .md file into my notes folder (notable[2]). Works very well. 1. https://addons.mozilla.org/en-US/firefox/addon/markdownload/ https://addons.mozilla.org/en-US/firefox/addon/markdownload/ 2. https://notable.app/ https://notable.app/
leephillips 4y ago
I find the SingleFile extension superb for this.
graderjs 4y ago
Some people use this tool (of mine) for saving web content from either bookmarks or just everything you browse: https://github.com/crisdosyago/Diskernet https://github.com/crisdosyago/Diskernet There's also plent¥ of other similar tools: - https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox - https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile
- randomguy12 4y ago
  Zotero also saves snapshots of pages if you already cite academic pages
- zote 4y ago
  How have I never seen your tool before. >22120 archives content exactly as it is received and presented by a browser, and it also replays that content exactly as if the resource were being taken from online. I've been looking for this for a really long time.
- leslielurker 4y ago
  I'd never heard of SingleFile before but it looks excellent. It would be great if Firefox could incorporate it into its save function too. Firefox save page works but as shown on the SingleFile demo video, it's not really what a user would expect, it's often not complete and splitting it across multiple files/directories isn't ideal either.
  - Springtime 4y ago
    It's unfortunate as Firefox used to have excellent MHTML support (which similarly achieves an all-in-one file) via addons, particularly the feature-rich UnMHT. While Chromium and its derivatives support MHTML saving natively (and in the past Opera Presto and IE). If they brought back MHTML saving support it'd be a great win.
- tetris11 4y ago
  > Coming to a future release, soon!: The ability to publish your own search engine that you curated with the best resources based on your expert knowledge and experience. This would be fantastic, being able to browse a curated internet made of accumulated lists from other trusted users on the net, similar to how ad blocking lists work today. You are genuinely trying to steer the internet into what it used to be: a museum of knowledge and expert discussion. Edit: Ah but wow, Polyform license. Huh.
b1zguy 4y ago
I've been wanting to run my own search engine sorta thingy that indexes websites I feed it. I sometimes find little nooks of the net that post resources I may need in the future. Like my own mini-Google that indexes a list of sites. How can I go about creating this? Are their off-the-shelf solutions, will I need to say combine scrapy with elastic search? The links in this thread look promising.
gwbas1c 4y ago
I just save the pdf of a site that's really important to me. When I did a major college project in 2003, I made sure to make pdfs of any academic article that I referenced. It actually saved me, because some articles disappeared by the time I went to revise my references.
deleted 4y ago
[deleted]
RupertEisenhart 4y ago
I wanted to say 'au contrer' to your 'screenshots are not searchable' and link this[0] but I don't actually see images in the readme.. I swear it was there, maybe it's a buried extra flag.. [0] https://github.com/phiresky/ripgrep-all https://github.com/phiresky/ripgrep-all

by veqq 4y ago

dspillett 4y ago

> I've started saving I do similar. I've had https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox bookmarked for a while as something to try better organise all that, but like a great many things I haven't go around to it yet.

circustaco 4y ago
I use Raindrop for this. It’s a pretty great bookmark manager made by an indie dev, but it also can create archives of bookmarked pages. https://help.raindrop.io/backups#permanent-library https://help.raindrop.io/backups#permanent-library
- patrickdward 4y ago
  I use Raindrop but didn't know abou that feature. Thanks.
- dspillett 4y ago
  Thanks, I'll add that to the list of things to try out.
- dspillett 4y ago
  “Only available in Pro plan”. No information on whether that is a general limitation or if these features are present if self-hosted, so I assume the former. And there seems to be little obvious information for self-hosting. So probably not one for my use case.

082349872349872 4y ago

While the author(s) are still alive, they are often a productive contact. (In one case I was able to give back: bundling up the several scans an author had of a half-century old paper from their student days into a single, hopefully cromulent, PDF) Edit: recall also that accepting that links are one-way and might be dead was the key simplification that allowed HTTP to take off after prior attempts at hypermedia had failed.

mrybczyn 4y ago
Good (Edit) point! It's good that the web accepts dead links by design, we can't expect perfection from our distributed information, but it seems that the bitrot of information is too high compared to the information storage technologies available. 2 spinning rust drive can store the library of congress. ~ 2,000 drives would store the web (1). How many millions of these drives get manufactured per year? Our technology systems are failing us - all those words are being lost, like tears in rain. (1)Back of the envelope estimation: https://www.worldwidewebsize.com/ https://www.worldwidewebsize.com/ ~ estimates 50 billion websites, with some estimates ~ 6 pages of information per website. Let's say 1mb per page on average. So ~2,000 drives would store the entire web.
- 8bitsrule 4y ago
  2000 drives x $100 = $200,000. Double that for backup, $400,000. Admin, maintenance, let's say total, $1M/year. So, Wikipedia could end its own deadlink problem (IF the reference sources would agree. But stuff goes missing at Wayback because people don't agree to their pages being backed-up. Copyright, whatever. So it's like Global Heating, the tech is there, but people just can't agree. So 'pirate' backer-uppers go to jail. And island-nations and expensive ocean-side properties are being submerged. So it goes.
  - toomuchtodo 4y ago
    The Internet Archive has a bot that updates dead Wikipedia references to point to archived content. https://meta.wikimedia.org/wiki/InternetArchiveBot https://meta.wikimedia.org/wiki/InternetArchiveBot

msravi 4y ago

I use the markdownload extension[1] on firefox and move the .md file into my notes folder (notable[2]). Works very well. 1. https://addons.mozilla.org/en-US/firefox/addon/markdownload/ https://addons.mozilla.org/en-US/firefox/addon/markdownload/ 2. https://notable.app/ https://notable.app/

leephillips 4y ago

I find the SingleFile extension superb for this.

graderjs 4y ago

Some people use this tool (of mine) for saving web content from either bookmarks or just everything you browse: https://github.com/crisdosyago/Diskernet https://github.com/crisdosyago/Diskernet There's also plent¥ of other similar tools: - https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox - https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile

randomguy12 4y ago
Zotero also saves snapshots of pages if you already cite academic pages
zote 4y ago
How have I never seen your tool before. >22120 archives content exactly as it is received and presented by a browser, and it also replays that content exactly as if the resource were being taken from online. I've been looking for this for a really long time.
leslielurker 4y ago
I'd never heard of SingleFile before but it looks excellent. It would be great if Firefox could incorporate it into its save function too. Firefox save page works but as shown on the SingleFile demo video, it's not really what a user would expect, it's often not complete and splitting it across multiple files/directories isn't ideal either.
- Springtime 4y ago
  It's unfortunate as Firefox used to have excellent MHTML support (which similarly achieves an all-in-one file) via addons, particularly the feature-rich UnMHT. While Chromium and its derivatives support MHTML saving natively (and in the past Opera Presto and IE). If they brought back MHTML saving support it'd be a great win.
tetris11 4y ago
> Coming to a future release, soon!: The ability to publish your own search engine that you curated with the best resources based on your expert knowledge and experience. This would be fantastic, being able to browse a curated internet made of accumulated lists from other trusted users on the net, similar to how ad blocking lists work today. You are genuinely trying to steer the internet into what it used to be: a museum of knowledge and expert discussion. Edit: Ah but wow, Polyform license. Huh.

b1zguy 4y ago

I've been wanting to run my own search engine sorta thingy that indexes websites I feed it. I sometimes find little nooks of the net that post resources I may need in the future. Like my own mini-Google that indexes a list of sites. How can I go about creating this? Are their off-the-shelf solutions, will I need to say combine scrapy with elastic search? The links in this thread look promising.

gwbas1c 4y ago

I just save the pdf of a site that's really important to me. When I did a major college project in 2003, I made sure to make pdfs of any academic article that I referenced. It actually saved me, because some articles disappeared by the time I went to revise my references.

deleted 4y ago

[deleted]

RupertEisenhart 4y ago

I wanted to say 'au contrer' to your 'screenshots are not searchable' and link this[0] but I don't actually see images in the readme.. I swear it was there, maybe it's a buried extra flag.. [0] https://github.com/phiresky/ripgrep-all https://github.com/phiresky/ripgrep-all