4 ms·

Anyone archiving every website they visit? I often want to search for something I read recently, which is awkward to do if you don't remember where you read it.

Anyone archiving every website they visit? I often want to search for something I read recently, which is awkward to do if you don't remember where you read it. I'd like to store everything I visit with my browser once the page is loaded, but haven't found any nice solutions for this.

iansinnott 3y ago
A bit of a late response, but yes I've been storing full text of every website I visit and it's excellent for finding stuff again. The idea is to index pages as you visit them using a browser extension, thus avoiding all the pitfalls of being treated like a bot. Here's the project: https://github.com/iansinnott/full-text-tabs-forever https://github.com/iansinnott/full-text-tabs-forever
daniel31x13 3y ago
Created Linkwarden exactly for this: https://linkwarden.app https://linkwarden.app
- beeboobaa 3y ago
  Nice! This is very close to what I'm hoping to find. My main issue is that it seems the pages are fetched from the linkwarden server, instead of preserved by my browser extension. For example if I use the browser extension to save a HN page, the saved version is an unauthenticated view of the page instead of showing my username. If the pages were preserved directly from my browser now additional requests to the server would be required, and it would be realistic to archive everything automatically, instead of only when I press the button.
autocole 3y ago
I'm using the Forethink web extension for that. It keeps a local index of every page I visit for searching https://forethink.ai https://forethink.ai
112233 3y ago
was that not literally a built-in feature of Opera 12?
bandie91 3y ago
YaCY (self-hosted search engine + index) can do this when using it as a proxy. https://yacy.net/ https://yacy.net/
- jeroenhd 3y ago
  Interesting! How well does this proxy deal with bot detection (Cloudflare and such)? Most proxies I've used, usually without archiving capabilities, have me fill out CAPTCHAs for every other site.
robjwells 3y ago
On MacOS HistoryHound gives you a searchable index of webpage content, though it doesn't store the full contents of pages in a way that can be viewed offline. https://www.stclairsoft.com/HistoryHound/ https://www.stclairsoft.com/HistoryHound/

iansinnott 3y ago

A bit of a late response, but yes I've been storing full text of every website I visit and it's excellent for finding stuff again. The idea is to index pages as you visit them using a browser extension, thus avoiding all the pitfalls of being treated like a bot. Here's the project: https://github.com/iansinnott/full-text-tabs-forever https://github.com/iansinnott/full-text-tabs-forever

daniel31x13 3y ago

Created Linkwarden exactly for this: https://linkwarden.app https://linkwarden.app

beeboobaa 3y ago
Nice! This is very close to what I'm hoping to find. My main issue is that it seems the pages are fetched from the linkwarden server, instead of preserved by my browser extension. For example if I use the browser extension to save a HN page, the saved version is an unauthenticated view of the page instead of showing my username. If the pages were preserved directly from my browser now additional requests to the server would be required, and it would be realistic to archive everything automatically, instead of only when I press the button.

autocole 3y ago

I'm using the Forethink web extension for that. It keeps a local index of every page I visit for searching https://forethink.ai https://forethink.ai

112233 3y ago

was that not literally a built-in feature of Opera 12?

bandie91 3y ago

YaCY (self-hosted search engine + index) can do this when using it as a proxy. https://yacy.net/ https://yacy.net/

jeroenhd 3y ago
Interesting! How well does this proxy deal with bot detection (Cloudflare and such)? Most proxies I've used, usually without archiving capabilities, have me fill out CAPTCHAs for every other site.

robjwells 3y ago

On MacOS HistoryHound gives you a searchable index of webpage content, though it doesn't store the full contents of pages in a way that can be viewed offline. https://www.stclairsoft.com/HistoryHound/ https://www.stclairsoft.com/HistoryHound/