5 ms·

I think many of us wish for and have considered developing an open source, local archive of the websites we visit. We want full text search, full page screensho

I think many of us wish for and have considered developing an open source, local archive of the websites we visit. We want full text search, full page screenshots, good privacy (dont archive sensitive sites). We might wish for p2p encrypted syncing between devices. Since we wish for this, I presume its been attempted many times. My questions is my humble hacker news readers? Can any one provide a run down of the existing open source solutions and their features/pros/cons or simply which they prefer?

bloodm 7y ago
You visit a website that you may find intersting. e.g. https://businessfinancing.co.uk/the-oldest-company-in-almost-every-country/ https://businessfinancing.co.uk/the-oldest-company-in-almost... You have folders from a-z in your data directory. You save the website in /data/o/oldestcompanies or into a deeper directory to your liking. Let recoll take care of the rest. https://www.lesbonscomptes.com/recoll/ https://www.lesbonscomptes.com/recoll/
- codewithcheese 7y ago
  Thanks for sharing your flow/idea. If understand correctly here are the PROS and CONS. PROS - full text search - privacy: save only what you want - save webpage has OK rendering - easy backup using backup software CONS - need to save manually and specify directory/file name - no screenshot. (no screenshot in search results) - save webpage not 100% accurate - requires additional backup solution
  - bloodm 7y ago
    Yes. Works for me. There will be always Pros and Cons. - need to save manually and specify directory/file name This may be also a PRO for me since I may be able to find it without using recoll. Also, how many websites do you need to save? I assume only a tiny fraction. If you want to save everything, recoll does offer this more or less out of the box: "Indexing visited WEB pages with the Recoll Firefox extension" https://www.lesbonscomptes.com/recoll/faqsandhowtos/IndexWebHistory.html https://www.lesbonscomptes.com/recoll/faqsandhowtos/IndexWeb... This is another stand alone software that may do what you are looking for. https://getpolarized.io/2019/04/11/Polar-Initial-Crowdfunding-Campaign.html https://getpolarized.io/2019/04/11/Polar-Initial-Crowdfundin...
MayeulC 7y ago
I saw someone recommend Memex as a Firefox extension for full-text search in history and bookmarks the other day. I've started using it, but can't yet comment on its usefulness. It stills feels a bit complex to share data between my computers (I wish for p2p, Nextcloud support, or something alike). I don't like too much it moving DDG's instant answers to the bottom of the page, nor the default sidebar and highlighter, but that could just take some time getting used to.
sails 7y ago
Yet to get around to using, but this looks like it may be good: https://github.com/pirate/ArchiveBox https://github.com/pirate/ArchiveBox
- codewithcheese 7y ago
  I see they also have a good listing of alternatives https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm...
cparsons3000 7y ago
I would recommend Bookmark OS. It's not open sources but offers full text search, full page screenshots, and other neat features https://bookmarkos.com https://bookmarkos.com
- codewithcheese 7y ago
  I've seen many services like this come and go. As great as some of them are, having to trust a 3rd party with a full text archive of your complete web history is not a good value proposition. Not only due to privacy but also longevity, I don't want my archive to depends on this company remaining honest while also becoming successful enough to be around 10-20 years.
kissgyorgy 7y ago
Since I found Wallabag, I did not search for anything else. It fundamentally changed my reading habits. Highly recommended! https://github.com/wallabag/wallabag https://github.com/wallabag/wallabag
butterthebuddha 7y ago
What I really want is a script that (1) Takes a URL and optional comment as input (2) Saves the webpage it points to into a git repo (a simple curl should suffice for most websites) (3) Inserts that URL, title of the page pointed-to by the URL and the optional comment into an org-mode file that lives in the root of the repo The org-mode file is a highly-searchable and context-preserving database (I can add tags, create hierarchies, add links to and from other relevant (org-mode or not) files) in the most portable format ever: plain text. I really don't need a web interface. Actually, if I later decide that I need one, I can build one easily on top of this basic system. I really want to be able to use this across multiple devices: mainly my two computers, and an Android phone. Using git gives me a reliable protocol for syncing between multiple devices. I want it to be a smooth experience on my phone, which would probably require some sort of git-aware app. Something similar to the Android client for the pass password manager would be ideal. I hear that git repos can be GPG-encrypted. Ideally, I'm able to serve all this off of a repo hosted on a VPS. I don't want to rely on Dropbox (I'm trying to transition away from it) for syncing.
- bachmeier 7y ago
  (1) Takes a URL and optional comment as input (2) Saves the webpage it points to into a git repo (a simple curl should suffice for most websites) (3) Inserts that URL, title of the page pointed-to by the URL and the optional comment into an org-mode file that lives in the root of the repo If you're willing to change "git" to "version control", it should be pretty easy to implement that in Fossil. It doesn't require much to add an extension written in your language of choice if you're going to run it on your desktop. Plus you'd get the web interface for free if you decided to put it on a web server.
- Karunamon 7y ago
  Hmmm.. this wouldn't be too hard to write and sounds like an interesting weekend project. Would you be interested in using WARC for the webpage though? This way, everything is captured in a single file and you aren't littering your repo with random files and images.
  - 7y ago
linguaz 7y ago
What I'd find to be interesting & useful is: Every URL I visit is automatically recorded along with the time & date visited, title and other metadata. If the URL is visited via a link off another webpage, have that relationship recorded. Provide some sort of navigable tree / searchable database. Should be able to easily scale to tens / hundreds of millions of URLs across decades. If the URL is visited by manually inputting that URL, provide option to type into a field something like "heard this on the radio in a show about xyz..." or "so-and-so told me about this on 2-28-2020 at lunch". Provide option (when viewing page or drilling into history tree) to: - paste in a paragraph or two of text from the page to associate for context - save the entire page in WARC or similar - rate / star / tag that page Provide option to delete pages from history -- either entirely, or "scratched out" (maybe with a comment) so one can remember which branches of the tree are not worth following again. Provide Fuzzy matching as-you-type search across selectable metadata fields Search all content with regex. This would likely involve a browser plugin I guess, but it'd be nice to have a browser-independent way of doing this to facilitate multiple browsers on multiple machines. Also, would be good to avoid "extensions no longer supported after browser update" situations. In the time it took me to get around to typing this up I see there are a lot of other interesting suggestions here...will have to sit down & read through them (and the Linkalot docs) more closely when I've some free time.
javajosh 7y ago
Joplin. OSS Evernote with excellent web clipping; I especially like that you can capture text as markdown.

bloodm 7y ago

You visit a website that you may find intersting. e.g. https://businessfinancing.co.uk/the-oldest-company-in-almost-every-country/ https://businessfinancing.co.uk/the-oldest-company-in-almost... You have folders from a-z in your data directory. You save the website in /data/o/oldestcompanies or into a deeper directory to your liking. Let recoll take care of the rest. https://www.lesbonscomptes.com/recoll/ https://www.lesbonscomptes.com/recoll/

codewithcheese 7y ago
Thanks for sharing your flow/idea. If understand correctly here are the PROS and CONS. PROS - full text search - privacy: save only what you want - save webpage has OK rendering - easy backup using backup software CONS - need to save manually and specify directory/file name - no screenshot. (no screenshot in search results) - save webpage not 100% accurate - requires additional backup solution
- bloodm 7y ago
  Yes. Works for me. There will be always Pros and Cons. - need to save manually and specify directory/file name This may be also a PRO for me since I may be able to find it without using recoll. Also, how many websites do you need to save? I assume only a tiny fraction. If you want to save everything, recoll does offer this more or less out of the box: "Indexing visited WEB pages with the Recoll Firefox extension" https://www.lesbonscomptes.com/recoll/faqsandhowtos/IndexWebHistory.html https://www.lesbonscomptes.com/recoll/faqsandhowtos/IndexWeb... This is another stand alone software that may do what you are looking for. https://getpolarized.io/2019/04/11/Polar-Initial-Crowdfunding-Campaign.html https://getpolarized.io/2019/04/11/Polar-Initial-Crowdfundin...

MayeulC 7y ago

I saw someone recommend Memex as a Firefox extension for full-text search in history and bookmarks the other day. I've started using it, but can't yet comment on its usefulness. It stills feels a bit complex to share data between my computers (I wish for p2p, Nextcloud support, or something alike). I don't like too much it moving DDG's instant answers to the bottom of the page, nor the default sidebar and highlighter, but that could just take some time getting used to.

sails 7y ago

Yet to get around to using, but this looks like it may be good: https://github.com/pirate/ArchiveBox https://github.com/pirate/ArchiveBox

codewithcheese 7y ago
I see they also have a good listing of alternatives https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects https://github.com/pirate/ArchiveBox/wiki/Web-Archiving-Comm...

cparsons3000 7y ago

I would recommend Bookmark OS. It's not open sources but offers full text search, full page screenshots, and other neat features https://bookmarkos.com https://bookmarkos.com

codewithcheese 7y ago
I've seen many services like this come and go. As great as some of them are, having to trust a 3rd party with a full text archive of your complete web history is not a good value proposition. Not only due to privacy but also longevity, I don't want my archive to depends on this company remaining honest while also becoming successful enough to be around 10-20 years.

kissgyorgy 7y ago

Since I found Wallabag, I did not search for anything else. It fundamentally changed my reading habits. Highly recommended! https://github.com/wallabag/wallabag https://github.com/wallabag/wallabag

butterthebuddha 7y ago

What I really want is a script that (1) Takes a URL and optional comment as input (2) Saves the webpage it points to into a git repo (a simple curl should suffice for most websites) (3) Inserts that URL, title of the page pointed-to by the URL and the optional comment into an org-mode file that lives in the root of the repo The org-mode file is a highly-searchable and context-preserving database (I can add tags, create hierarchies, add links to and from other relevant (org-mode or not) files) in the most portable format ever: plain text. I really don't need a web interface. Actually, if I later decide that I need one, I can build one easily on top of this basic system. I really want to be able to use this across multiple devices: mainly my two computers, and an Android phone. Using git gives me a reliable protocol for syncing between multiple devices. I want it to be a smooth experience on my phone, which would probably require some sort of git-aware app. Something similar to the Android client for the pass password manager would be ideal. I hear that git repos can be GPG-encrypted. Ideally, I'm able to serve all this off of a repo hosted on a VPS. I don't want to rely on Dropbox (I'm trying to transition away from it) for syncing.

bachmeier 7y ago
(1) Takes a URL and optional comment as input (2) Saves the webpage it points to into a git repo (a simple curl should suffice for most websites) (3) Inserts that URL, title of the page pointed-to by the URL and the optional comment into an org-mode file that lives in the root of the repo If you're willing to change "git" to "version control", it should be pretty easy to implement that in Fossil. It doesn't require much to add an extension written in your language of choice if you're going to run it on your desktop. Plus you'd get the web interface for free if you decided to put it on a web server.
Karunamon 7y ago
Hmmm.. this wouldn't be too hard to write and sounds like an interesting weekend project. Would you be interested in using WARC for the webpage though? This way, everything is captured in a single file and you aren't littering your repo with random files and images.
- 7y ago

linguaz 7y ago

What I'd find to be interesting & useful is: Every URL I visit is automatically recorded along with the time & date visited, title and other metadata. If the URL is visited via a link off another webpage, have that relationship recorded. Provide some sort of navigable tree / searchable database. Should be able to easily scale to tens / hundreds of millions of URLs across decades. If the URL is visited by manually inputting that URL, provide option to type into a field something like "heard this on the radio in a show about xyz..." or "so-and-so told me about this on 2-28-2020 at lunch". Provide option (when viewing page or drilling into history tree) to: - paste in a paragraph or two of text from the page to associate for context - save the entire page in WARC or similar - rate / star / tag that page Provide option to delete pages from history -- either entirely, or "scratched out" (maybe with a comment) so one can remember which branches of the tree are not worth following again. Provide Fuzzy matching as-you-type search across selectable metadata fields Search all content with regex. This would likely involve a browser plugin I guess, but it'd be nice to have a browser-independent way of doing this to facilitate multiple browsers on multiple machines. Also, would be good to avoid "extensions no longer supported after browser update" situations. In the time it took me to get around to typing this up I see there are a lot of other interesting suggestions here...will have to sit down & read through them (and the Linkalot docs) more closely when I've some free time.

javajosh 7y ago

Joplin. OSS Evernote with excellent web clipping; I especially like that you can capture text as markdown.