3 ms·

What I'd love is a program that automatically downloads and archives any article-form website on which I spend more than a minute in a browser, and makes everyt

by jhrmnn 4y ago

What I'd love is a program that automatically downloads and archives any article-form website on which I spend more than a minute in a browser, and makes everything full-text searchable. Does something like that exist? Perhaps it could be solved with a local proxy and some smart heuristics on request timing in a browser-agnostic way.

MarcelOlsz 4y ago
I think what you are looking for is pinboard.in. I have thousands of bookmarks, and I've never opened it once.
kordlessagain 4y ago
I built https://mitta.us/ https://mitta.us/ to do this, then I added GPT-3 to it for discussions and tagging. There's a bookmarklet and a Chrome extension for this as well. It's not quite released, but I'll let anyone test it if they want!
- bosie 4y ago
  I would love to test it please
bosie 4y ago
Question is which search engine can handle this properly though? The first part can probably hacked together with something like DEVONthink and some browser extension but the search features aren’t useful when you have tens of thousands of docs that you just randomly browsed
gildas 4y ago
The auto-save feature of SingleFile browser extension [1] allows you to do this. You will just need to adjust the settings to change the delay after which the page is automatically saved. SingleFileZ [2] can also do this and produce full-text searchable archives. [1] https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile [2] https://github.com/gildas-lormeau/SingleFileZ https://github.com/gildas-lormeau/SingleFileZ
_a9 4y ago
Probably a little hacky but using a userscript & an archive tool could work. I have a userscript[1] that I've been using for years to archive some news sites but it just uses the Wayback Machine. It can probably be modified to wait more time and send the request to something like ArchiveBox[2] (or any archive tool that has a web interface). The downside is you'd need to whitelist specific sites (or whitelist everything and have a keybind or some content matching to activate the archive function? idk) [1] https://gist.github.com/antiops/00a37a1de289415fa7cbd9b5d1d29c54 https://gist.github.com/antiops/00a37a1de289415fa7cbd9b5d1d2... [2] ArchiveBox, can have full text search: https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox

by jhrmnn 4y ago

MarcelOlsz 4y ago

I think what you are looking for is pinboard.in. I have thousands of bookmarks, and I've never opened it once.

kordlessagain 4y ago

I built https://mitta.us/ https://mitta.us/ to do this, then I added GPT-3 to it for discussions and tagging. There's a bookmarklet and a Chrome extension for this as well. It's not quite released, but I'll let anyone test it if they want!

bosie 4y ago
I would love to test it please

bosie 4y ago

Question is which search engine can handle this properly though? The first part can probably hacked together with something like DEVONthink and some browser extension but the search features aren’t useful when you have tens of thousands of docs that you just randomly browsed

gildas 4y ago

The auto-save feature of SingleFile browser extension [1] allows you to do this. You will just need to adjust the settings to change the delay after which the page is automatically saved. SingleFileZ [2] can also do this and produce full-text searchable archives. [1] https://github.com/gildas-lormeau/SingleFile https://github.com/gildas-lormeau/SingleFile [2] https://github.com/gildas-lormeau/SingleFileZ https://github.com/gildas-lormeau/SingleFileZ

_a9 4y ago

Probably a little hacky but using a userscript & an archive tool could work. I have a userscript[1] that I've been using for years to archive some news sites but it just uses the Wayback Machine. It can probably be modified to wait more time and send the request to something like ArchiveBox[2] (or any archive tool that has a web interface). The downside is you'd need to whitelist specific sites (or whitelist everything and have a keybind or some content matching to activate the archive function? idk) [1] https://gist.github.com/antiops/00a37a1de289415fa7cbd9b5d1d29c54 https://gist.github.com/antiops/00a37a1de289415fa7cbd9b5d1d2... [2] ArchiveBox, can have full text search: https://github.com/ArchiveBox/ArchiveBox https://github.com/ArchiveBox/ArchiveBox