4 ms·

I used to use a browser extension that would track every page I visited and index everything. This was a few years ago, and I can't remember what it was called

I used to use a browser extension that would track every page I visited and index everything. This was a few years ago, and I can't remember what it was called or why I stopped using it. I think I was paying something like $5/mo. I'd like to find something like that again, it was really useful. I think it would be even more powerful with an AI agent that could organize all the information into categories, and answer questions like "What was the article I was reading last week about <x>?" Is anyone building something like this? (It would be great if I could run something on my own server.)

jcynix 4y ago
Browsers track visits in their history, Firefox for example in a file called "places.sqlite" which you could copy and use as a base log. If you don't mind a bit of tinkering and are a Linux user, you can start with a squid proxy on your server which will keep a log. That could be a Raspi at home. Your browser can connect to that server, either directly, or via SOCKS5 or even an ssh tunnel if the proxy is running on your (external) server. These ssh tunnels are very flexible things. I use an ssh tunnel and SOCK5 on top of that so my network cannot be traced by my provider. The tunnel is set up in a terminal window like ssh -D 8080 -l user proxy.mydomain.net sleep $[45*60] and the browser (Firefox for example) is configured to use the proxy at "localhost:8080" which routes its network connection through the tunnel exiting at my server. The "sleep ..." is any command you want to run instead of a real login session, i.e. you can just drop it if you want. One can add a proxy into this chain. Or use "ssh -v -D ..." to get a log where you can extract your destinations. The AI part would not trivial in general, and especially because of "content creators" and "SEE optimizers" ... and last but not least dynamic web pages. Edited: clarification and typo removal.
- nathan_f77 4y ago
  That sounds like an interesting idea. I like the idea of capturing all the text content I view on the internet and building a full-text search index. Would also be fun to play with NLP and machine learning. I run Tailscale on all my devices, so it would still be easy to use it when I'm away from home. I already have a Linux server that I use for Home Assistant, so I could set it up on there. I'm just not sure how this would work with SSL. I might have to install a self-signed SSL certificate on my devices, but it wouldn't work if a site uses certificate pinning. Is there an easy way around that? Maybe run Chromium with some CLI flags, or use a fork?
  - jcynix 4y ago
    > I'm just not sure how this would work with SSL. Hmm, you can terminate the SSL connections on a proxy. Some colleagues at work (but at a different location)maintain such a setup (i.e. I never set it up myself) but the relevant search terms are "reverse proxy" and "SSL termination" on that proxy. Only the proxy needs a proper certificate then, but that's where Let's Encrypt helps. Edit: here's an example https://bananatrunkingprotocol.com/nginx-reverse-proxy/ https://bananatrunkingprotocol.com/nginx-reverse-proxy/
phgn 4y ago
I am actually working on something like this! It doesn't do question answering right now, but it saves, indexes & categorises articles for you. The reader mode extension this is tied to is open-source, but the backend code not yet. Here's the private beta link if you're interested: library.lindylearn.io/signup

jcynix 4y ago

Browsers track visits in their history, Firefox for example in a file called "places.sqlite" which you could copy and use as a base log. If you don't mind a bit of tinkering and are a Linux user, you can start with a squid proxy on your server which will keep a log. That could be a Raspi at home. Your browser can connect to that server, either directly, or via SOCKS5 or even an ssh tunnel if the proxy is running on your (external) server. These ssh tunnels are very flexible things. I use an ssh tunnel and SOCK5 on top of that so my network cannot be traced by my provider. The tunnel is set up in a terminal window like ssh -D 8080 -l user proxy.mydomain.net sleep $[45*60] and the browser (Firefox for example) is configured to use the proxy at "localhost:8080" which routes its network connection through the tunnel exiting at my server. The "sleep ..." is any command you want to run instead of a real login session, i.e. you can just drop it if you want. One can add a proxy into this chain. Or use "ssh -v -D ..." to get a log where you can extract your destinations. The AI part would not trivial in general, and especially because of "content creators" and "SEE optimizers" ... and last but not least dynamic web pages. Edited: clarification and typo removal.

nathan_f77 4y ago
That sounds like an interesting idea. I like the idea of capturing all the text content I view on the internet and building a full-text search index. Would also be fun to play with NLP and machine learning. I run Tailscale on all my devices, so it would still be easy to use it when I'm away from home. I already have a Linux server that I use for Home Assistant, so I could set it up on there. I'm just not sure how this would work with SSL. I might have to install a self-signed SSL certificate on my devices, but it wouldn't work if a site uses certificate pinning. Is there an easy way around that? Maybe run Chromium with some CLI flags, or use a fork?
- jcynix 4y ago
  > I'm just not sure how this would work with SSL. Hmm, you can terminate the SSL connections on a proxy. Some colleagues at work (but at a different location)maintain such a setup (i.e. I never set it up myself) but the relevant search terms are "reverse proxy" and "SSL termination" on that proxy. Only the proxy needs a proper certificate then, but that's where Let's Encrypt helps. Edit: here's an example https://bananatrunkingprotocol.com/nginx-reverse-proxy/ https://bananatrunkingprotocol.com/nginx-reverse-proxy/

phgn 4y ago

I am actually working on something like this! It doesn't do question answering right now, but it saves, indexes & categorises articles for you. The reader mode extension this is tied to is open-source, but the backend code not yet. Here's the private beta link if you're interested: library.lindylearn.io/signup