3 ms·

Show HN: Intercepting proxy for semantic search over visited pages

14 points by lynx97 1y ago

jauntywundrkind 1y ago
I love the general shape of this, of having some semantic record of what you browse. It's been a long goal/dream and this is an interesting neat hack to get there, a path towards that future that hasn't been open until recently. I haven't run this yet/don't know what the UI looks like (nudge nudge), but looks like this is less for record-keeping? More for finding similar thing to what you are seeing now, although it does have a cached view?
- lynx97 1y ago
  It stores the text version of every page you visited via the proxy, and lets you do semantic search on all of them. You can retrieve the stored text in the search interface, much like the "Cached" link on old Google worked. The search UI is as minimal as it can get, likely too minimal for those used to the modern web... The idea is to be able to search for something you've seen recently, but maybe haven't bookmarked. btw, currently powered by pure.md I am considering to add markitdown support, but my initial tests of markitdown gave rather subpar results. I'll wait a few weeks until the author (of markitdown) comes back from holidays... If the project continues to improve, it might be a good match. Right now, pure.md gives way better results.
luke-stanley 1y ago
Sending requests to pure.md (?) seems pretty insecure and inefficient to me. If you're going to MitM yourself, KEEP IT LOCAL, then use reliable code to convert HTML to Markdown efficiently with a queue, drop text parts that are noise, summarise if that's the format desired, and THEN embed!
luke-stanley 1y ago
I made another approach that detects browser tab text changes with a Userscript that sends changes as Markdown to a server that appends to a JSONL file. MitM might be interesting to compare: https://gist.github.com/lukestanley/c3a37ab61a45e72b74995a5cb7d20f70 https://gist.github.com/lukestanley/c3a37ab61a45e72b74995a5c... (two files here BTW)

14 points by lynx97 1y ago

I love the general shape of this, of having some semantic record of what you browse. It's been a long goal/dream and this is an interesting neat hack to get there, a path towards that future that hasn't been open until recently. I haven't run this yet/don't know what the UI looks like (nudge nudge), but looks like this is less for record-keeping? More for finding similar thing to what you are seeing now, although it does have a cached view?

lynx97 1y ago
It stores the text version of every page you visited via the proxy, and lets you do semantic search on all of them. You can retrieve the stored text in the search interface, much like the "Cached" link on old Google worked. The search UI is as minimal as it can get, likely too minimal for those used to the modern web... The idea is to be able to search for something you've seen recently, but maybe haven't bookmarked. btw, currently powered by pure.md I am considering to add markitdown support, but my initial tests of markitdown gave rather subpar results. I'll wait a few weeks until the author (of markitdown) comes back from holidays... If the project continues to improve, it might be a good match. Right now, pure.md gives way better results.

luke-stanley 1y ago

Sending requests to pure.md (?) seems pretty insecure and inefficient to me. If you're going to MitM yourself, KEEP IT LOCAL, then use reliable code to convert HTML to Markdown efficiently with a queue, drop text parts that are noise, summarise if that's the format desired, and THEN embed!

luke-stanley 1y ago

I made another approach that detects browser tab text changes with a Userscript that sends changes as Markdown to a server that appends to a JSONL file. MitM might be interesting to compare: https://gist.github.com/lukestanley/c3a37ab61a45e72b74995a5cb7d20f70 https://gist.github.com/lukestanley/c3a37ab61a45e72b74995a5c... (two files here BTW)