2 ms·

A few years ago I thought about building a personal search engine. The idea was to save the HTML of every site I surfed and search it with a document engine lik

A few years ago I thought about building a personal search engine. The idea was to save the HTML of every site I surfed and search it with a document engine like Apache Lucene. Because text compresses well and isn't big to begin with, a terabyte drive would last a long time and maybe forever. At the time, I thought RPi's would be a good idea because I didn't know any better. Now I might prototype it on an old Thinkpad. Basically, it smells like a solved problem wiht open source tools built for Enterprise. I thought then and think now it could be scaled down to a hardware appliance that sits on a home network. But I am probably wrong about all of it. Good luck.

enether 2y ago
What open source tools are you referring to? Do you just mean the search component? There'd be two hard parts to this problem I reckon: - gathering the data If you make it too cumbersome and with friction, it won't be used. If you make it too easy to dump data, the useful info might get drown out. - ensuring search gives you good results We have open source engines like Lucene that let you search extensively, but what happens when you get 200 results back? How do you know what's the best/most-useful one? It's likely most users would get exhausted having to sift through everything and just default back to a Google
- brudgers 2y ago
  A script running curl on my browsing history would collect the html. I’d solve the 200 result problem if and when it was an actual problem in a way that addressed the actual problem. There’s a lot of success before too many results is a problem. The idea that it might be too much friction than it was worth is why I didnt build it. Probably why nobody has built it and perhaps why you just listed a bunch of imagined problems as reasons not to build it. I mean it would probably be shit if I built it and I liked my idea better than the idea of the work. That’s most things. — For what it is worth, I would default to google for the things google does better and use my personalized historic search when I wanted to see what I had seen before. Its both-and not either-or.
- dotcoma 2y ago
  I’d start from one’s bookmarked websites on Pinboard.
- RulerOf 2y ago
  > gathering the data I've been of the opinion that website content monitoring should be implemented with a browser extension (plus possibly a local agent app)[1]. An extension-based approach would work well and be easy to use IMO. I've been extremely disappointed by how Chrome in particular likes to forget everything about my browsing history (except for tracking cookies) after three months. I don't see why a link I clicked on a year ago on any given page should turn blue just because computers from 2004 might have performance problems with it. [1]: Enterprises seem to prefer MITM here instead, but I'd argue it's not truly required, given the overwhelming popularity of agent-based EDR solutions.
karencarits 2y ago
There are chrome (and firefox) extensions such as Full Text Tabs Forever [1] that offers some of the functinos you describe [1] https://github.com/iansinnott/full-text-tabs-forever https://github.com/iansinnott/full-text-tabs-forever
- brudgers 2y ago
  Thanks. I think I was thinking about the problem in 2016 or so.

enether 2y ago

What open source tools are you referring to? Do you just mean the search component? There'd be two hard parts to this problem I reckon: - gathering the data If you make it too cumbersome and with friction, it won't be used. If you make it too easy to dump data, the useful info might get drown out. - ensuring search gives you good results We have open source engines like Lucene that let you search extensively, but what happens when you get 200 results back? How do you know what's the best/most-useful one? It's likely most users would get exhausted having to sift through everything and just default back to a Google

brudgers 2y ago
A script running curl on my browsing history would collect the html. I’d solve the 200 result problem if and when it was an actual problem in a way that addressed the actual problem. There’s a lot of success before too many results is a problem. The idea that it might be too much friction than it was worth is why I didnt build it. Probably why nobody has built it and perhaps why you just listed a bunch of imagined problems as reasons not to build it. I mean it would probably be shit if I built it and I liked my idea better than the idea of the work. That’s most things. — For what it is worth, I would default to google for the things google does better and use my personalized historic search when I wanted to see what I had seen before. Its both-and not either-or.
dotcoma 2y ago
I’d start from one’s bookmarked websites on Pinboard.
RulerOf 2y ago
> gathering the data I've been of the opinion that website content monitoring should be implemented with a browser extension (plus possibly a local agent app)[1]. An extension-based approach would work well and be easy to use IMO. I've been extremely disappointed by how Chrome in particular likes to forget everything about my browsing history (except for tracking cookies) after three months. I don't see why a link I clicked on a year ago on any given page should turn blue just because computers from 2004 might have performance problems with it. [1]: Enterprises seem to prefer MITM here instead, but I'd argue it's not truly required, given the overwhelming popularity of agent-based EDR solutions.

karencarits 2y ago

There are chrome (and firefox) extensions such as Full Text Tabs Forever [1] that offers some of the functinos you describe [1] https://github.com/iansinnott/full-text-tabs-forever https://github.com/iansinnott/full-text-tabs-forever

brudgers 2y ago
Thanks. I think I was thinking about the problem in 2016 or so.