4 ms·
About a decade ago, I took on a personal toy project to try and teach myself larger scale programming in java, as well as the APIs provided by multiple internet
by compsciphd 2y ago
About a decade ago, I took on a personal toy project to try and teach myself larger scale programming in java, as well as the APIs provided by multiple internet services (google, twitter, facebook, ....)
The project was to try and collect and make searchable my "internet self". something I called "personal search". i.e. the idea was to try to index every web page I look at, every e-mail I get / see. Social media content shared to me by my social graph (using said APIs), and further indexing pages shared.
The indexing itself wasn't the hard part (per se, as at the time, the APIs facebook, twitter et al were very expansive, much more limited these days, one can attempt to deal with it with intelligent dom scraping, but that's a never ending race where your sources are consistently changing things and you are chasing their changes), the question is how does this information really create significant value for myself? i.e. how often am I going to actually be searching this personal archive. I search google many times a day to find new things (or refind things I already found through it), but how often do I search for things that are within my personal index during a normal day? a handful at most I'd think (and many times, not even once).
With that said, the concept that I then decided to try and teach myself was trying to write a browser extension that could do a similarity search (of sorts) between the documents in my personal index and the web page that I'm currently looking at + content related to that (ex: looking at a news article about current events, idea is that it should surface other articles your friends have shared on the topic and their comments on it). That ended up being an area i didn't have the time (or expertise) to really go far with, so it sort of ended there.
- joshcanhelp 2y ago> how does this information really create significant value for myself? This was a really important question I asked myself. If I build it, will I use it? I had two answers. First, I do, somewhat regularly, have reasons to look back into this archive. If it mission-critical? No. Is it nice to have, yes? I like being able to see what happened on a certain day in the past, I like seeing how trends between activities come together, and I like tying systems together locally rather than in the cloud. I use Obsidian so bringing everything together locally is a big motivation. Second, this is just a hobby, like any other. Having conversations like this, contributing to the universe of open source, connecting with people on the things I feel strongly about (data privacy, local first, life logging) ... it's fun and interesting and a good use of my life. There is a tendency to get a little obsessed with it all (family history of OCD forces me to be mindful about it) but I've enjoyed the process so far. I love your browser extension idea. Managing content from the web that I want to save has been a big challenge locally.