4 ms·
Ask HN: Would you try an app to consolidate and search your digital footprint?
The other day I spent a couple of hours trying to extract the data for my runs from Nike Run Club's servers. They have shut down their public API but thankfully some GitHub repos presented working solutions.
As I browsed through ten years of my run data, I had the idea of creating a software that let's you extract all (or most) of your online data and then search it using natural language queries. It is intended to help you piece together the puzzle of your digital footprint/history.
For example, by integrating with cloud storage (Dropbox/Drive), browser history (Firefox/Chrome), fitness apps (NRC/Strava), blogging/note-taking software (Obsidian/Bear), bookmark services (Pocket/Omnivore) and entertainment apps (YouTube/Spotify), you should be able to get answers to questions like –
* "What was my average running pace in July 2019? Show me all of my trail runs from that time."
* "What all recipes did I save in 2020?"
* "When did I start writing about TypeScript? Which tech blogs was I reading at that time?"
* "What videos was I watching between 2015-2017?"
* "Show me the artists I discovered on Spotify during the pandemic."
My intention is to build a local-first and privacy-first solution with a simple SQLite database. The user should be able to do whatever they want with the extracted data. A savvy user might build their own GUI, while a not as savvy one might just like to archive their data in a personal storage server.
Does this sound like a good idea? Is it something that you would want to try? Do similar solutions already exist?
P.S. When I told my brother about it he called it "the god app" which I thought was pretty funny and accurate.
- Jeremy1026 2y agoWould I use it? Probably once just out of curiosity. Do I think I'll see it in my lifetime? Not a shot. The odds of someone gaining access to all the data in all the services I use/used have to be close to nil. Companies aren't going to give up their user data, that's their bread and butter. Each service will need to have custom extractors written, and likely rewritten in a never ending game of cat and mouse. That's even if you don't get sued for accessing their systems to extract their data they have on your user. Then the storage required if you are able to get all the data. I requested my data from Apple a few years ago. It was something like 10GB of information. I assume shopping, social media, fitness, vehicle, mapping, etc. services have similar amounts of data on me. I wouldn't be surprised if the average digital identity has 1TB+ of data associated with it. Then, you have to normalize all the data. Each service is going to have the data in its own format with their own nuances that'll be a huge pain to get to a singular searchable format.
- thatgurjot 2y agoThat’s a fair point! Exactly what my brother pointed out when he called it the “god app”. However, this isn’t intended to be the “ultimate” app with your entire digital footprint logged to a database. It’s supposed to have a few meaningful connectors to services that people would actually be interested in. And a lot of these services offer public APIs as well (Spotify, Omnivore, Strava). Plus, browser history and local files (like markdown notes) are easy to access. Do you think this reduced scope would be of any interest to you?
- warrenm 2y agoI have one ... it's called IFTTT.com heck! I have a couple different accounts on it for different purposes :)
- thatgurjot 2y agoBut IFTTT or Zapier don’t have the integrations for exporting your historical data, right? At least I didn’t find those plugins!
- runjake 2y agoSounds like a good idea with some hard challenges. Cloud storage, browser history, bookmarks, and blogging/notes will generally be easier. Fitness won't be unless you're using Apple Health or Garmin. It sounds a lot like IFTTT with some AI thrown in. What have you done so far?
- thatgurjot 2y agoI would say that it is different from IFTTT and Zapier because it is not meant to “do something” when something happens in your cloud service. It’s only meant to periodically save your data in a local database on your computer. Plus there’s a GUI that can help you query that data (which is where the AI can come in, if necessary). So far I have worked out the connectors for a handful of services (prioritising those with local data and public APIs) and a general-purpose database schema to store and query the data. I have an existing app (made using the Tauri framework in Rust + Svelte) that acts as a local file search engine that I hope to use as a starting point.
- runjake 2y agoVery cool. Consider posting screenshots to the thread! I do a bunch of this, in a set of messy Python scripts that need refactoring, for personal use. It's definitely largely doable, but not without it's challenges for some services. Thankfully, at a personal level, I can choose not to use those services and move to something that is more open.
- thatgurjot 2y agoThanks for the vote of confidence! I’ll post screenshots when I have something more than haphazardly put together Python scripts :) You are right about switching to an open service! I am prioritising such services. Like Omnivore[1] is for bookmarking. [1]: https://omnivore.app https://omnivore.app
- runjake 2y agoThis is called prototyping and there's not a thing wrong with it. Good luck and keep us updated.
- KomoD 2y agoNope, definitely not. Not going to give all my data to some app.
- thatgurjot 2y agoThe app is only going to extract your data from those services and save it to a database on your local device. I am not proposing a rent seeking SaaS solution. Just an open source solution for a problem, say like Calibre is for ebooks.
- KomoD 2y agoYeah, still wouldn't do that. A vulnerability with just that app is enough to get my data from all those other places, too risky for me
- thatgurjot 2y agoFair point. Thanks for sharing your perspective! Would you consider it a risk even if the app was open source?
- solardev 2y agoIsn't this what Microsoft was trying to build with Recall?
- thatgurjot 2y agoI’d differ. Recall is more akin to what Rewind AI[1] is building. I don’t care to know what happened in a chat box in my Zoom call last Friday. I just want to be able to explore structured data that I have generated across different services. For example, Nike Run Club offers some stats on the app but I’d like to be able to do more. Spotify shows me I follow this artist but I’d like to know since when have I followed them. YouTube has my watch history but I’d like to be able to process what kind of content I was consuming during a certain time period. [1]: https://www.rewind.ai/ https://www.rewind.ai/
- solardev 2y agoSo not a point in time snapshot to go back to, but natural language analysis of aggregate data? I can see how that'd be useful.
- thatgurjot 2y agoExactly! Consider you have an SQLite database on your computer with your aggregated data from Strava, Spotify, and everything else. You do whatever you want with it. The app will come with a GUI that will allow some additional analysis or querying but you do your thing if you like!
- solardev 2y agoYeah, that'd be cool. I'd love to feed it my fitness, health, etc. data. I feel like it was always a missed opportunity with Google... they had all my Google Fit, Google Health, 23andme, Gmail (and thus order confirmations, healthcare appointments), text messages, Google Maps locations, etc. data that I expressly opted-in for, hoping that one day they'd be able to use that to provide actionable insights. But nope, it was all just ads and spam :(