4 ms·
I'm building something like that, although my goal is to create a certain UI, not to warehouse my own data. I want the usual photos timeline, but extended to in
by nicbou 6y ago
I'm building something like that, although my goal is to create a certain UI, not to warehouse my own data. I want the usual photos timeline, but extended to include other artefacts of my daily life. The goal is to roll back to a point in time and find my photos, conversations, transactions, actions and location history.
Working on my own files is easy: pull files incrementally to a central place with rsync, then process the changed files. Use their checksum to create previews without duplicates.
Working on my own websites is also easy. I just need to add an RSS feed.
Unfortunately, fetching data from other sources is much harder. It made me realise how much of my data is held hostage. Most of it can be retrieved manually, but not with a script that runs regularly. Instant messaging and location history are two big examples.
Repeatability is another problem. For example, Reddit only lets you access your last 1000 comments.
And at last, you must deal with updates. If you go back and revise a comment or a post, the data should be updated.
- dnautics 6y agoI'm also interested in working on something like that but lets you access your data on the go even when your personal data is behind an isp nat layer.
- nicbou 6y agoYou could always host the files elsewhere. I don't do that because this is first and foremost a file backup tool (rsync-based). Otherwise I'd put it on a cheap DigitalOcean VPN.
- simonw 6y agoHave you tried Tailscale? It uses WireGuard to setup a secure mesh network between your devices that punches through NAT, and it is incredibly easy to setup.
- adkadskhj 6y agoThat's really cool. I'm also in this space and have been thinking about this recently, with no answers. Tailscale seems like a great option but i'm mildly concerned about relying on a company for this. Plus the solo plan with a family option sounds a little.. meh. Definitely an interesting take on this problem, appreciated :)
- dnautics 6y agoMy idea is to use webrtc very cleverly. You could make money as a default handshake server. It's an interesting model for disruption because you don't have to worry about legal exposure to the rapidly eroding safe harbor landscape.
- 867-5309 6y ago> Most of it can be retrieved manually, but not with a script that runs regularly this may be down to ability of the script or its author. most things retrievable manually are retrievable by scripts, bots, scrapers, etc. the bigger data captors also provide APIs which can help to a degree, and where limits are imposed there are usually workarounds
- nicbou 6y agoSure, but it's hard to build something reliable and long-lasting that relies on scraping websites that actively try to prevent scrapers. It's not that I can't build it, but rather that I won't.
- 3np 6y agoI feel you. For IM specifically, one thing you could do (if the IM platforms you want are supported) do is to set up a Synapse homeserver with bridges and then you have everything (encrypted or unencrypted depending on config) in a SQL database. May not be worth the overhead depending on your hassle-tolerance if you're not interested in using matrix.org otherwise (I recommend it though). At least a lot of the annoying groundwork is already done. If you do hit some snags, bug reports and PRs are generally very much appreciated in the bridges projects.
- karlicoss 6y agoIt's great that you're building an interface for that, please keep us updated! I think it's great if we solve complementary problems (i.e. I've been heavily on the 'liberate and access data' so far), and plug into each other's solutions.
- nicbou 6y agoIt's at github.com/nicbou/backups. It's already live on my home server, but not really built to be distributed. I build some software like meals: to be enjoyed by myself and a few guests.
- karlicoss 6y agoThanks! Are there screenshots or something like that? "not really built to be distributed" -- completely understandable... sharing has been much harder than I imagined! For myself I'd probably be better off with some huge monorepo.
- kilroy123 6y agoI too have wanted something like this for a long time.
- nitrogen 6y agoLate to the thread, but there's an open source app called fluxtream (no S) that was designed as a personal data logger and aggregator. It might be useful for inspiration. I haven't gotten around to trying it, and don't know how well maintained it is now.