3 ms·
My version of this is called Dogsheep, and involves building tools that suck my data from various sources into SQLite databases so I can query it all using Data
by simonw 7y ago
My version of this is called Dogsheep, and involves building tools that suck my data from various sources into SQLite databases so I can query it all using Datasette.
https://dogsheep.github.io/ https://dogsheep.github.io/
I enjoyed reading this author's essay on why he doesn't think loading everything into relational databases is the best approach: https://beepb00p.xyz/unnecessary-db.html https://beepb00p.xyz/unnecessary-db.html
I think I have a workarounds for most of the issues he describes there. One example: I mostly create my base table schema automatically based on the shape of the incoming JSON, and automatically add new columns if that JSON has keys I haven't seen before.
Very thought-provoking writing (and software) here.
- karlicoss 7y agoOoooh, I totally forgot to add your link, I remember you commenting, sorry! Will amend now. Ah I see, so your approach is using databases as an 'intermediate' layer, rather than the main source of 'truth'. My main objection to databases is the fricton with the maintenance and the speed of prototyping, but if it's automatic and used as a cache, I agree it's reasonable. Although I'd still be worried about normalising something wrong, but this is my personal sort of anxiety :) I've got something similar [0]: if you have a sequence of NamedTuples/dataclasses, you can get a database 'for free'. [0] https://beepb00p.xyz/unnecessary-db.html#cachew_cachew https://beepb00p.xyz/unnecessary-db.html#cachew_cachew
- mholt 7y agoCool, that's similar to my project Timeliner: https://github.com/mholt/timeliner https://github.com/mholt/timeliner It downloads your data from social media (or other online sites) into a single sqlite database. It can even draw relations between various accounts/people. I use it to back up my Google Photos.
- mathnmusic 7y agoThere's also heedy: https://heedy.org/ https://heedy.org/
- nikisweeting 7y agoAnd Perkeep https://perkeep.org/ https://perkeep.org/, and Memex https://hyfen.net/memex/ https://hyfen.net/memex/, and a bunch of others...
- anantse 7y agoOh wow that's awesome. This sounds a lot like my project, in philosophy and practice: https://github.com/NickSto/life-browser https://github.com/NickSto/life-browser I've focused on Google Takeout exports so far. Only gotten Hangouts and Voice working at this point. I feel like all of us need to be working on a common codebase for all these parsers. I just found out a week ago that the Hangouts format changed significantly and totally broke my parser (which I actually adapted from https://bitbucket.org/dotcs/hangouts-log-reader/ https://bitbucket.org/dotcs/hangouts-log-reader/)
- karlicoss 7y agoThanks, I'll check it out! Yep, agree about the common codebase, that's why I'm trying to keep everything modular and thinking a lot about it. I describe this bit of my philosophy/design here [0]. I've only written parsers/exporters if I haven't found any (or any that wouldn't be a complete pain to adapt) so far. Google Takeout processing is a part of HPI, but I was going to extract it in a separate repository, so perhaps we both could benefit from it. [0] https://beepb00p.xyz/exports.html#bindings https://beepb00p.xyz/exports.html#bindings
- anantse 7y agoHmm it's interesting how diverse the data sources people are actually using, but there's definitely overlap. I also have tried searching before I write anything myself, and often I find something (both Hangouts and Voice were like that). But it's funny how my use case is always significantly different, so I end up having to put in plenty of work on top of what they've already done. I suppose one thing that could be universally useful is common documentation about what we've discovered in reverse-engineering the formats. I often have extensive notes like this: https://github.com/NickSto/life-browser/blob/master/drivers/hangouts/hangouts.txt https://github.com/NickSto/life-browser/blob/master/drivers/... I remember finding a wiki about this a while back but I can't find it right now..
- aasasd 7y agoIt sounds like a ‘schema-less’/document database like Mongo might work better for API results, which are often non-relational but sorta-hierarchical instead. (Shakes fist at the RTM API which would return one and many child tasks in different ways, so it took me an extra half-hour to write the convoluted Jq query.)
- pradn 7y agoWow, Datasette is super useful.