3 ms·
This is a video and annotated summary of a talk I gave for the GitHub OCTO Speaker Series yesterday, describing how I've been building my own personal data ware
by simonw 6y ago
This is a video and annotated summary of a talk I gave for the GitHub OCTO Speaker Series yesterday, describing how I've been building my own personal data warehouse on top of my Datasette and Dogsheep open source projects.
I feel like every nerd eventually tries to build something like this. My main contribution here is the observation that importing data from personal sources into SQLite massively reduces the amount of work needed to run further analysis, especially with a web interface that lets you run and bookmark queries.
- heimatau 6y agoI'd like to encourage you to consider doing something for people too. Even if it's in the 'cloud' but maybe where something like telegram/signal has to 'pair' each device with your account and ultimately, the user has the keys. I don't have the time to watch your presentation but I skimmed your slides. Wolfram is legit. I didn't know that about him and I plan on reading that article about his productivity. Also, I didn't realize that others were actually interested in this. I've been trying to do this type of stuff for....about 4 years now. I'm not super tech savvy and I also keep getting busier. I really want this for myself. I have a ton of webpages that I want to put into a system (dated on when I enter it, tags, etc). I currently salvaged my entire email system, even after a lot of headache (years....worth). I have Todoist for tasks. I have Pocket for bookmarks. I want a place for all my notes, tidbits, all my digital content to be put into one place. Even recently, I was thinking, I could get an ANN/ML that would see what I'm 'saving' and doing and it could suggest higher value ideas/content for me to consider. (Like a Youtube recommendation engine but...instead of trying to get you to waste your time on something, instead the ANN is focused on trying to give you a ROI for what you've coming across; becoming a better father, as ME, as an individual..., or taking my first step to learning something that I didn't realize that I wanted to know [accounting is one strange topic that has arise in my life], etc, etc). I would like to eventually get a team and hire a few personal assistants but...ultimately my game plan is what I wrote above. A decent algorithm that has a solid feedback loop based on my experience through life, to help me become more productive on what my goals and thought processes are. I say this all to you, hoping that maybe you or someone in HN could build this. I'd probably be willing to go up to $1k a year or something. Until I generate a lot more money for myself, through my own brute force, I won't have the cash flow to get someone to help me build this myself. I wouldn't want to sell it. I'd want it for myself because there is a 'good enough' threshold that I think would be incredibly valuable to me. (Sure, you could keep iterating...but I suspect that after the threshold is reached, it would be a certain limit of diminishing returns until some major algo breakthroughs happen) P.S. I do understand than you want to target businesses, etc. That's where the money is, sure. But...I wish people could finally get a solid product for once. What's that email service that reduces everyone's clutter that people are paying significantly for? Yeah. Be that, but for the personal data warehouse. I'd pay the premium.
- dnautics 6y agoHey thanks! I'm interested in building this over the next year or so, and I was thinking of using SQLite for exactly these reasons, thanks for confirming my suspicions
- BenGosub 6y agoIf I store data from different sources in the same sqlite db file, is it possible to do joins? I think that there are a lot of use cases if I can join data from different sources of my knowledge base
- simonw 6y agoYes, absolutely. SQLite can even join data together from multiple different database files, though the Datasette interface doesn't support cross database joins just yet.
- BenGosub 6y agoCan I use the same database to pull from multiple sources? And Does datasette provide tools to make it easier to do joins on data from different sources?
- simonw 6y agoYes, you can load data from different sources into the same database - each of the Dogsheep tools lets you pick the database file that you are importing data into. You can create bookmark SQL queries that perform joins in Datasette, or set up views. Here's an example: https://covid-19.datasettes.com/covid?sql=select%0D%0A++++++ny_times_us_counties.date%2C%0D%0A++++++ny_times_us_counties.county%2C%0D%0A++++++ny_times_us_counties.state%2C%0D%0A++++++ny_times_us_counties.fips%2C%0D%0A++++++ny_times_us_counties.cases%2C%0D%0A++++++ny_times_us_counties.deaths%2C%0D%0A++++++us_census_county_populations_2019.population%2C%0D%0A++++++1000000+*+ny_times_us_counties.deaths+%2F+us_census_county_populations_2019.population+as+deaths_per_million%2C%0D%0A++++++1000000+*+ny_times_us_counties.cases+%2F+us_census_county_populations_2019.population+as+cases_per_million%0D%0A++++from%0D%0A++++++ny_times_us_counties%0D%0A++++++join+us_census_county_populations_2019+on+ny_times_us_counties.fips+%3D+us_census_county_populations_2019.fips%0D%0A++++where%0D%0A++++++%22date%22+%3D+%28%0D%0A++++++++select%0D%0A++++++++++max%28date%29%0D%0A++++++++from%0D%0A++++++++++ny_times_us_counties%0D%0A++++++%29%0D%0A++++order+by%0D%0A++++++deaths_per_million+desc%3B https://covid-19.datasettes.com/covid?sql=select%0D%0A++++++...