8 ms·
So using git instead of a database when a database would be more suited?
by baby 6y ago
So using git instead of a database when a database would be more suited?
- jetpackjoe 6y agoCan you expand on why a database would be more well suited? It seems to me that if the goal is to track changes over time, git is very well suited to that.
- baby 6y agogit is not suited to search through the different fields that have changed (it's content agnostic). A database allows you to plot and search efficiently.
- gitgud 6y agoWell you could say folders are like "tables" and files are like "fields" within those tables... The main difference between git and a traditional DB is querying and relationships between fields
- Frost1x 6y agoDidn't cha know, innovation in the tech industry is all about using things differently than intended and building new compositions that result in the same end goals as older less sophisticated compositions.
- yoz 6y agoGit is a database. It's designed for storing the changes made to files over time. But hey, if there's a different kind of database that you think would be better here, I'd be fascinated to see it. Just make sure that it can 1. handle major structural changes to the files being tracked (e.g. JSON schema revisions) 2. store and retrieve these changes as efficiently as git 3. do the above with as little work required by the user Good luck!
- VWWHFSfQ 6y agothe point is that this isnt particularly useful except to just see a raw diff of the files. the example shows some interesting data points about fires changing periodically but there's basically nothing you can do with that information unless you put it in a real database.
- simonw 6y agoSure - that's what I did with my PG&E outages project (https://simonwillison.net/2019/Oct/10/pge-outages/ https://simonwillison.net/2019/Oct/10/pge-outages/). I wrote a Python script that iterated through the git commits and used them to create a SQLite database so I could run queries. Essentially I was using the commit log as the point of truth for the data, and building a database as an ephemeral asset derived from that data.
- baby 6y agoso full cycle, back to a database :)
- simonw 6y agoWhat's different here is what you treat as the point of truth. If the point of truth is the git repository and its history, then the SQLite database that you build from it is essentially a fancy caching layer - just like if you were to populate a memcached or redis instance or build an Elasticsearch index.
- LukeEF 6y agoA decentralised revision control database revolution is brewing!
- baby 6y agoIs that sarcasm? Because again you're describing a database.
- 6y ago
- simonw 6y agoI actually think git is MUCH better suited to this than a traditional database. I've built track-changes-over-times things in relational databases before and it's a real pain to implement! I've actually started using a GitHub repository to back up the PostgreSQL database that powers my blog, because it's a really cheap way of getting a revision history of that database without having to write any extra code: https://github.com/simonw/simonwillisonblog-backup/commits/main https://github.com/simonw/simonwillisonblog-backup/commits/m...
- tfigment 6y agoI think you want temporal tables for historical dimensions. Postgres doesnt have good support as its a bolt on. MariaDB has good support for it now.
- baby 6y agoI'm not sure I understand the difficulty, your columns are the fields in your object, the rows are the different points in time
- dragonwriter 6y agoA database is probably better if the schema doesn't evolve and the present state of the data fits neatly into a relational schema, because then you are just extending it with versioning and the transformation to a version tracking database is fairly mechanical. But if the data is already in a textual format where [or which can be canonicalized so that] each line is an atomic unit, Git + existing common diff tools get you a lot with very little work and are resilient to schema changes. You could also reduce the data to EAV triples and do a change-tracking DB for that, wwhich would get you immunity to schema changes at the cost of losing all the value that a tracking DB has for known-schema data. So, really, I'd say Git is the easy & general solution, though a DB might be worth the extra work if the schema was known to be fixed.
- jjwiseman 6y agoI’ve put a git-scraped history of 12 years of the FAA registration database into an EAVT database (Datomic) and it worked reasonably well.