A few years ago I tried writing an application (something like Status Hero for internal use) with a non-traditional database. I used Badger, which is just a tr

A few years ago I tried writing an application (something like Status Hero for internal use) with a non-traditional database. I used Badger, which is just a transactional k/v store, and stored each "row" as protobuf value and an ID number key. (message { int id = 1 }, query by type + ID, store anything with interface { GetId() int }.) I had additional messages for indexes and per-message-type IDs. (I like auto-incrementing IDs, sue me.) A typical transaction would read indexes, retrieve rows, manipulate them, save the rows, save the indexes, and commit. The purity in my mind before I wrote the application was impressive; this is all a relational database is doing under the hood (it has some bytes and some schema to tell it what the bytes mean, just like protos). But it was actually a ton of work that distracted me from writing the app. The code to handle all the machinery wasn't particularly large or anything, but the app also wasn't particularly large. I would basically say, it wasn't worth it. I should have just used Postgres. The one ray of sunshine was how easy it is to ship a copy of the database to S3; the app just backed itself up every hour, which is a better experience than I've had with Postgres (where the cloud provider deletes your backups when you delete the instance... so you have to do your own crazy thing instead). The article is on-point about managing the lifecycle of data. Database migrations are a stressful part of every deployment. The feature I want is to store a schema ID number in every row and teach the database how to run a v1 query against a v2 piece of data. Then you can migrate the data while the v1 app is running, then update the app to make v2 queries, then delete the v1 compatibility shim. If you store blobs in a K/V store, you can do this yourself. If you use a relational model, it's harder. You basically take the app down that knows v1 of your schema, upgrade all the rows to v2, and deploy the app that understands v2. The "upgrade all the rows to v2" step results in your app being unavailable. (The compromise I've seen, and used, which is horrible is "just let the app fail certain requests while the database is being migrated, and then have a giant mess to clean up when the migration fails". Tests lower the risk of a giant mess, and selective queries result in fewer requests that can't be handled by the being-migrated database, so in general people don't realize what a giant risk they're taking. But it can all go very wrong and you should be horrified when you do this.)