3 ms·
The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it. There's anothe
by lobster_johnson 9y ago
The problem with an RDBMS is that the schema is retroactive -- every time you update the schema, both old and new data must validate against it.
There's another approach, which is to also version the schema. Every record is conceptually a pair of [schema, data]. This puts a burden on the application -- every client must be able to understand old schemas -- but the benefits are considerable.
The most trivial benefit is that no data is ever lost, even if the business case is taken away. Over time, the burden of supporting obsolete features mean that you want to remove columns that don't apply anymore. But that does remove interesting and potentially valuable historical data. If the schema evolves together with the data, you can keep even the cruft, at little expense.
In practice, the evolution of the schema means that data often has to be transformed from one schema to another (e.g. v1 -> v2). In an RDBMS, this transformation (schema migration) is performed once, and if it contains bugs, you have to resort to backups to restore the old data. But with a versioned schema, the transformations are simply functions on immutable data. You can go back and re-run transformations on old data to get new data, non-destructively.
- turbohedgehog 9y agoI think this is what Valve does with Dota 2 replays by using protobuf
- seanp2k2 9y agoCouchDB is cool if you want to mess with revisions: http://docs.couchdb.org/en/2.1.0/intro/api.html http://docs.couchdb.org/en/2.1.0/intro/api.html When you're talking billions - trillions of events, check out something like https://www.slideshare.net/AmazonWebServices/how-netflix-uses-amazon-kinesis-streams-to-monitor-and-optimize-largescale-networks-in-realtime https://www.slideshare.net/AmazonWebServices/how-netflix-use...
- qaq 9y agoid uuid ts timestsmp data jsonb solved it for you :) and thats will have ACID and all the other features of RDBMS that kafka does not have. It will also allow you to create transactionally consistent "views" representing whatever you need implemented in a few lines of SQL vs complex standalone services.
- manish_gill 9y agoNot quite. The JSON overhead can become quite problematic. Not to mention, Postgres's JSON support isn't where I'd like it to be. :(
- qaq 9y agoWhat features are you missing?