4 ms·
I suppose you can always trade those CPU cycles off against storage and cache the N+1 version (in a separate Kafka topic or elsewhere), so now reading the lates
by dkersten 9y ago
I suppose you can always trade those CPU cycles off against storage and cache the N+1 version (in a separate Kafka topic or elsewhere), so now reading the latest-version data is fast, yet you still retain the original data intact, at the expense of more storage. This does complicate the storage though, as you now have multiple days a sources, but nothing that can't be solved.
- raulk 9y agoBingo. If you equate this technique to a DB migration, you could have "up" and "down" directions for the translations from version N <=> N+1. Then if you have 90% confidence you'll only ever need to replay the upgraded stream, you can upgrade it and destroy the previous version. If at some point (the remaining 10%) you need to rescue the old stream, you can run the "down" direction and rehydrate the old version of the stream.
- tedmiston 9y agoIt sounds good in theory. In practice, I haven't heard much around running backwards migrations on a data warehouse / massive collection of events but I'm sure some out there already do it.
- dkersten 9y agoI suppose one needs to take care that migrations are never lossy so that the full information for upgrading or downgrading a version is available.
- tedmiston 9y agoYeah, that's the challenge. For instance, how do you handle when a column was one data type but then down the road was changed to another type when the two aren't cross compatible or could potentially break?
- raulk 9y agoYou could retain this info in a meta field of flexible type. For a DB, it could a JSON type. For messages, it could be an extra _meta field on the message that the systems themselves ignore.