3 ms·
Martin Kleppmann was presenting architectures that solve this problem: http://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ http://w
by hosh 10y ago
Martin Kleppmann was presenting architectures that solve this problem:
http://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ http://www.confluent.io/blog/turning-the-database-inside-out...
http://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ http://www.confluent.io/blog/bottled-water-real-time-integra...
He does not just identify secondary indices and materialized views as possible candidates, but also other applications such as caching and search indexing as within the same class of problem.
Here, rather than trying to figure out a tradeoff between normalized and denormalized views, Samza and Kafka are used as the heart of what the above article was talking about as "generalized denormalization engine". It's done by capturing data change as a stream to incrementally build whatever denormalized view you want.
The downside is that this is still fairly clunky, if you have this expectation that this should be built into the database. Kleppmann's ideas breaks it out into a whole infrastructure to support that. On the other hand, if you do have a full time ops team, it sounds like it can work very well at scale.