3 ms·
For a primer on materialized views, and one of the key rationales for Materialize's existence, there's no better presentation than Martin Kleppman's "Turning th
by btown 6y ago
For a primer on materialized views, and one of the key rationales for Materialize's existence, there's no better presentation than Martin Kleppman's "Turning the Database Inside-Out" (2015). (At my company it's required viewing for engineers across our stack, because every data structure is a materialized view no matter where on frontend or backend that data structure lives.)
https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...
Confluent is building an incredible business helping companies to build these types of systems on top of Kafka, Samza, and architectural principles originally developed at LinkedIn, but more along the lines of "if you'd like this query to be answered, or this recommender system to be deployed for every user, we can reliably code a data pipeline to do so at LinkedIn scale" than "you can run this query right away against our OLAP warehouse without knowing about distributed systems." (If it's more nuanced than this please correct me!)
On the other hand, Materialize could allow businesses to realize this architecture, with its vast benefits to millisecond-scale data freshness and analytical flexibility, simply by writing SQL queries as if it was a traditional system. As its capabilities expand beyond parity with SQL (though I agree that's absolutely the best place for them to start and optimize), there are tremendous wins here that could power the next generation of real-time systems.
EDIT: some clarifications and additional examples
- Liron 6y agoI also wrote a primer for why the world needs Materialize [1]. It had a big discussion on HN [2], and Materialize's cofounder said it was part of his motivation [3]. [1] https://medium.com/@lironshapira/data-denormalization-is-broken-7b697352f405 https://medium.com/@lironshapira/data-denormalization-is-bro... [2] https://news.ycombinator.com/item?id=12613586 https://news.ycombinator.com/item?id=12613586 [3] https://twitter.com/narayanarjun/status/1241450203095465986 https://twitter.com/narayanarjun/status/1241450203095465986
- quodlibetor 6y agoHa! Your blog post was one of the reasons that I trusted in the future of Materialize enough to decide to work here! I agree, that is exactly the problem that I, in particular, think we are solving.
- Liron 6y agoThat's awesome, thanks for fixing denormalization :)