4 ms·

I'm so psyched about Materialize. An old coworker explained to me about how his previous company used DBT to create many different projections of messy data to

I'm so psyched about Materialize. An old coworker explained to me about how his previous company used DBT to create many different projections of messy data to serve many applications, rather than trying to come up with the One Canonical Representation. It truly blew my mind in terms of thinking about how to model data within a business. The huge limitation with this vision is that it only works in places where you can tolerate some pretty significant staleness. So the promise of this approach excludes most OLTP applications. I simply assumed it wouldn't be reasonable to create something that allows for unconstrained SQL-based transformations in real time, and that no one was working on this. Oh well. But several months back, I discovered Materialize and it was an "oh shit" moment. Someone was actually doing this, and in a very first principles-driven approach. I'm really excited for how this project evolves.

thejosh 6y ago
dbt is fantastic. It depends on your usecase, but for ours having an hourly sync of data is fine for reporting.
- drewbanin 6y ago
  We're super excited about materialize at Fishtown Analytics (the company that makes dbt). I think that the "reporting" use-case is well served by Snowflake/BigQuery/etc, but I do think that operational use-cases are left behind by batch-based transformation models. The thing I'm most excited about in materialize is the ability to create Sinks (https://materialize.com/docs/sql/create-sink/ https://materialize.com/docs/sql/create-sink/). The combination of 1) streaming transformations over realtime source data and 2) streaming outputs into external systems feels like the way of the future IMO. I saw a dbt-materialize plugin (https://github.com/jwills/dbt-materialize https://github.com/jwills/dbt-materialize) out there in the wild. My guess is it's not ready for primetime yet, but would love to bake support for materialize into dbt when the time is right :)

thejosh 6y ago

dbt is fantastic. It depends on your usecase, but for ours having an hourly sync of data is fine for reporting.

drewbanin 6y ago
We're super excited about materialize at Fishtown Analytics (the company that makes dbt). I think that the "reporting" use-case is well served by Snowflake/BigQuery/etc, but I do think that operational use-cases are left behind by batch-based transformation models. The thing I'm most excited about in materialize is the ability to create Sinks (https://materialize.com/docs/sql/create-sink/ https://materialize.com/docs/sql/create-sink/). The combination of 1) streaming transformations over realtime source data and 2) streaming outputs into external systems feels like the way of the future IMO. I saw a dbt-materialize plugin (https://github.com/jwills/dbt-materialize https://github.com/jwills/dbt-materialize) out there in the wild. My guess is it's not ready for primetime yet, but would love to bake support for materialize into dbt when the time is right :)