3 ms·
DBT in simplest terms is just a way to orchestrate transformations in dependency order. Table "A" needs to be updated before table "B," because "B" selects fro
by hn72774 3y ago
DBT in simplest terms is just a way to orchestrate transformations in dependency order. Table "A" needs to be updated before table "B," because "B" selects from "A." It works well for that.
I've seen it used to get wild west SQL logic into version control. To replace scheduled SQL workbooks running entire data pipelines.
I've also seen over-engineered integrations with other orchestration tools in the "stack."
Once a DBT project grows to a certain size, roughly 500-1000 .SQL files, it gets hard to manage. Not impossible, though it takes intentionality about how to group things together and scale them out operationally.
"Slim CI" is a recent buzzword that has some nice ideas about build automations.
Cosmos is supposed to solve a lot of the automation gripes with dbt. I haven't tried it yet but would like to.
https://www.astronomer.io/cosmos/ https://www.astronomer.io/cosmos/