21 ms·
dbt does have testing built in, but of course there are only certain cases for which that kind of testing works. dbt can't know if your metrics 'look' right, on
by dave_infuseai 4y ago
dbt does have testing built in, but of course there are only certain cases for which that kind of testing works. dbt can't know if your metrics 'look' right, only you will know.
As others have mentioned, you want to compare the results of your queries against a previously known 'good' state of the data. So, as you're making data model changes, you can regularly check your development environment against production to see how your changes affect the data.
Data profiling is the perfect tool for this, especially when your pipeline reaches a certain size, or you're dealing with very large datasets.
I work on the team creating PipeRider.io, which uses data profiling comparisons as a method of "code review for data".
It becomes particularly useful when you automate generating data profiles of development and production environments in CI, and attach the data profile comparison to the pull request comment. It makes seeing the impact of changes so much easier.
Here's an article that discusses the benefits of this:
https://blog.infuseai.io/why-you-lack-confidence-merging-dbt-data-model-prs-c4f1bc35e7b8 https://blog.infuseai.io/why-you-lack-confidence-merging-dbt...