3 ms·

The issue with dbt is now folks end up needing to spin up the entire repository to run a bunch of sql queries. The issue is this is a tool that’s targeted at te

The issue with dbt is now folks end up needing to spin up the entire repository to run a bunch of sql queries. The issue is this is a tool that’s targeted at teams that don’t have rigorous engineering backgrounds so it’s bound to be used in suboptimal ways. If the dbt team can do things about it, it might be even better.

adeelk93 5y ago
Before dbt, I didn't even really think about structuring how I'd be running my queries. My experience with it has been - it's made it clear to me what has been sub-optimally laid out. Organize my fact tables by subject, adopt a common schema before applying business logic, utilize re-usable views instead of copy-pasting CTEs, etc. And I like its tight git integration. The alternative, quite often, is a folder of scripts on an analyst's desktop.
- CactusOnFire 5y ago
  Yeah "forcing version control" seems like a feature (pertaining to data governance), not a bug.
bearfood 5y ago
Shameless, but very relevant plug: My team have been working to solve this problem (and more). We recently released our CLI tool, palm, as OSS. Along with a plugin, palm-dbt, which we designed specifically for working with dbt projects in an optimal way. Palm requires running code in containers, so teams can ensure dependencies and versions are correct. Then the plugin provides sensible defaults, with solutions for common problems (like idempotent dev/ci runs) and an abstraction in front of the most common commands you need to run in the container. It's been a huge boon for our analysts productivity, improved the reliability and quality of our models, and decreased onboarding times for new people joining our data team. Links: https://github.com/palmetto/palm-cli https://github.com/palmetto/palm-cli https://github.com/palmetto/palm-dbt https://github.com/palmetto/palm-dbt

3 ms·

The issue with dbt is now folks end up needing to spin up the entire repository to run a bunch of sql queries. The issue is this is a tool that’s targeted at te

by ramraj07 5y ago

adeelk93 5y ago
Before dbt, I didn't even really think about structuring how I'd be running my queries. My experience with it has been - it's made it clear to me what has been sub-optimally laid out. Organize my fact tables by subject, adopt a common schema before applying business logic, utilize re-usable views instead of copy-pasting CTEs, etc. And I like its tight git integration. The alternative, quite often, is a folder of scripts on an analyst's desktop.
- CactusOnFire 5y ago
  Yeah "forcing version control" seems like a feature (pertaining to data governance), not a bug.
bearfood 5y ago
Shameless, but very relevant plug: My team have been working to solve this problem (and more). We recently released our CLI tool, palm, as OSS. Along with a plugin, palm-dbt, which we designed specifically for working with dbt projects in an optimal way. Palm requires running code in containers, so teams can ensure dependencies and versions are correct. Then the plugin provides sensible defaults, with solutions for common problems (like idempotent dev/ci runs) and an abstraction in front of the most common commands you need to run in the container. It's been a huge boon for our analysts productivity, improved the reliability and quality of our models, and decreased onboarding times for new people joining our data team. Links: https://github.com/palmetto/palm-cli https://github.com/palmetto/palm-cli https://github.com/palmetto/palm-dbt https://github.com/palmetto/palm-dbt