4 ms·
dbt tool. getdbt.com
by hbarka 4y ago
dbt tool. getdbt.com
- mywittyname 4y agoCan dbt run arbitrary code? If it can, it's not well advertised in the documentation. Every time I've looked into dbt, I found that it's mostly a scheduled SQL runner. The primary reason we run Airflow is because it can execute Python code natively, or other programs via Bash. It's very rare that a DAG I write is entirely SQL-based.
- hbarka 4y agoYou’re right. I think the strength of dbt is in the T part of ELT. I wrote ELT to make a distinction in principle from the traditional ETL. (E)xtract and (L)oad is the data ingestion phase that would probably be better served by Dagster, where you could use Python. (T)transform is decoupled and would be served in set-based operations managed by dbt.
- igrayson 4y agodbt has just opened a serious conversation about supporting Python models. I'm sure they'd value your viewpoint! https://github.com/dbt-labs/dbt-core/discussions/5261 https://github.com/dbt-labs/dbt-core/discussions/5261
- KptMarchewa 4y agoDbt is great, but solves only a small part of what Airflow does.