3 ms·
I have been trying to migrate our legacy ETLs to DBT, which is mostly strings peppered around in python code. I like the macros, templates, the visualisation to
by naruvimama 5y ago
I have been trying to migrate our legacy ETLs to DBT, which is mostly strings peppered around in python code.
I like the macros, templates, the visualisation tool ..... and would like to make our code as DBT compatible as possible.
However, the DB we currently use (Vertica) does not have official support. There are dependency problems installing all of DBT. All I would like to get for my first milestone is to use refs and macros and better organise the SQLs. It is good enough if it generates standard SQL, that I can run outside DBT.
My wish:
I wish I could install select DBT packages just for what I need (templates, macros, refs, dependency) and still make sure I can gradually achieve DBT compatibility. At the moment it looks like all or nothing.
- Tarq0n 5y agoIf your transformations are already in Python scripts wouldn’t you be better off with Airflow or Luigi? dbt seems to be aimed at SQL users primarily.
- naruvimama 5y agoThey are essentially SQL transformation at the reporting stage. Python is just the glue code, a lot of attention has gone into the connectors but no so much into the complexity within the SQL. The problem I would like to address is complex SQL written as strings. Some parts of these repeat over multiple reports, column transformations, look ups, joins with dimensions .....