4 ms·
Imo dbt sits in the analysts space. SQL is the language they know and dbt is the best solution to scale that. In that sense it should only be used for last mil
by jochem9 3y ago
Imo dbt sits in the analysts space. SQL is the language they know and dbt is the best solution to scale that.
In that sense it should only be used for last mile data transformations. Not for wrangling raw data into neat tables.
- fifilura 3y agoI am curious, what do you perceive as the difference between data transformations and "wrangling raw data into tables"? Edit: I am also curious why you consider SQL to be inferior for the latter?
- jochem9 3y agoThe difference is that raw data can come in many shapes (e.g. impossibly nested jsons) and unexpected quality (changing field names...). I cannot easily work with this in SQL and definitely not write tests to cover the ever increasing complexity of dealing with messy data. After that step the data is a lot more uniform, so then it's easy to use SQL.