3 ms·

I am curious, what do you perceive as the difference between data transformations and "wrangling raw data into tables"? Edit: I am also curious why you conside

I am curious, what do you perceive as the difference between data transformations and "wrangling raw data into tables"? Edit: I am also curious why you consider SQL to be inferior for the latter?

jochem9 3y ago
The difference is that raw data can come in many shapes (e.g. impossibly nested jsons) and unexpected quality (changing field names...). I cannot easily work with this in SQL and definitely not write tests to cover the ever increasing complexity of dealing with messy data. After that step the data is a lot more uniform, so then it's easy to use SQL.