4 ms·
Different Airbyte engineer here! Wanted to help answer your question as to what "optional normalized schemas" means. When writing data into your data warehouse
by cgardens 6y ago
Different Airbyte engineer here!
Wanted to help answer your question as to what "optional normalized schemas" means. When writing data into your data warehouse we provide 2 options: 1. write each record as a json blob. 2. infer the schema of the data and write each value in a record to its own column with an appropriate type.
We are betting on EL(T), meaning we think that Transform should be considered separately from EL. To give a more a concrete example, if you are already using DBT in your data warehouse to normalize your data, you likely prefer operating on the "raw" (json blob) data than an arbitrarily normalized form of your data that your EL pipeline has decided on for you. I am seeing this trend pretty pervasively where a lot of, nominally, ELT pipelines are outsourcing the Transform to best in breed tools like DBT.
Thanks for asking this question btw, we'll do our best to clarify in our docs!
- specialist 6y agoThank you for replying. Just scanned DBT. Cool. It's reliance on SQL is The Correct Answer™. (For SQL capable systems, of course.) The solutions based on schemas, where the SQL is then somehow code generated, are terrible. I'm skeptical of DBT's inference of dependencies between queries, but I'll keep an open mind until I have direct experience. Happy hunting.