2 ms·
You're not necessarily ingesting semi-unstructured data. Common Data Lake file formats (Avro, Parquet, ORC) are in fact highly structured, and even self-describ
by derefr 7y ago
You're not necessarily ingesting semi-unstructured data. Common Data Lake file formats (Avro, Parquet, ORC) are in fact highly structured, and even self-describing in their schema, with format-features like schema evolution allowing sibling datasets produced at different times to have "different" schemas which nevertheless have a single defined schema as the output when the datasets are unioned together.
The idea, though, is that, if your Data Warehouse wants the data in the form of e.g. a daily-aggregate accounting ledger, then your data sources might be of various time granularities and might be denormalized in different ways (one source with separate Invoices with Transactions foreign-keyed to an Invoice; another with just Transactions with root-level metadata like timestamp directly on them; etc.)
All of the transformations between the source formats and the destination format here are, in some sense, "transparent"—a sufficiently-advanced DBMS query planner could generate an OLAP expression to turn one into the other without understanding the problem domain. It's precisely because of this that, in many cases, it's cheaper to not worry about these kinds of transformations until you need to compute on the data. It's just a bunch of trivial stuff, that you can easily normalize in the computation step, but where fixing it on ingest would have been a whole expensive cluster operation to rewrite terabytes of data, and would require the OpEx of a whole additional set of always-online Hadoop cluster-nodes to fix marginal data as it comes in. Even though you're just going to be touching it all again anyway when you run it through the compute step.