4 ms·
What's the difference between a Delta Lake and Change Data Capture? Seems like in both cases you're creating a type 2 dimension against a source table.
by iblaine 7y ago
What's the difference between a Delta Lake and Change Data Capture? Seems like in both cases you're creating a type 2 dimension against a source table.
- mobileexpert 7y agoThe Delta here isn’t so related to CDC. This is a replacement for the ad how methods people use now to keep multiple readers and writers of datasets in parquet format in agreement when using Spark’s dataframe API.
- century19 7y agoCDC you are looking at the change logs of a Relational Database typically. With a (batch) Data Lake you accept as input different file types (JSON, CSV, AVRO) from various systems. Could be Hadoop systems, could be from a COBOL system on a mainframe.
- atwebb 7y agoIt is a different technology entirely. CDC is just the log of changes on a relational table. Delta Lake appears to be providing more native administrative capabilities to a data lake implementation (schemas, transactions, cataloging).