3 ms·
Disclaimer: I work for Databricks. With that out of the way, the Data Lakehouse name has a Marketing motivation, but the concept is “let’s not have a DW and a
by bobbruno 4y ago
Disclaimer: I work for Databricks.
With that out of the way, the Data Lakehouse name has a Marketing motivation, but the concept is “let’s not have a DW and a Lake as separate structures based on different technical stacks - let’s have one stack that can do both”. That is a very valid proposition, because the information needs tend to span what a DW or a Lake can easily do by themselves. Also, having one common stack makes it much easier to manage all data and generally be productive.
If you look back even at Inmon’s Corporate Information Factory, you’ll see mentions to unstructured data coming in, and the expectation that all data, structure or unstructured, should become useful information for some business purpose. This was back in the 90s, the need was already there, but the technology was not.
The Lakehouse as proposed by Databricks is essentially a combination of technologies (including but not limited to Delta Lake format) to deliver that unified design. It’s a platform where you can process all kinds of data, apply structuring where needed, manage quality, do DataOps and generally deliver information, regardless of the structure or lack of it in the original data.
So, to put it simply, a Lakehouse is a platform (or technical stack, if you want to build your own) where you can implement a DW, a Data Lake and mix capabilities of both to deliver data products more efficiently. It’s more than just the union of DW and Lake, because the combined capabilities allow you to do things that neither could by itself. For instance, parse a large set of documents for sentiment and context to generate metrics on some business topic. Environment, Sustainability and Governance - ESG - is one example area where the results are BI-like metrics for reports and dashboards, but the sources are usually text documents. Social media processing for Marketing purposes is another.