5 ms·

For smaller use cases data lakes probably don't make sense. But data lakes have arisen from the enterprise where the centralised data warehouse was the standar

For smaller use cases data lakes probably don't make sense. But data lakes have arisen from the enterprise where the centralised data warehouse was the standard for the last few decades. They know how to use a database. They know how to model and schema the data. And they know about all of the problems it has. They didn't buy into the data lake concept because it's trendy. Fact is that for large enterprises and for those with problematic data sets e.g. telemetry databases simply don't scale. You will always have priority workloads e.g. reporting during which time users and non-priority ETL jobs come second. And often Data Science use cases are banned altogether. The reason data lakes make sense is because it is effectively unlimited scalability. You can have as many crazy ETL jobs, inexperienced users, Data Scientists all reading/writing at the same time with no impact. Generally you want a hybrid model. Databases for SQL users and data lake for everything else.

FridgeSeal 7y ago
> Generally you want a hybrid model. Databases for SQL users and data lake for everything else. I do a mix of data science and software engineering, dealing with the datalake is a nightmare and I avoid it at almost all costs. You know what the first thing everyone I worked with wanted to do after pointlessly pouring everything into the black hole that was the datalake? Re-implement some kind of SQL (and database semantics) back on top of it again; except now it's worse.
oconnore 7y ago
This doesn’t make any sense in the context of tools like Snowflake and BigQuery, where the allocation of compute is separated from the data itself. You can scale each of these use cases independently without cross-domain impact. The data lake model seems to be more about not wanting to commit to a warehouse (for example: future proofing, looking at non-relational data, etc.).
dikei 7y ago
> The reason data lakes make sense is because it is effectively unlimited scalability. You can have as many crazy ETL jobs, inexperienced users, Data Scientists all reading/writing at the same time with no impact. Eh, almost all Data Lakes cannot handle small files well. All it takes is for someone to write 100 million of tiny files into the Data Lake to make life miserable for everyone else.
- threeseed 7y ago
  So don't write small files ? Every time I've seen someone do this it was a mistake and quickly resolved. Either you have way too many partitions in a Spark job or you are treating S3 like it's a queue. And if you really do need lots of delta records then just simply have a compaction job.
  - dikei 7y ago
    Well, inexperience users/data scientists tend to not care about what they write out :) Nevertheless, my point is a Data Lake's does not offer free unlimited scalability. It takes a lots of effort and good engineering practice to make a Data Lake run smoothly at scale.
    - derefr 7y ago
      Data scientists shouldn't be writing anything to the Data Lake. Data Lakes store raw datasets (sort of like Event Streaming databases store raw events.) In academic terms, they store primary-source data. Once data has been through some transformations at the hands of a Data Scientist, it's now a secondary source—a report, usually—and exists in a form better suited to living in a Data Warehouse. Data Lakes need a priesthood to guard their interface, like DBAs are for DBMSes. The difference being that DBAs need to guard against misarchitected read workloads, while the manager of a Data Lake doesn't need to worry about that. They only need to worry about people putting the wrong things (= secondary-source data) into the Data Lake in the first place. In most Data Lakes I've seen, usually there are specific teams with write privilege to it, where "putting $foo in the Data Lake" is their whole job: researchers who write scrapers, data teams that buy datasets from partners and dump them in, etc. Nobody else in the company needs to write to the Data Lake, because nobody else has raw data; if your data already lives in a company RDBMS, you don't move it from there into the Data Lake to process it; you write your query to pull data from both. An analogy: there is a city by a lake. The city has water treatment plants which turn lakewater into drinking water and pump it into the city water system. Let's say you want to do an analysis of the lake water, but you need the water more dilute (i.e. with fewer impurities) than the lake water itself is. What would you do: pump the city water supply into the lake until the whole lake is properly dilute? Or just take some lake water in a cup and pour some water from your tap into the cup, and repeat?

FridgeSeal 7y ago

> Generally you want a hybrid model. Databases for SQL users and data lake for everything else. I do a mix of data science and software engineering, dealing with the datalake is a nightmare and I avoid it at almost all costs. You know what the first thing everyone I worked with wanted to do after pointlessly pouring everything into the black hole that was the datalake? Re-implement some kind of SQL (and database semantics) back on top of it again; except now it's worse.

oconnore 7y ago

This doesn’t make any sense in the context of tools like Snowflake and BigQuery, where the allocation of compute is separated from the data itself. You can scale each of these use cases independently without cross-domain impact. The data lake model seems to be more about not wanting to commit to a warehouse (for example: future proofing, looking at non-relational data, etc.).

dikei 7y ago

> The reason data lakes make sense is because it is effectively unlimited scalability. You can have as many crazy ETL jobs, inexperienced users, Data Scientists all reading/writing at the same time with no impact. Eh, almost all Data Lakes cannot handle small files well. All it takes is for someone to write 100 million of tiny files into the Data Lake to make life miserable for everyone else.

threeseed 7y ago
So don't write small files ? Every time I've seen someone do this it was a mistake and quickly resolved. Either you have way too many partitions in a Spark job or you are treating S3 like it's a queue. And if you really do need lots of delta records then just simply have a compaction job.
- dikei 7y ago
  Well, inexperience users/data scientists tend to not care about what they write out :) Nevertheless, my point is a Data Lake's does not offer free unlimited scalability. It takes a lots of effort and good engineering practice to make a Data Lake run smoothly at scale.
  - derefr 7y ago
    Data scientists shouldn't be writing anything to the Data Lake. Data Lakes store raw datasets (sort of like Event Streaming databases store raw events.) In academic terms, they store primary-source data. Once data has been through some transformations at the hands of a Data Scientist, it's now a secondary source—a report, usually—and exists in a form better suited to living in a Data Warehouse. Data Lakes need a priesthood to guard their interface, like DBAs are for DBMSes. The difference being that DBAs need to guard against misarchitected read workloads, while the manager of a Data Lake doesn't need to worry about that. They only need to worry about people putting the wrong things (= secondary-source data) into the Data Lake in the first place. In most Data Lakes I've seen, usually there are specific teams with write privilege to it, where "putting $foo in the Data Lake" is their whole job: researchers who write scrapers, data teams that buy datasets from partners and dump them in, etc. Nobody else in the company needs to write to the Data Lake, because nobody else has raw data; if your data already lives in a company RDBMS, you don't move it from there into the Data Lake to process it; you write your query to pull data from both. An analogy: there is a city by a lake. The city has water treatment plants which turn lakewater into drinking water and pump it into the city water system. Let's say you want to do an analysis of the lake water, but you need the water more dilute (i.e. with fewer impurities) than the lake water itself is. What would you do: pump the city water supply into the lake until the whole lake is properly dilute? Or just take some lake water in a cup and pour some water from your tap into the cup, and repeat?