5 ms·

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)?

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name.

detaro 6y ago
Having one central place to find and access data, even if its messy, can be better for understanding and using the data than each business unit/team/... having their own different place for (still messy) data. It can of course also be utterly pointless if done badly, not maintained, not used, ...
xapata 6y ago
Nah, it's a messy pile of data that you want to understand rather than throw away.
LexSiga 6y ago
You are not wrong on the buzzword, but you are not absolutely right if you suggest that it is merely that; this is a recurring question and interrogation in that specific area. The fact that it happens to be kinda buzz worthy is a collateral aspect: everything that answers what some people wonder and that is not yet answered plainly, is. (and I mean, the first on the front page at this very second has : "We hacked apple" in the title.)
PeterisP 6y ago
The reason for data lakes appears in large enough organizations where it becomes exceedingly likely that there is some data that may be useful to you that's maintained by people you'll never meet in a department you don't know about, where it's impractical or even impossible to get them involved in your project that would consume this data. It's not so much about data itself as an attempt to solve a communications and coordination organizational problem; you decouple sources of data and consumers of data (not the technical systems/databases, but the people and organizational units) to a 'hub-and-spoke' model where the providers of data just supply raw data without getting into a multinational project that takes a year just to identify the potential stakeholders for that data throughout a distributed organization with tens of thousands of employees.
- curiousllama 6y ago
  Yea, data lakes are a tech solution to an org problem. Good or bad, it’s what it is.
  - disgruntledphd2 6y ago
    Done right, they are awesome though. As a DS you can iterate a lot faster if you don't have to access multiple different stores for features or data.
dcolkitt 6y ago
With the caveat that "data lake" means different things to different people, I think it's important to have a repository where the raw data exists un-manipulated in the form that it was ingested. From the end-user standpoint that's not very useful. But that's why you have data marts that normalize the raw data into a standardized format. Ultimately though the raw data needs to remain the single source of truth. If you skip that step, and only store the post-normalized format, you're likely to run into problems down the road. This could either be because you want to change the normalization format. Or you want to use some aspect of the data that isn't captured in the normalized form. Or even you discover a pre-existing bug that affected all the previous post-ETL data.
dkarl 6y ago
> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name. That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the data you have time to nicely structure and validate. We want all of it. It doesn't matter if a column is getting truncated to three characters or columns are mislabeled or there are amounts in dollars and euros mixed together; we can still get signal out of it."
- zekrioca 6y ago
  More or less how a proof-of-concept for new type of wine turns out to be sold as Champagne..
  - loopz 6y ago
    As another poster said, if it's the raw data, the raw deal, it's the bets source and nice to have (but could become expensive, so better it is paid for in scale). If it's something else, it's the antithesis of data science.
- freewilly1040 6y ago
  > we can still get signal out of it Today you can, but then when the app owner drops a column (or worse, stop populating it!) in a month that signal will break, and the data lake maintainer will be in the unenviable position of navigating the completely undocumented dependency.
  - dmpetrov 6y ago
    That's might be fine. Anyway, you need to retrain your models quite often due to the concept drift. Solving data problems like this is just a part of model retraining (and concept drift).
sixdimensional 6y ago
I say this as someone who built data infrastructure before and after the invention of data lakes, and I have done it every way - the old and the new.. For the record, for a lot of scenarios, the "old ways" actually do still work fine. But there are new opportunities/possibilities too. I really understand what you are saying... I know the hype problem, I lived it. It makes me both frustrated and sad - because the hype is annoying, there is a lot of vaporware - but there is also something real that is happening too which is part of the story of the evolution of data architecture/infrastructure. My strong advice is, being open-minded is helpful - learn and take what was good/real and leave behind the stigma/hype. Something real and useful happened in terms of architecture, so take the benefits - but of course, don't compromise on delivering real, working solutions. Regarding "data lakehouse", I struggle with the buzzword term also, but once again, I recommend looking at what is good/real and ignoring the stigma of the buzzwords. One way of looking at it is the literal translation - a data warehouse made from the components used to make data lakes. To be honest, it is a marketing term, but it is also an architectural pattern we had even before the term existed - for example - you could use a data warehouse product such as Vertica, and back it by HDFS - guess what, there's a data "lakehouse".. and most of the big database vendors can do this trick now - a full traditional data warehouse engine sitting on top of lake storage infrastructure. There are several "real" use cases for data lakes. Precursor architectures could be seen to be "operational data stores" [1]. Data lakes are real, they are one approach to solving some problems. These use cases could include: 1) raw, long term storage of large volumes of diversely structured data for staging/historical purposes; 2) data discovery/exploration of this data to identify patterns/models and relationships (this is both an AI and analytics use case for power users and BI/analytics/data scientists, etc.); and 3) an opportunity to change the paradigm of traditional ETL - instead of pulling from sources, one way you look at it is, allowing many diverse/distributed sources to push their data into the lake for powering analytics, exploration, AI model building, etc. It makes sense as part of lambda/kappa architecture as well - some of the "push" in can come from streaming sources as well. Use case #1 is very much a "data infrastructure" kind of use case that we do anyway in data warehouses - especially those that do ELT (vs. ETL) - staging databases. If you want an architecture that actually helps make some sense of such a use case of data lakes more formally, one could look into Dan Lindstedt's data vault architecture [2]. While data vault modelling doesn't necessarily require "data lakes", the "raw" part of the data vault architecture use case overlaps nicely with data lakes. [1] https://en.wikipedia.org/wiki/Operational_data_store https://en.wikipedia.org/wiki/Operational_data_store [2] https://en.wikipedia.org/wiki/Data_vault_modeling https://en.wikipedia.org/wiki/Data_vault_modeling
iammru 6y ago
spot on. Data lake approach is a lazy approach. Throw a pile of garbage to the storage layer and figure out how to use it later. People just kicking the can.
laichzeit0 6y ago
What you’re referring to is sometimes called a “data swamp”.
slotrans 6y ago
The simple way to understand the utility of a data lake is "S3 is phenomenally cheap". Some people treat this as an excuse to throw whatever they want into it, without any organization or standardization... and the consequences of that are quite predictable. But it doesn't have to be that way. You can accumulate diverse and large data sets, in cheap cloud storage, while knowing what everything is and where you can find it. As a trivial example, let's say you have a typical OLTP database (or perhaps many), with useful data that is, unfortunately, mutable. You can store entire copies of those tables in your data lake for pennies a day, giving you the ability to recall a transactionally-consistent view of that data from various past times. This is something we've always been able to do using traditional tools, the difference is that storing the data in a "data lake" (i.e. S3) is orders-of-magnitude cheaper. Another major use case, perhaps the most significant one, is storing the raw ingested data -- e.g. from telemetry collection, 3rd party exports, etc -- along with each stage of its transformation. By preserving the original input, along with all intermediate outputs, no information is ever lost. If a buggy transformation is discovered it no longer means your output is irrevocably corrupted, fixed results can be re-computed from wherever in the transform pipeline the bug manifested. And again, this was always possible, a data lake just makes it cheap enough to actually do on a large scale.
- laichzeit0 6y ago
  S3 storage is exactly what modern databases like Snowflake work on. It’s ridiculously easy to turn compressed unstructured (say, json) data into a SQL queryable table.
cpard 6y ago
I think a reason that leads to the need of something like a 'data lake' or anything that looks like a messy pile of incoherent data, is the difference between how data scientists and traditionally data analysts deal with noise in the data. Most BI tasks require the data to be as clean as possible, it's important to be aware of the quality of the data before you calculate your MRR for example. On the other side, data scientists deal with noise more as a parameter of the models they are building. This difference leads to different requirements on how the pipelines should operate. In such a context, having a messy pile of data might help increase the velocity and the independence of some teams inside the company.
- jamesblonde 6y ago
  Noise yes, null or missing values will crash your training and out-of-distribution values (which your schema will not help you with!) will be even worse, hurting performance but impossible to find.
  - cpard 6y ago
    Totally agree on that, both have still to work with data quality but in different ways and with a different mindset.
sjg007 6y ago
Data is the new oil right so you want to decouple your data consumers from the data generators. This means you dump all the data into a lake. That way a reporting team or analytics team doesn’t need to connect to your production database.
ipiz0618 6y ago
In large organizations it is common to have multiple databases maintained by many different people. When a DS needs some data he/she needs to talk to all these different people, wait for them or their managers to grant access, and join the inconsistent labels. This takes a long time and some people are extremely protective of their (messy anyways) tables, requiring higher ups to straight up demand them to grant access. Having nicely organized data is perfect, but I'd rather fetch the data myself from a pile of messy data instead of dealing with all these organizational nightmare.

detaro 6y ago

Having one central place to find and access data, even if its messy, can be better for understanding and using the data than each business unit/team/... having their own different place for (still messy) data. It can of course also be utterly pointless if done badly, not maintained, not used, ...

xapata 6y ago

Nah, it's a messy pile of data that you want to understand rather than throw away.

LexSiga 6y ago

You are not wrong on the buzzword, but you are not absolutely right if you suggest that it is merely that; this is a recurring question and interrogation in that specific area. The fact that it happens to be kinda buzz worthy is a collateral aspect: everything that answers what some people wonder and that is not yet answered plainly, is. (and I mean, the first on the front page at this very second has : "We hacked apple" in the title.)

PeterisP 6y ago

The reason for data lakes appears in large enough organizations where it becomes exceedingly likely that there is some data that may be useful to you that's maintained by people you'll never meet in a department you don't know about, where it's impractical or even impossible to get them involved in your project that would consume this data. It's not so much about data itself as an attempt to solve a communications and coordination organizational problem; you decouple sources of data and consumers of data (not the technical systems/databases, but the people and organizational units) to a 'hub-and-spoke' model where the providers of data just supply raw data without getting into a multinational project that takes a year just to identify the potential stakeholders for that data throughout a distributed organization with tens of thousands of employees.

curiousllama 6y ago
Yea, data lakes are a tech solution to an org problem. Good or bad, it’s what it is.
- disgruntledphd2 6y ago
  Done right, they are awesome though. As a DS you can iterate a lot faster if you don't have to access multiple different stores for features or data.

dcolkitt 6y ago

With the caveat that "data lake" means different things to different people, I think it's important to have a repository where the raw data exists un-manipulated in the form that it was ingested. From the end-user standpoint that's not very useful. But that's why you have data marts that normalize the raw data into a standardized format. Ultimately though the raw data needs to remain the single source of truth. If you skip that step, and only store the post-normalized format, you're likely to run into problems down the road. This could either be because you want to change the normalization format. Or you want to use some aspect of the data that isn't captured in the normalized form. Or even you discover a pre-existing bug that affected all the previous post-ETL data.

dkarl 6y ago

> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name. That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the data you have time to nicely structure and validate. We want all of it. It doesn't matter if a column is getting truncated to three characters or columns are mislabeled or there are amounts in dollars and euros mixed together; we can still get signal out of it."

zekrioca 6y ago
More or less how a proof-of-concept for new type of wine turns out to be sold as Champagne..
- loopz 6y ago
  As another poster said, if it's the raw data, the raw deal, it's the bets source and nice to have (but could become expensive, so better it is paid for in scale). If it's something else, it's the antithesis of data science.
freewilly1040 6y ago
> we can still get signal out of it Today you can, but then when the app owner drops a column (or worse, stop populating it!) in a month that signal will break, and the data lake maintainer will be in the unenviable position of navigating the completely undocumented dependency.
- dmpetrov 6y ago
  That's might be fine. Anyway, you need to retrain your models quite often due to the concept drift. Solving data problems like this is just a part of model retraining (and concept drift).

sixdimensional 6y ago

I say this as someone who built data infrastructure before and after the invention of data lakes, and I have done it every way - the old and the new.. For the record, for a lot of scenarios, the "old ways" actually do still work fine. But there are new opportunities/possibilities too. I really understand what you are saying... I know the hype problem, I lived it. It makes me both frustrated and sad - because the hype is annoying, there is a lot of vaporware - but there is also something real that is happening too which is part of the story of the evolution of data architecture/infrastructure. My strong advice is, being open-minded is helpful - learn and take what was good/real and leave behind the stigma/hype. Something real and useful happened in terms of architecture, so take the benefits - but of course, don't compromise on delivering real, working solutions. Regarding "data lakehouse", I struggle with the buzzword term also, but once again, I recommend looking at what is good/real and ignoring the stigma of the buzzwords. One way of looking at it is the literal translation - a data warehouse made from the components used to make data lakes. To be honest, it is a marketing term, but it is also an architectural pattern we had even before the term existed - for example - you could use a data warehouse product such as Vertica, and back it by HDFS - guess what, there's a data "lakehouse".. and most of the big database vendors can do this trick now - a full traditional data warehouse engine sitting on top of lake storage infrastructure. There are several "real" use cases for data lakes. Precursor architectures could be seen to be "operational data stores" [1]. Data lakes are real, they are one approach to solving some problems. These use cases could include: 1) raw, long term storage of large volumes of diversely structured data for staging/historical purposes; 2) data discovery/exploration of this data to identify patterns/models and relationships (this is both an AI and analytics use case for power users and BI/analytics/data scientists, etc.); and 3) an opportunity to change the paradigm of traditional ETL - instead of pulling from sources, one way you look at it is, allowing many diverse/distributed sources to push their data into the lake for powering analytics, exploration, AI model building, etc. It makes sense as part of lambda/kappa architecture as well - some of the "push" in can come from streaming sources as well. Use case #1 is very much a "data infrastructure" kind of use case that we do anyway in data warehouses - especially those that do ELT (vs. ETL) - staging databases. If you want an architecture that actually helps make some sense of such a use case of data lakes more formally, one could look into Dan Lindstedt's data vault architecture [2]. While data vault modelling doesn't necessarily require "data lakes", the "raw" part of the data vault architecture use case overlaps nicely with data lakes. [1] https://en.wikipedia.org/wiki/Operational_data_store https://en.wikipedia.org/wiki/Operational_data_store [2] https://en.wikipedia.org/wiki/Data_vault_modeling https://en.wikipedia.org/wiki/Data_vault_modeling

iammru 6y ago

spot on. Data lake approach is a lazy approach. Throw a pile of garbage to the storage layer and figure out how to use it later. People just kicking the can.

laichzeit0 6y ago

What you’re referring to is sometimes called a “data swamp”.

slotrans 6y ago

The simple way to understand the utility of a data lake is "S3 is phenomenally cheap". Some people treat this as an excuse to throw whatever they want into it, without any organization or standardization... and the consequences of that are quite predictable. But it doesn't have to be that way. You can accumulate diverse and large data sets, in cheap cloud storage, while knowing what everything is and where you can find it. As a trivial example, let's say you have a typical OLTP database (or perhaps many), with useful data that is, unfortunately, mutable. You can store entire copies of those tables in your data lake for pennies a day, giving you the ability to recall a transactionally-consistent view of that data from various past times. This is something we've always been able to do using traditional tools, the difference is that storing the data in a "data lake" (i.e. S3) is orders-of-magnitude cheaper. Another major use case, perhaps the most significant one, is storing the raw ingested data -- e.g. from telemetry collection, 3rd party exports, etc -- along with each stage of its transformation. By preserving the original input, along with all intermediate outputs, no information is ever lost. If a buggy transformation is discovered it no longer means your output is irrevocably corrupted, fixed results can be re-computed from wherever in the transform pipeline the bug manifested. And again, this was always possible, a data lake just makes it cheap enough to actually do on a large scale.

laichzeit0 6y ago
S3 storage is exactly what modern databases like Snowflake work on. It’s ridiculously easy to turn compressed unstructured (say, json) data into a SQL queryable table.

cpard 6y ago

I think a reason that leads to the need of something like a 'data lake' or anything that looks like a messy pile of incoherent data, is the difference between how data scientists and traditionally data analysts deal with noise in the data. Most BI tasks require the data to be as clean as possible, it's important to be aware of the quality of the data before you calculate your MRR for example. On the other side, data scientists deal with noise more as a parameter of the models they are building. This difference leads to different requirements on how the pipelines should operate. In such a context, having a messy pile of data might help increase the velocity and the independence of some teams inside the company.

jamesblonde 6y ago
Noise yes, null or missing values will crash your training and out-of-distribution values (which your schema will not help you with!) will be even worse, hurting performance but impossible to find.
- cpard 6y ago
  Totally agree on that, both have still to work with data quality but in different ways and with a different mindset.

sjg007 6y ago

Data is the new oil right so you want to decouple your data consumers from the data generators. This means you dump all the data into a lake. That way a reporting team or analytics team doesn’t need to connect to your production database.

ipiz0618 6y ago

In large organizations it is common to have multiple databases maintained by many different people. When a DS needs some data he/she needs to talk to all these different people, wait for them or their managers to grant access, and join the inconsistent labels. This takes a long time and some people are extremely protective of their (messy anyways) tables, requiring higher ups to straight up demand them to grant access. Having nicely organized data is perfect, but I'd rather fetch the data myself from a pile of messy data instead of dealing with all these organizational nightmare.