5 ms·

Having spent the last ~8 months at my work grappling with the consequences and downsides of a Data lake, all I want to do is never deal with one again. Nothing

Having spent the last ~8 months at my work grappling with the consequences and downsides of a Data lake, all I want to do is never deal with one again. Nothing about it was superior or even on par with simply fixing our current shortcomings OLAP database setup. The data lake is not faster to write to; it’s definitely not faster to read from. Querying using Athena/etc was slow, painful to use, broke exceedingly often and would have resulted in us doing so much work stapling in schemas/etc that we would have been net better off to just do things properly from the start and use a database. The data lake also does not have better access semantics and our implementation has resulted in some of my teammates practically reinventing consistency from first principles. By hand. Except worse. Save yourself from this pain: find the right database and figure out how to use it, don’t reinvent one from first principles.

towelpluswater 7y ago
Completely agree with you. Data lakes were marketed well because, well... data warehousing is hard, and a lot of work. Data lakes don't make that hard work disappear, it just changes how and where it happens. I've found data lakes complement DW's (in databases) well. Keep the raw data in the lake and query as needed for discovery, and load it into structured tables as the business needs arise. Data lakes alone are doomed to be failures.
- luckydata 7y ago
  I don't think anyone ever suggested that. The use case for a data lake is precisely the one you describe, it allows you to start collecting data without having to do a lot of work ahead of time before y9u know how you actually want to structure things. Allows for schema evolution too. It's not a panacea, it's just a way to avoid the inertia most large data projects have.
  - towelpluswater 7y ago
    Nobody here suggested it, just something I see organizations doing quite often. (edit: the rationale behind this tends to be that you can avoid the heavy lifting of ETL/transformation logic by just using a data lake - obviously not the case, as most of us know)
    - threeseed 7y ago
      I've worked on nearly a dozen Data Lakes. I have never seen nor heard of anyone who said that Data Lakes meant you could avoid ETL. If anything it has necessitated more of it as users expect to join these disparate data sets. There is after all a reason that the role Data Engineer became popular just as Data Lakes become popular.
      - towelpluswater 7y ago
        Just means we have different anecdotal experience, then. Very little of mine has been in the tech industry.
- threeseed 7y ago
  No. Data lakes were marketed well because they are significantly cheaper and solve long standing problems. S3 is basically free and has unlimited scalability. Oracle, DB2, HANA, SQL Server etc are ridiculously expensive and struggle under high concurrent load even with QoS in place.
  - FridgeSeal 7y ago
    They _appear_ to solve a bunch of problems by simply punting them down the road into downstream applications. None of the databases you listed there are OLAP databases. Clickhouse, TiDB, Redshift, Snowflake, etc are significantly more suitable and should be the target of comparison here.
  - MSM 7y ago
    S3 != a data lake. If you're able to solve the problems that you were previously using oracle or SQL Server for with S3, more power to you, but the truth is that to replicate the functionality of that old Oracle server you'll start with S3, but you'll also want some querying (Aurora? RDS? Hbase?), probably some analytics and ingestion (Redshift? Kinesis? Elastic? Hive? Oozie? Airflow?), along with some security now that you've got multiple tools interacting (Ranger? Knox?), probably some load balancing (Zookeeper?), maybe some lineage and data cataloging (Atlas?), etc. In my experience what starts with "Just throw some data in S3, forget that old crusty expensive server!" ends with 22 technologies trying to cohesively exist because each one provides a small but necessary slice of your platform. Your organization will never be able to find one person who is an expert in all of these (on the contrary, you can find an Oracle, or DB2, or SQL Server expert for half the money) so you end up with seven folks who are each an expert in three of the 22 pieces you've cobbled together, but they all have slightly different ideas on how things should work together, so you end up with a barely functioning platform after a year's worth of work because you didn't want to just start with a $400k license from Oracle.
    - threeseed 7y ago
      Not sure what you are talking about. If you have S3 you can use Athena, Redshift Spectrum or Spark as query layer. It's not 22 technologies. You don't need ElasticSearch, Ranger, Knox, Zookeeper etc as they have nothing to do with querying.
threeseed 7y ago
For smaller use cases data lakes probably don't make sense. But data lakes have arisen from the enterprise where the centralised data warehouse was the standard for the last few decades. They know how to use a database. They know how to model and schema the data. And they know about all of the problems it has. They didn't buy into the data lake concept because it's trendy. Fact is that for large enterprises and for those with problematic data sets e.g. telemetry databases simply don't scale. You will always have priority workloads e.g. reporting during which time users and non-priority ETL jobs come second. And often Data Science use cases are banned altogether. The reason data lakes make sense is because it is effectively unlimited scalability. You can have as many crazy ETL jobs, inexperienced users, Data Scientists all reading/writing at the same time with no impact. Generally you want a hybrid model. Databases for SQL users and data lake for everything else.
- FridgeSeal 7y ago
  > Generally you want a hybrid model. Databases for SQL users and data lake for everything else. I do a mix of data science and software engineering, dealing with the datalake is a nightmare and I avoid it at almost all costs. You know what the first thing everyone I worked with wanted to do after pointlessly pouring everything into the black hole that was the datalake? Re-implement some kind of SQL (and database semantics) back on top of it again; except now it's worse.
- oconnore 7y ago
  This doesn’t make any sense in the context of tools like Snowflake and BigQuery, where the allocation of compute is separated from the data itself. You can scale each of these use cases independently without cross-domain impact. The data lake model seems to be more about not wanting to commit to a warehouse (for example: future proofing, looking at non-relational data, etc.).
- dikei 7y ago
  > The reason data lakes make sense is because it is effectively unlimited scalability. You can have as many crazy ETL jobs, inexperienced users, Data Scientists all reading/writing at the same time with no impact. Eh, almost all Data Lakes cannot handle small files well. All it takes is for someone to write 100 million of tiny files into the Data Lake to make life miserable for everyone else.

towelpluswater 7y ago

Completely agree with you. Data lakes were marketed well because, well... data warehousing is hard, and a lot of work. Data lakes don't make that hard work disappear, it just changes how and where it happens. I've found data lakes complement DW's (in databases) well. Keep the raw data in the lake and query as needed for discovery, and load it into structured tables as the business needs arise. Data lakes alone are doomed to be failures.

luckydata 7y ago
I don't think anyone ever suggested that. The use case for a data lake is precisely the one you describe, it allows you to start collecting data without having to do a lot of work ahead of time before y9u know how you actually want to structure things. Allows for schema evolution too. It's not a panacea, it's just a way to avoid the inertia most large data projects have.
- towelpluswater 7y ago
  Nobody here suggested it, just something I see organizations doing quite often. (edit: the rationale behind this tends to be that you can avoid the heavy lifting of ETL/transformation logic by just using a data lake - obviously not the case, as most of us know)
  - threeseed 7y ago
    I've worked on nearly a dozen Data Lakes. I have never seen nor heard of anyone who said that Data Lakes meant you could avoid ETL. If anything it has necessitated more of it as users expect to join these disparate data sets. There is after all a reason that the role Data Engineer became popular just as Data Lakes become popular.
    - towelpluswater 7y ago
      Just means we have different anecdotal experience, then. Very little of mine has been in the tech industry.
threeseed 7y ago
No. Data lakes were marketed well because they are significantly cheaper and solve long standing problems. S3 is basically free and has unlimited scalability. Oracle, DB2, HANA, SQL Server etc are ridiculously expensive and struggle under high concurrent load even with QoS in place.
- FridgeSeal 7y ago
  They _appear_ to solve a bunch of problems by simply punting them down the road into downstream applications. None of the databases you listed there are OLAP databases. Clickhouse, TiDB, Redshift, Snowflake, etc are significantly more suitable and should be the target of comparison here.
- MSM 7y ago
  S3 != a data lake. If you're able to solve the problems that you were previously using oracle or SQL Server for with S3, more power to you, but the truth is that to replicate the functionality of that old Oracle server you'll start with S3, but you'll also want some querying (Aurora? RDS? Hbase?), probably some analytics and ingestion (Redshift? Kinesis? Elastic? Hive? Oozie? Airflow?), along with some security now that you've got multiple tools interacting (Ranger? Knox?), probably some load balancing (Zookeeper?), maybe some lineage and data cataloging (Atlas?), etc. In my experience what starts with "Just throw some data in S3, forget that old crusty expensive server!" ends with 22 technologies trying to cohesively exist because each one provides a small but necessary slice of your platform. Your organization will never be able to find one person who is an expert in all of these (on the contrary, you can find an Oracle, or DB2, or SQL Server expert for half the money) so you end up with seven folks who are each an expert in three of the 22 pieces you've cobbled together, but they all have slightly different ideas on how things should work together, so you end up with a barely functioning platform after a year's worth of work because you didn't want to just start with a $400k license from Oracle.
  - threeseed 7y ago
    Not sure what you are talking about. If you have S3 you can use Athena, Redshift Spectrum or Spark as query layer. It's not 22 technologies. You don't need ElasticSearch, Ranger, Knox, Zookeeper etc as they have nothing to do with querying.

threeseed 7y ago

For smaller use cases data lakes probably don't make sense. But data lakes have arisen from the enterprise where the centralised data warehouse was the standard for the last few decades. They know how to use a database. They know how to model and schema the data. And they know about all of the problems it has. They didn't buy into the data lake concept because it's trendy. Fact is that for large enterprises and for those with problematic data sets e.g. telemetry databases simply don't scale. You will always have priority workloads e.g. reporting during which time users and non-priority ETL jobs come second. And often Data Science use cases are banned altogether. The reason data lakes make sense is because it is effectively unlimited scalability. You can have as many crazy ETL jobs, inexperienced users, Data Scientists all reading/writing at the same time with no impact. Generally you want a hybrid model. Databases for SQL users and data lake for everything else.

FridgeSeal 7y ago
> Generally you want a hybrid model. Databases for SQL users and data lake for everything else. I do a mix of data science and software engineering, dealing with the datalake is a nightmare and I avoid it at almost all costs. You know what the first thing everyone I worked with wanted to do after pointlessly pouring everything into the black hole that was the datalake? Re-implement some kind of SQL (and database semantics) back on top of it again; except now it's worse.
oconnore 7y ago
This doesn’t make any sense in the context of tools like Snowflake and BigQuery, where the allocation of compute is separated from the data itself. You can scale each of these use cases independently without cross-domain impact. The data lake model seems to be more about not wanting to commit to a warehouse (for example: future proofing, looking at non-relational data, etc.).
dikei 7y ago
> The reason data lakes make sense is because it is effectively unlimited scalability. You can have as many crazy ETL jobs, inexperienced users, Data Scientists all reading/writing at the same time with no impact. Eh, almost all Data Lakes cannot handle small files well. All it takes is for someone to write 100 million of tiny files into the Data Lake to make life miserable for everyone else.