5 ms·

"Open table formats" here seem to be the same as "data lakehouse" I've read other places. It'd be great to agree on some nomenclature. Neither are very descrip

"Open table formats" here seem to be the same as "data lakehouse" I've read other places. It'd be great to agree on some nomenclature. Neither are very descriptive (or simple) in my view.

benjaminwootton 3y ago
They are closely related. A lakehouse would be a collection of tables. There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables. The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables. The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable. I always say, Lakehouse is a stupid name but an amazing concept and I’m sure the data industry will go this way.
- lelanthran 3y ago
  Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".
  - ibotty 3y ago
    "(Data) lakehouse" is an amalgamation of data warehouses and data lakes. It's meant to enable querying and all the support (transaction, etc) of traditional data warehouses on a data lake (unstructured data lying on cheap storage).
    - lelanthran 3y ago
      > "(Data) lakehouse" is an amalgamation of data warehouses and data lakes. It's meant to enable querying and all the support (transaction, etc) of traditional data warehouses on a data lake (unstructured data lying on cheap storage). Thank you for that. Do you have any suggestions on where one would start if they wanted to get a better idea and/or some experience using lakehouses?
  - benjaminwootton 3y ago
    Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vendors like Snowflake have to play nicely with it.
    - lelanthran 3y ago
      > Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. This is all very interesting, and thank you for taking the time to explain. Any good starting points for someone who would like to know more?
      - benjaminwootton 3y ago
        Databricks popularised the concept and explain it very well - https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f It’s a technology independent pattern though.
      - dmtryshmtv 3y ago
        See Databricks authors' paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
    - hermitcrab 3y ago
      Data warehouse, data lake, data lakehouse. The data world has some terrible terminology.
      - CyberDildonics 3y ago
        You can't resell someone a file system if you don't rename it to a "data lake". You can't resell someone an indexed file system from the 80's that can be queried with SQL unless you rename it a "data lakehouse".
  - atwebb 3y ago
    That's what it originally meant, at least in my experience. It was when warehouses got access to commodity storage through virtualization options (Hey! I can read S3 from Redshift and it looks like a Redshift table). Similar to Postgres foreign data wrappers or polybase in sql server. Databricks (with Delta as the underpinning) seems to have lead the charge of lakehouse meaning, your data lake+file formats/helpers+compute==data lake+datawarehouse==lakehouse. The latter seems to be the prevailing definition today with the former aging in place.

"Open table formats" here seem to be the same as "data lakehouse" I've read other places. It'd be great to agree on some nomenclature. Neither are very descriptive (or simple) in my view.

benjaminwootton 3y ago

They are closely related. A lakehouse would be a collection of tables. There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables. The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables. The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable. I always say, Lakehouse is a stupid name but an amazing concept and I’m sure the data industry will go this way.

lelanthran 3y ago
Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".
- ibotty 3y ago
  "(Data) lakehouse" is an amalgamation of data warehouses and data lakes. It's meant to enable querying and all the support (transaction, etc) of traditional data warehouses on a data lake (unstructured data lying on cheap storage).
  - lelanthran 3y ago
    > "(Data) lakehouse" is an amalgamation of data warehouses and data lakes. It's meant to enable querying and all the support (transaction, etc) of traditional data warehouses on a data lake (unstructured data lying on cheap storage). Thank you for that. Do you have any suggestions on where one would start if they wanted to get a better idea and/or some experience using lakehouses?
- benjaminwootton 3y ago
  Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vendors like Snowflake have to play nicely with it.
  - lelanthran 3y ago
    > Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. This is all very interesting, and thank you for taking the time to explain. Any good starting points for someone who would like to know more?
    - benjaminwootton 3y ago
      Databricks popularised the concept and explain it very well - https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f It’s a technology independent pattern though.
    - dmtryshmtv 3y ago
      See Databricks authors' paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
  - hermitcrab 3y ago
    Data warehouse, data lake, data lakehouse. The data world has some terrible terminology.
    - CyberDildonics 3y ago
      You can't resell someone a file system if you don't rename it to a "data lake". You can't resell someone an indexed file system from the 80's that can be queried with SQL unless you rename it a "data lakehouse".
- atwebb 3y ago
  That's what it originally meant, at least in my experience. It was when warehouses got access to commodity storage through virtualization options (Hey! I can read S3 from Redshift and it looks like a Redshift table). Similar to Postgres foreign data wrappers or polybase in sql server. Databricks (with Delta as the underpinning) seems to have lead the charge of lakehouse meaning, your data lake+file formats/helpers+compute==data lake+datawarehouse==lakehouse. The latter seems to be the prevailing definition today with the former aging in place.