7 ms·
Comparison of Data Lake Table Formats (Iceberg, Hudi and Delta Lake)
- pid-1 4y agoThe repo comparison was really cool. I guess that could be made into a product.
- infogulch 4y agoAgreed, the repo comparison was surprisingly thorough.
- alexmerced 4y agoI'll be presenting this content at this meetup if you want to learn more: https://www.meetup.com/subsurface-global/events/286442080/ https://www.meetup.com/subsurface-global/events/286442080/
- ttunguz 4y agoDoes anyone have experience running either of these three in production?
- curiousDog 4y agoYes, we run iceberg and it’s pretty thorough
- ramesh31 4y agoCurious why you chose it over Delta Lake? What does your ETL/BI stack look like?
- venki80 4y agoThe big tech companies like Apple and Netflix all use Iceberg
- alexott 4y agoApple uses Delta: https://databricks.com/session_na20/patterns-and-operational-insights-from-the-first-users-of-delta-lake https://databricks.com/session_na20/patterns-and-operational..., similarly Disney+: https://databricks.com/p/webinar/unlock-the-potential-inside-your-data-lake-aws https://databricks.com/p/webinar/unlock-the-potential-inside..., and hundreds other companies (https://databricks.com/customers https://databricks.com/customers)
- bushbaba 4y agoApple has named committees and tech talks on iceberg. Like all big companies they use multiple technologies
- alexmerced 4y agoI'll be presenting this content at this meetup if you want to learn more: https://www.meetup.com/subsurface-global/events/286442080/ https://www.meetup.com/subsurface-global/events/286442080/
- venki80 4y agoWondering if this is basically what all data lakes will look like in the future. All data stored in these table formats…
- SpicyLemonZest 4y agoI definitely think that some kind of semi-structured storage is the future. "Here's a giant heap of files" was always kinda a hack for when you outscaled your RDBMS but didn't have time to build something better.
- jerf 4y agoConsider the whole data sector as: Generation -> Ingestion -> Transform (and possible looping back as derived data is created) -> Resting place -> Final Useful Product Not saying that's a perfect model, just something to hang the terms I use in this post on. And then consider that in order to get from the beginning of that process to the end, there is a certain amount of "Data Cleanup" to be done, ranging from merely validating that the data is sensible to in the limit literally handing huge blobs of text and unstructured data to humans and making them input something useful into the system out of it. My assessment of the whole data community right now (and please, by all means react to this with your own opinions, I'm curious about them) is that the entire flush of fads going back and forth right now amounts to an argument about how exactly to distribute the necessary Data Cleanup work across that pipeline. The theoretical ideal is for everything to just be super awesome at the generation phase and nothing else has to worry about it, but it was rapidly discovered that making the generation part so expensive inhibits the data from ever being generated. With clean data, downstream could do all sorts of database-y storage technologies and do all sorts of clever things with the clean data, but the data is never clean. The natural overreaction is to flip entirely in the other direction and just get it in and push the validation as far down the pipeline as possible. Here you get the "big piles of vaguely organized files". You get more data this way because you lower the costs of generation and to some extent ingestion, but you complicate everything downstream. It seems to me we're currently in a phase where everyone is just sort of hoping somebody else will do it, and we're flailing around a bit. Very opinionated: Where we're going to settle in, and where you can already see the shape forming up, is that it'll be a little mix & match at each level. Do what's easiest in each level at that level, and you end up with the cheapest and most effective result across the pipeline considered as a whole, even though no individuals working in any part of it will be 100% happy. There won't be a magic solution, but if, for instance, Ingestion demands that the Generation at least be amenable to some tabular view, even if there are some escape hatches for generic JSON bits, they can start operating with sensible tools (SQL-ish like Clickhouse or something) instead of just having a pile of opaque nothingness, and then the next levels down won't be able to count on data quality or coherence 100% but you can start layering in cleanliness and coherence as you go, etc. There just isn't a magic solution that fits into bullet points cleanly. (There's this "bronze/silver/gold" thing going on, which I think is silly because there's really not much benefit to trying to force an arbitrarily-deep and complicated pipeline into such classifications, but the idea is there.) Or, in short, yes I expect to see more tabular data. It just won't be tabular for the same reason that relational DBs use tables. It'll be tables even fairly early just because you need some sort of handle on the data to do any sort of useful manipulation on it. If relational DBs use tables as an emphasis on tables qua tables of data, data lakes will use tables as defined handles on individual pieces of data to be able to manipulate them as opposed to pure unstructured piles of "something". It reminds me of the 20+ year, still ongoing argument about where in the "Browser -> Server -> Backend Services (including DB)" stack the work needs to be done. There's a certain amount of work that has to be done. You've got a bajillion choices about where to do it, and it's been sloshing back and forth across the entire time the web has existed ("do it all in SQL procedures! Do it all on the client!") because there is no simple hard & fast correct answer that everyone can follow for every case. Just as with that world, this reality won't stop a pile of vendors from promising they can somehow make this problem go away, but they really can't. They can reduce the accidental complexity, and that's cool and may be worth paying for, but there's essential complexity that isn't going anywhere. (Stretching even more abstractly, I'm writing a bit about how to do stream processing with io.Reader in Go, and it reminds me a bit of that, too. Stream processing is too complicated to write a single-shot conversion from "whatever's coming in" to the golden data you're looking for in many cases, and the solution is to fold in several transforms at a time, each comprehensible and testable, until you get what you need. The whole composed stream would be impossible to understand at once, but each piece can make sense. Trying by ideological fiat to jam it all into one piece or forcing the wrong place to do something is a recipe for disaster. You have to let the problem guide along its solution, or you'll end up wasting effort fighting to impose your beliefs on a system that doesn't care about them at all.)
- ajantha 4y agoNicely summarised and visualised :+1
- alexmerced 4y ago
- hrosen 4y agoHelpful to see a concise comparison!
- alexmerced 4y ago
- evilturnip 4y agoWe're currently looking into datalake implementations. Right now, we only have 1 or 2 data sources. Current thinking is reading them on the fly, combine them using pandas dataframe and query that. Anyone have experience with doing something similar?
- mason55 4y agoDepends on the type of data you're processing, your business goals and the existing consumers that you need to support. At a minimum I'd suggest planning to load the data from the data lake into an RDBMS (OLAP/columnar preferably). Then it's accessible to more than just Python scripts (BI tools, users of other languages, etc). Depending on how much data there is, should also plan on data summarization strategies. You can either build some common rollups to ensure that consumers are all looking at the same summaries or you can let consumers build their own transform/load pipelines from the raw data lake or you can let consumers build their own transform pipelines from the data in the data warehouse (using something like dbt). The benefits of a data lake architecture really appear when you have lots of sources, lots of disparate consumers, and lots of data, with some schema evolution & unstructured parts thrown in. If you only have 1 or 2 sources, small enough data to query raw data in Pandas, and consumers are restricted to Python scripts, then you can skip a lot of the architectural headache of building a data lake for now (just make sure to archive your raw data somewhere if you want to be able to pull it into a data lake in the future).
- tomnipotent 4y agoIt can seem like overkill, but I can't recommend Snowflake enough. It's so simple to setup and manage, and JSON support makes it easy to just drop JSON line files into a table and query on the fly. Depending on your data/query volume, it can also be very cheap.
- chrisjc 4y agoAlso recommend Snowflake! However, this article was about table formats so maybe OP wasn't thinking about JSON. If so, looks like Snowflake only supports read operations atm. Snow Summit this week, so maybe that will change?
- diptnt 4y agoThanks for bringing this comparison out!
- alexmerced 4y ago
- anonymousDan 4y agoHow does the concept of a table here differ from that of a standard relational table (if at all)? Is it that the table is a logical abstraction over a distributed set of files?
- SpicyLemonZest 4y agoThe underlying storage is less structured, which means that some common database optimizations (in particular row-level indexes) are unavailable in these kind of systems, but it's conceptually the same as a standard relational table.
- chrisjc 4y agoThese are effectively external tables. They likely exist on blob storage and not in any particular DB's storage system. As a result, you have a myriad of ways to interact with that data. Moreover, compute and storage resources are now decoupled.
- alexmerced 4y ago
- henrydark 4y agoA major problem with these table formats that will surface soon enough is that they use serial numerical ordering for versions. It's like inventing SVN for data. Soon enough git will have to be invented as well.
- ozkatz 4y agoCheck out lakeFS (https://github.com/treeverse/lakeFS https://github.com/treeverse/lakeFS). It doesn't rely on the object store for ordering and is highly influenced by Git itself (but designed to work on object stores at very large scales)
- mrry 4y ago"Project Nessie: Transactional Catalog for Data Lakes with Git-like Semantics" https://projectnessie.org/ https://projectnessie.org/ This supports lightweight branches, and transactional commits and merges. I haven't used it—and it seems cool—but it also seems a little heavyweight to get cross-table transactions on top of these table formats (which would be my primary use case).
- alexmerced 4y agoProject Nessie also powers Dremio's Arctic service, so you can get all the branching and benefits of Nessie with an intuitive UI to browse branches, create branches and merge branches. Also, it is a cloud managed service with a free tier.
- kristjansson 4y agoSee also https://www.dolthub.com/ https://www.dolthub.com/
- divbzero 4y agoDoes anyone have good real life stories of how data from a data lake made a real difference in a product or a business?
- alexmerced 4y agoYep, in this talk I go over an example from Netflix where Apache Iceberg originated: https://www.youtube.com/watch?v=LiC9vZATv0o&t=8s https://www.youtube.com/watch?v=LiC9vZATv0o&t=8s
- broberts2261 4y agoGreat comparison!