4 ms·

No, it’s not just API layer, but significant addon on top of existing tech… It brings transactions, faster queries, time travel, etc.

riku_iki 4y ago
Maybe, but "infinitely scalable storage" part is because of lower lever layer.
- Epa095 4y ago
  If course, but there are many ways one could make the storage format such that it would not scale nicely with the underlying layer.
  - riku_iki 4y ago
    Like how?
FridgeSeal 4y ago
It’s a parquet file, with some json as metadata to define the version boundaries. That’s all. You too can have transactions if you implement your access layer appropriately. It’s a useful format, but let’s note pretend it’s any more magic than it actually is. I’ve also not noticed any improved performance above what partitioning would give you.
- Epa095 4y ago
  >You too can have transactions if you implement your access layer appropriately. Yes, we can all "just" make it ourself, still nice when others make it. Personally I very much like that it's conseptually easy and that it builds on an open source format (parquet). I also expect there to be a dragons, both small and large, in actually getting the acid stuff right,so I am happy we can fight them together.
- alexott 4y ago
  Regarding performance- try version 1.2.x or higher - it includes data skipping, etc. from what was Databricks-only implementation before. Regarding “that’s all” - can you give an estimate how much time it will take to reimplement it from scratch on multiple cloud platforms?
- manish_gill 4y ago
  Delta metadata based min/max statistics, Parquet row groups and min/max in-file, combined with the right combination of bucketing and sorting, allowed us to actually ignore all partitioning and drastically improve performance on PB scale. The metadata can be a very useful way to do indexing/allows you to skip a lot of things. It's not magic, and you can understand what the _delta_log is doing fairly easily, but I can testify that the performance improvement over partitioning can be achieved.
  - alexott 4y ago
    And some queries, like, max(col) could be answered without touching the data…

riku_iki 4y ago

Maybe, but "infinitely scalable storage" part is because of lower lever layer.

Epa095 4y ago
If course, but there are many ways one could make the storage format such that it would not scale nicely with the underlying layer.
- riku_iki 4y ago
  Like how?

FridgeSeal 4y ago

It’s a parquet file, with some json as metadata to define the version boundaries. That’s all. You too can have transactions if you implement your access layer appropriately. It’s a useful format, but let’s note pretend it’s any more magic than it actually is. I’ve also not noticed any improved performance above what partitioning would give you.

Epa095 4y ago
>You too can have transactions if you implement your access layer appropriately. Yes, we can all "just" make it ourself, still nice when others make it. Personally I very much like that it's conseptually easy and that it builds on an open source format (parquet). I also expect there to be a dragons, both small and large, in actually getting the acid stuff right,so I am happy we can fight them together.
alexott 4y ago
Regarding performance- try version 1.2.x or higher - it includes data skipping, etc. from what was Databricks-only implementation before. Regarding “that’s all” - can you give an estimate how much time it will take to reimplement it from scratch on multiple cloud platforms?
manish_gill 4y ago
Delta metadata based min/max statistics, Parquet row groups and min/max in-file, combined with the right combination of bucketing and sorting, allowed us to actually ignore all partitioning and drastically improve performance on PB scale. The metadata can be a very useful way to do indexing/allows you to skip a lot of things. It's not magic, and you can understand what the _delta_log is doing fairly easily, but I can testify that the performance improvement over partitioning can be achieved.
- alexott 4y ago
  And some queries, like, max(col) could be answered without touching the data…