4 ms·

An Empirical Evaluation of Columnar Storage Formats [pdf]

77 points by eatonphil 2y ago

jauntywundrkind 2y ago
Nice to see methodology here. Ideally Lancedb lance v2 and nimble would also both be represented here. It feels like there's huge appetite to do better than Parquet; ideally work like this would help inform where we go next. https://blog.lancedb.com/lance-v2/ https://blog.lancedb.com/lance-v2/ https://github.com/facebookincubator/nimble https://github.com/facebookincubator/nimble
- apavlo 2y ago
  Lance v2 looks interesting. I like their meta-data + container story. Lacking SOTA encoding schemes though. There is also Vortex (https://github.com/fulcrum-so/vortex https://github.com/fulcrum-so/vortex). That has modern encoding schemes that we want to use. BtrBlocks (https://github.com/maxi-k/btrblocks https://github.com/maxi-k/btrblocks) from the Germans is another Parquet alternative. Nimble (formerly Alpha) is a complicated story. We worked with the Velox team for over a year to open-source and extend it. But plans got stymied by legal. This was in collaboration with Meta + CWI + Nvidia + Voltron. We decided to go a separate path because Nimble code has no spec/docs. Too tightly coupled with Velox/Folly. Given that, we are working on a new file format. We hope to share our ideas/code later this year.
  - jauntywundrkind 2y ago
    Honored to have your reply, wonderful view of the scene, thanks Andy. 2c remark, zero horses in this race: I was surprised how few encodings were in Nimble at release. The skeleton superficially seemed fine I guess, I don't know, but not much meat on the bones. Without nice interesting optimized encodings, the container for them doesn't feel compelling. But also starting with some inarguable clear options makes some kind of sense too, is some kind of tactic. They claim they're trying to figure out a path to decoupling from Velox/Folly, so hopefully that can come about. I tend to believe so, godspeed. The "implementation not specification" does seem really scary though, isn't how we usually get breakout industry-changimg successes. I wish I had the savy to contrast lance (V2) vs nimble a little better. Both seem to be containerizing systems, allowing streams to define their own encodings. Your comment about meta-data + encodings makes me feel like there's dimensions to the puzzle I haven't identified yet (mostly after chugging VeloxCon talks). (Thanks for everything Andy, you're doing the good work (practicing and informing). Very very excited to see ya'll's alternative!!)
  - intelVISA 2y ago
    Glad to see new ideas in this space, hopefully will be unleashing my own Parquet successor soon(tm).
0cf8612b2e1e 2y ago
Third, faster and cheaper storage devices mean that it is better to use faster decoding schemes to reduce computation costs than to pursue more aggressive compression to save I/O bandwidth. Formats should not apply general-purpose block compression by default because the bandwidth savings do not justify the decompression overhead. Not sure I agree with that. Have a situation right now where I am bottlenecked by IO and not compute.
- RhodesianHunter 2y ago
  Is this because you're using some sort of network backed storage like EBS?
- epistasis 2y ago
  This is extremely common in genomics settings, and in the past I have spent far more time allocating disk iops, network bandwidth, and memory amounts for various pipeline stages than I have on CPUs in this space. Muck up and launch 30x as many processes as your compute node has, and it's fairy fixable, but muck up the RAM allocation and disk IO and you may not be able to fix it in any reasonable time. And if you misallocate your network storage, that can bring the entire cluster to a halt, not just a few nodes.
  - jltsiren 2y ago
    I think the idea is that you should design tools and pipelines to take advantage of current hardware. Individual nodes have more CPU cores, more RAM, and more and faster local storage than they used to. Instead of launching many small jobs that compete for shared resources, you should have large jobs that run the entire pipeline locally, using network and network storage only when it's unavoidable.
    - epistasis 2y ago
      That is exactly right, and optimizing for the current distribution of hardware is always the case; however most interesting problems still do not fit on a single node. For example, large LLMs that whose training data, or sometimes even model itself, do not fit on a single node. Lots of the same principles of allocation show up again.
SPascareli13 2y ago
Love this paper! I read it after watching Dr. Pavlo lessons on YouTube.
gregw2 2y ago
(2023) (VLDB Volume 17 no 2 is 2023)
- lmwnshn 2y ago
  You're correct, but for additional context, this paper will actually be presented at VLDB 2024 [0]. > All papers published in this issue will be presented at the 50th International Conference on Very Large Data Bases, Guangzhou, China, 2024. And that's because in the submission guidelines [1], > The last three revision deadlines will be May 15, June 1, and July 15, 2023. Note that the June deadline is on the 1st instead of the 15th, and it is the final revision deadline for consideration to present at VLDB 2023; submissions received after this deadline will roll over to VLDB 2024. So whether it is (2023) or (2024) is a little ambiguous. [0] https://www.vldb.org/pvldb/vol17/FrontMatterVol17No2.pdf https://www.vldb.org/pvldb/vol17/FrontMatterVol17No2.pdf [1] https://vldb.org/pvldb/volumes/16/submission https://vldb.org/pvldb/volumes/16/submission
  - gregw2 2y ago
    Ah, thanks!
infogulch 2y ago
One of the authors summarized this on Twitter, with a bit of discussion: https://twitter.com/andy_pavlo/status/1790524064995688660?s=19 https://twitter.com/andy_pavlo/status/1790524064995688660?s=...

77 points by eatonphil 2y ago

Nice to see methodology here. Ideally Lancedb lance v2 and nimble would also both be represented here. It feels like there's huge appetite to do better than Parquet; ideally work like this would help inform where we go next. https://blog.lancedb.com/lance-v2/ https://blog.lancedb.com/lance-v2/ https://github.com/facebookincubator/nimble https://github.com/facebookincubator/nimble

apavlo 2y ago
Lance v2 looks interesting. I like their meta-data + container story. Lacking SOTA encoding schemes though. There is also Vortex (https://github.com/fulcrum-so/vortex https://github.com/fulcrum-so/vortex). That has modern encoding schemes that we want to use. BtrBlocks (https://github.com/maxi-k/btrblocks https://github.com/maxi-k/btrblocks) from the Germans is another Parquet alternative. Nimble (formerly Alpha) is a complicated story. We worked with the Velox team for over a year to open-source and extend it. But plans got stymied by legal. This was in collaboration with Meta + CWI + Nvidia + Voltron. We decided to go a separate path because Nimble code has no spec/docs. Too tightly coupled with Velox/Folly. Given that, we are working on a new file format. We hope to share our ideas/code later this year.
- jauntywundrkind 2y ago
  Honored to have your reply, wonderful view of the scene, thanks Andy. 2c remark, zero horses in this race: I was surprised how few encodings were in Nimble at release. The skeleton superficially seemed fine I guess, I don't know, but not much meat on the bones. Without nice interesting optimized encodings, the container for them doesn't feel compelling. But also starting with some inarguable clear options makes some kind of sense too, is some kind of tactic. They claim they're trying to figure out a path to decoupling from Velox/Folly, so hopefully that can come about. I tend to believe so, godspeed. The "implementation not specification" does seem really scary though, isn't how we usually get breakout industry-changimg successes. I wish I had the savy to contrast lance (V2) vs nimble a little better. Both seem to be containerizing systems, allowing streams to define their own encodings. Your comment about meta-data + encodings makes me feel like there's dimensions to the puzzle I haven't identified yet (mostly after chugging VeloxCon talks). (Thanks for everything Andy, you're doing the good work (practicing and informing). Very very excited to see ya'll's alternative!!)
- intelVISA 2y ago
  Glad to see new ideas in this space, hopefully will be unleashing my own Parquet successor soon(tm).

0cf8612b2e1e 2y ago

Third, faster and cheaper storage devices mean that it is better to use faster decoding schemes to reduce computation costs than to pursue more aggressive compression to save I/O bandwidth. Formats should not apply general-purpose block compression by default because the bandwidth savings do not justify the decompression overhead. Not sure I agree with that. Have a situation right now where I am bottlenecked by IO and not compute.

RhodesianHunter 2y ago
Is this because you're using some sort of network backed storage like EBS?
epistasis 2y ago
This is extremely common in genomics settings, and in the past I have spent far more time allocating disk iops, network bandwidth, and memory amounts for various pipeline stages than I have on CPUs in this space. Muck up and launch 30x as many processes as your compute node has, and it's fairy fixable, but muck up the RAM allocation and disk IO and you may not be able to fix it in any reasonable time. And if you misallocate your network storage, that can bring the entire cluster to a halt, not just a few nodes.
- jltsiren 2y ago
  I think the idea is that you should design tools and pipelines to take advantage of current hardware. Individual nodes have more CPU cores, more RAM, and more and faster local storage than they used to. Instead of launching many small jobs that compete for shared resources, you should have large jobs that run the entire pipeline locally, using network and network storage only when it's unavoidable.
  - epistasis 2y ago
    That is exactly right, and optimizing for the current distribution of hardware is always the case; however most interesting problems still do not fit on a single node. For example, large LLMs that whose training data, or sometimes even model itself, do not fit on a single node. Lots of the same principles of allocation show up again.

SPascareli13 2y ago

Love this paper! I read it after watching Dr. Pavlo lessons on YouTube.

gregw2 2y ago

(2023) (VLDB Volume 17 no 2 is 2023)

lmwnshn 2y ago
You're correct, but for additional context, this paper will actually be presented at VLDB 2024 [0]. > All papers published in this issue will be presented at the 50th International Conference on Very Large Data Bases, Guangzhou, China, 2024. And that's because in the submission guidelines [1], > The last three revision deadlines will be May 15, June 1, and July 15, 2023. Note that the June deadline is on the 1st instead of the 15th, and it is the final revision deadline for consideration to present at VLDB 2023; submissions received after this deadline will roll over to VLDB 2024. So whether it is (2023) or (2024) is a little ambiguous. [0] https://www.vldb.org/pvldb/vol17/FrontMatterVol17No2.pdf https://www.vldb.org/pvldb/vol17/FrontMatterVol17No2.pdf [1] https://vldb.org/pvldb/volumes/16/submission https://vldb.org/pvldb/volumes/16/submission
- gregw2 2y ago
  Ah, thanks!

infogulch 2y ago

One of the authors summarized this on Twitter, with a bit of discussion: https://twitter.com/andy_pavlo/status/1790524064995688660?s=19 https://twitter.com/andy_pavlo/status/1790524064995688660?s=...