8 ms·
This could use a bit more "why". Shortcomings of Parquet are mentioned as overcome by this, which ones? Certainly not wide tool support... Why should one leav
by largbae 4mo ago
This could use a bit more "why".
Shortcomings of Parquet are mentioned as overcome by this, which ones? Certainly not wide tool support...
Why should one leave Parquet or ORC for this structure?
- altairprime 4mo agoThe ‘why’ is referenced in the bibliography at the end of the readme; this repo is not meant to be consumed standalone. Start with the paper instead: https://doi.org/10.1145/3749163 https://doi.org/10.1145/3749163
- skrtskrt 4mo agoYeah it seems like most of this can be handled by some more dev hours to Parquet
- dj_axl 4mo agoPaper mentions Parquet, ORC, Nimble, Lance, TSFile, Bullion, and BtrBlocks.
- dietr1ch 4mo agoI also had no idea what they were talking about, but there's good points about how hardware oblivious and somewhat global is Parquet around metadata. I found this post interesting, - https://medium.com/@reliabledataengineering/f3-the-future-proof-file-format-that-finally-gets-it-right-0e7f0ddd2e72 https://medium.com/@reliabledataengineering/f3-the-future-pr...