3 ms·
Can we just all converge on Parquet + Arrow and call it a day please? Too much effort being put into 1..N ways to solve a problem that would be better put towar
by andenacitelli 3y ago
Can we just all converge on Parquet + Arrow and call it a day please? Too much effort being put into 1..N ways to solve a problem that would be better put towards a single standard.
We work with Parquet + Arrow every day at $DAYJOB in a ML and Big Data context and it's been great. We don't even think we're using it to its fullest potential, but it's never been the bottleneck for us.
- pdimitar 3y agoHow is the data schema description language btw? I haven't used either yet.
- andenacitelli 3y agoHaven't used it directly myself. We mostly just use it for DataFrame crunching via pandas and/or polars (our usage is mixed) which tends to benefit nicely from columnar access.
- jonbaer 3y agoCheck out DFLib (https://dflib.org https://dflib.org)