4 ms·
I can't imagine that `.to_parquet` takes any time at all, relative to `groupby.agg`. But yeah, It would be nice to get seperate benchnmarks for the two parts of
by topper-123 6y ago
I can't imagine that `.to_parquet` takes any time at all, relative to `groupby.agg`. But yeah, It would be nice to get seperate benchnmarks for the two parts of your benchmark.
Maybe one-factor groupbys are faster in pandas, while two-factor groupbys (as in https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/) are slower?
- akhong 6y agoYes, I agree with you in Pandas' case. However, for other libraries, a good chunk of the run time comes from reading the parquet files and concatenating the partial datasets. Pandas and Spark are particularly really good with reading a directory of 12 Parquet files with no noticeable performance penalty.