6 ms·

Shakti (the latest K implementation by the author of K) claims [1] to load a 50gb csv in 1.6 seconds which according to them takes 265 seconds with Polars. Has

by rustc 2y ago

Shakti (the latest K implementation by the author of K) claims [1] to load a 50gb csv in 1.6 seconds which according to them takes 265 seconds with Polars. Has anyone independently verified these claims? Is Polars really leaving 2 orders of magnitude performance on the table? [1]: https://shakti.com/ https://shakti.com/ -> Compare -> h2o.k

bear8642 2y ago
> [1]: https://shakti.com/ https://shakti.com/ -> Compare -> h2o.k You can link to the subsections: https://shakti.com/compare/h2o.k https://shakti.com/compare/h2o.k
orlp 2y ago
Disclaimer: I work for Polars inc. As a sanity check I just cloned https://github.com/h2oai/db-benchmark https://github.com/h2oai/db-benchmark, ran the data generation script and ran on a 64 core AMD EPYC (AWS c7a.16xlarge): import polars as pl lf = pl.scan_csv("G1_1e9_1e2_0_0.csv") print(lf.select(pl.col.v1.sum()).collect()) The above script ran in 7.58 seconds. If I change the collect() to collect(new_streaming=True) to use the new streaming engine I've been working on, it runs in 6.90 seconds. I can't realistically time the full "read CSV to memory" with this 50 GB file on this machine as we start swapping (this machine has 128GiB memory) and/or evicting data from disk cache (this machine has a slow EC2 SSD attached to it), so we do have a blow-up of memory usage (which could be as simple as loading small integers into an 8-byte Uint64 column). I think it's likely that on K's machine the "read full CSV to memory" approach also started swapping, giving the large runtime. However, in Polars you'd typically write your query using LazyFrames, which means we don't actually have to load the full CSV into memory. EDIT: running on a m7a.16xlarge with twice the memory (256GiB) once the CSV file is in disk cache Polars can parse the full CSV file into an in-memory dataframe in 7.68 seconds. K's claim that it parses the full 50GB CSV in 1.6 seconds if true is very impressive regardless.
- mananaysiempre 2y ago
  Honestly 7 seconds even just to parse the CSV is already pretty impressive, 7GB/s would be simdjson speeds if you did it on a single core. Do you have a single-threaded parser with really well-tuned SIMD, or a speculative parallel one, or ..?
  - orlp 2y ago
    We have a single-threaded chunker that scans serially over the file. This chunker exclusively finds unquoted newlines (using SIMD) to find clean parallelization boundaries, it doesn't do any further parsing. Those parallelization boundaries are then used to feed worker threads chunks of data to properly parse into our in-memory representation (which mostly follows Arrow).
    - LegionMammal978 2y ago
      Would you know how much of the total runtime is devoted to the initial chunking process? Amdahl's law would prefer an entirely speculative approach in the limit, but I could imagine that the 2x overhead might not be worth it for reasonable file sizes and core counts. (But even then, 1.6 s would be quite a feat. It makes me wonder if the K implementation is partially lazy, as you say typical Polars usage is.)
      - orlp 2y ago
        It seems from a profile that on the eager engine the serial scanner is able to feed ~32 threads worth of decoding: https://share.firefox.dev/4hS1eJa https://share.firefox.dev/4hS1eJa. It might be worth speculating, or at least optimizing the serial chunker more. You could theoretically start a second serial chunker from the end working backwards but that would not be wise with our ordered streams, as the decoded data would have to be buffered for a long time. Similarly on the new streaming engine, each thread is active ~half of the time, except the thread running the chunking task: https://share.firefox.dev/3WQV9og https://share.firefox.dev/3WQV9og. Note that in a lot of realistic workloads on the streaming engine compute can happen in between decodes, completely hiding the bottleneck. Also all of the above is with the file being completely in file cache, if fed from a slow SSD it's not a bottleneck whatsoever.

by rustc 2y ago

bear8642 2y ago

> [1]: https://shakti.com/ https://shakti.com/ -> Compare -> h2o.k You can link to the subsections: https://shakti.com/compare/h2o.k https://shakti.com/compare/h2o.k

orlp 2y ago

Disclaimer: I work for Polars inc. As a sanity check I just cloned https://github.com/h2oai/db-benchmark https://github.com/h2oai/db-benchmark, ran the data generation script and ran on a 64 core AMD EPYC (AWS c7a.16xlarge): import polars as pl lf = pl.scan_csv("G1_1e9_1e2_0_0.csv") print(lf.select(pl.col.v1.sum()).collect()) The above script ran in 7.58 seconds. If I change the collect() to collect(new_streaming=True) to use the new streaming engine I've been working on, it runs in 6.90 seconds. I can't realistically time the full "read CSV to memory" with this 50 GB file on this machine as we start swapping (this machine has 128GiB memory) and/or evicting data from disk cache (this machine has a slow EC2 SSD attached to it), so we do have a blow-up of memory usage (which could be as simple as loading small integers into an 8-byte Uint64 column). I think it's likely that on K's machine the "read full CSV to memory" approach also started swapping, giving the large runtime. However, in Polars you'd typically write your query using LazyFrames, which means we don't actually have to load the full CSV into memory. EDIT: running on a m7a.16xlarge with twice the memory (256GiB) once the CSV file is in disk cache Polars can parse the full CSV file into an in-memory dataframe in 7.68 seconds. K's claim that it parses the full 50GB CSV in 1.6 seconds if true is very impressive regardless.

mananaysiempre 2y ago
Honestly 7 seconds even just to parse the CSV is already pretty impressive, 7GB/s would be simdjson speeds if you did it on a single core. Do you have a single-threaded parser with really well-tuned SIMD, or a speculative parallel one, or ..?
- orlp 2y ago
  We have a single-threaded chunker that scans serially over the file. This chunker exclusively finds unquoted newlines (using SIMD) to find clean parallelization boundaries, it doesn't do any further parsing. Those parallelization boundaries are then used to feed worker threads chunks of data to properly parse into our in-memory representation (which mostly follows Arrow).
  - LegionMammal978 2y ago
    Would you know how much of the total runtime is devoted to the initial chunking process? Amdahl's law would prefer an entirely speculative approach in the limit, but I could imagine that the 2x overhead might not be worth it for reasonable file sizes and core counts. (But even then, 1.6 s would be quite a feat. It makes me wonder if the K implementation is partially lazy, as you say typical Polars usage is.)
    - orlp 2y ago
      It seems from a profile that on the eager engine the serial scanner is able to feed ~32 threads worth of decoding: https://share.firefox.dev/4hS1eJa https://share.firefox.dev/4hS1eJa. It might be worth speculating, or at least optimizing the serial chunker more. You could theoretically start a second serial chunker from the end working backwards but that would not be wise with our ordered streams, as the decoded data would have to be buffered for a long time. Similarly on the new streaming engine, each thread is active ~half of the time, except the thread running the chunking task: https://share.firefox.dev/3WQV9og https://share.firefox.dev/3WQV9og. Note that in a lot of realistic workloads on the streaming engine compute can happen in between decodes, completely hiding the bottleneck. Also all of the above is with the file being completely in file cache, if fed from a slow SSD it's not a bottleneck whatsoever.