3 ms·

I would love to see the benchmarks. That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). CSV underperfor

by wenc 2y ago

I would love to see the benchmarks. That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). CSV underperforms in almost every other domain, like joins, aggregations, filters. Parquet lets you do that lazily without reading the entire Parquet dataset into memory.

riku_iki 2y ago
> That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). Yes, I think duckdb only reads CSV, then projects necessary data into internal format (which is probably more efficient than parquet, again based on my benchmarks), and does all ops (joins, aggregations) on that format.
- wenc 2y ago
  Yes, it does that, assuming you read in the entire CSV, which works for CSVs that fit in memory. With Parquet you almost never read in the entire dataset and it's fast on all the projections, joins, etc. while living on disk.
  - riku_iki 2y ago
    > which works for CSVs that fit in memory. what? Why CSV is required to fit in memory in this case? I tested CSVs which are far larger than memory, and it works just fine.
    - geysersam 2y ago
      The entire csv doesn't have to fit in memory, but the entire csv has to pass through memory at some point during the processing. The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network.
      - riku_iki 2y ago
        > The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network. this makes sense, and what I hoped to have. But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs.
        wenc 2y ago
        >But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs. Anecdotally having worked with large CSVs and large on-disk Parquet datasets, my experience is the opposite of yours. My DuckDB queries operate directly on Parquet on disk and never load the entire dataset, and is always much faster than the equivalent operation on CSV files. I think your experience might be due to -- what it sounds like -- parsing the entire CSV into memory first (CREATE TABLE) and then processing after. That is not an apples-to-apples comparison because we usually don't do this with Parquet -- there's no CREATE TABLE step. At most there's a CREATE VIEW, which is lazy. I've seen your comments bashing Parquet in DuckDB multiple times, and I think you might be doing something wrong.
        riku_iki 2y ago
        > I think your experience might be due to -- what it sounds like -- parsing the entire CSV into memory first (CREATE TABLE) and then processing after. That is not an apples-to-apples original discussion was about CSV vs parquet "reader" part, so this is exactly apple to apple testing, easy to benchmark and I stand my ground. What you are doing downstream, it is another question which is not possible to discuss because no code for your logic is available. > I've seen your comments bashing Parquet in DuckDB multiple times, and I think you might be doing something wrong. like running one command from DuckDB doc. Also, I am not "bashing", I just state that CSV reader is faster.
        deleted 2y ago
        [deleted]

by wenc 2y ago

riku_iki 2y ago

> That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). Yes, I think duckdb only reads CSV, then projects necessary data into internal format (which is probably more efficient than parquet, again based on my benchmarks), and does all ops (joins, aggregations) on that format.

wenc 2y ago
Yes, it does that, assuming you read in the entire CSV, which works for CSVs that fit in memory. With Parquet you almost never read in the entire dataset and it's fast on all the projections, joins, etc. while living on disk.
- riku_iki 2y ago
  > which works for CSVs that fit in memory. what? Why CSV is required to fit in memory in this case? I tested CSVs which are far larger than memory, and it works just fine.
  - geysersam 2y ago
    The entire csv doesn't have to fit in memory, but the entire csv has to pass through memory at some point during the processing. The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network.
    - riku_iki 2y ago
      > The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network. this makes sense, and what I hoped to have. But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs.
      - wenc 2y ago
        >But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs. Anecdotally having worked with large CSVs and large on-disk Parquet datasets, my experience is the opposite of yours. My DuckDB queries operate directly on Parquet on disk and never load the entire dataset, and is always much faster than the equivalent operation on CSV files. I think your experience might be due to -- what it sounds like -- parsing the entire CSV into memory first (CREATE TABLE) and then processing after. That is not an apples-to-apples comparison because we usually don't do this with Parquet -- there's no CREATE TABLE step. At most there's a CREATE VIEW, which is lazy. I've seen your comments bashing Parquet in DuckDB multiple times, and I think you might be doing something wrong.
        riku_iki 2y ago
        > I think your experience might be due to -- what it sounds like -- parsing the entire CSV into memory first (CREATE TABLE) and then processing after. That is not an apples-to-apples original discussion was about CSV vs parquet "reader" part, so this is exactly apple to apple testing, easy to benchmark and I stand my ground. What you are doing downstream, it is another question which is not possible to discuss because no code for your logic is available. > I've seen your comments bashing Parquet in DuckDB multiple times, and I think you might be doing something wrong. like running one command from DuckDB doc. Also, I am not "bashing", I just state that CSV reader is faster.
        deleted 2y ago
        [deleted]