4 ms·

Good question! As a disclaimer, I work for DuckDB Labs. There are 2 big benefits to working with Parquet files in DuckDB, and both relate to speed! DuckDB can

Good question! As a disclaimer, I work for DuckDB Labs. There are 2 big benefits to working with Parquet files in DuckDB, and both relate to speed! DuckDB can query parquet right where it sits, so there is no need to insert it into the db first. This is typically much faster. Also, DuckDB's engine is columnar (SQLite is row based), so it can do faster analytical queries using that format. I have seen 20-100x speed improvements over SQLite in analytical workloads. Happy to answer any questions!

arpinum 4y ago
Do you see DuckDB as a possible replacement for AWS Athena? Where would Athena still be better than DuckDB + Parquet + Lambda?
- wenc 4y ago
  DuckDB user here. As far as I can tell, DuckDB doesn’t support distributed computation so you have to set that up yourself, whereas Athena is essentially Presto — it handles that detail for you. It also doesn’t support Avro or Orc yet. DuckDB excels at single machine compute where everything fits in memory or is streamable (data can be local or on S3) — it’s lightweight and vectorized. I use it in Jupyter notebooks and in Python code. But it may not be the right tool if you need distributed compute over a very large dataset.
  - youngtaff 4y ago
    > But it may not be the right tool if you need distributed compute over a very large dataset I’m really interested in what the limits of DuckDB and Parquet are. Can you give me an idea of what size you mean by “a very large dataset”
    - wenc 4y ago
      In the distributed computing world, the rule is you start to scale horizontally when your compute workload is too large to fit in the memory of a single machine. So it depends on your compute workload and your hardware. (There’s no fixed number for what a large dataset is) DuckDB itself doesn’t have any baked in limits. If it fits in memory, single machine compute is usually faster than distributed compute — and DuckDB is faster than Pandas, and definitely faster than local Spark.
- henrydark 4y ago
  I've gone almost completely off Athena in favor of duckdb. Duckdb running with 12 vcpus on mid-size data (1tb) that's on ssd is routinely faster than Athena with the data on s3. If there are a lot of partitions, then the difference is huge.
  - deleted 4y ago
    [deleted]
pmeira 4y ago
Another benefit that is often overlooked is that Parquet support various compression methods, right? SQLite's official support for compression is a commercial product, so basically useless for general usage in free open-source software. There are a few 3rd party alternatives, but it becomes cumbersome. Considering DuckDB's and Parquet's features, if DuckDB manages to support the emerging GeoParquet standard someday, I believe it could be a nice alternative to the typical SQLite + Spatialite combo, and GeoPackage as a whole.

arpinum 4y ago

Do you see DuckDB as a possible replacement for AWS Athena? Where would Athena still be better than DuckDB + Parquet + Lambda?

wenc 4y ago
DuckDB user here. As far as I can tell, DuckDB doesn’t support distributed computation so you have to set that up yourself, whereas Athena is essentially Presto — it handles that detail for you. It also doesn’t support Avro or Orc yet. DuckDB excels at single machine compute where everything fits in memory or is streamable (data can be local or on S3) — it’s lightweight and vectorized. I use it in Jupyter notebooks and in Python code. But it may not be the right tool if you need distributed compute over a very large dataset.
- youngtaff 4y ago
  > But it may not be the right tool if you need distributed compute over a very large dataset I’m really interested in what the limits of DuckDB and Parquet are. Can you give me an idea of what size you mean by “a very large dataset”
  - wenc 4y ago
    In the distributed computing world, the rule is you start to scale horizontally when your compute workload is too large to fit in the memory of a single machine. So it depends on your compute workload and your hardware. (There’s no fixed number for what a large dataset is) DuckDB itself doesn’t have any baked in limits. If it fits in memory, single machine compute is usually faster than distributed compute — and DuckDB is faster than Pandas, and definitely faster than local Spark.
henrydark 4y ago
I've gone almost completely off Athena in favor of duckdb. Duckdb running with 12 vcpus on mid-size data (1tb) that's on ssd is routinely faster than Athena with the data on s3. If there are a lot of partitions, then the difference is huge.
- deleted 4y ago
  [deleted]

pmeira 4y ago

Another benefit that is often overlooked is that Parquet support various compression methods, right? SQLite's official support for compression is a commercial product, so basically useless for general usage in free open-source software. There are a few 3rd party alternatives, but it becomes cumbersome. Considering DuckDB's and Parquet's features, if DuckDB manages to support the emerging GeoParquet standard someday, I believe it could be a nice alternative to the typical SQLite + Spatialite combo, and GeoPackage as a whole.