3 ms·

Whenever I see DuckDB, it's the perfect one needed for OLAP queries in terms of performance and efficiency. One thing I couldn't wrap my head around, why there

Whenever I see DuckDB, it's the perfect one needed for OLAP queries in terms of performance and efficiency. One thing I couldn't wrap my head around, why there is no standalone version? What would be typical data pipeline/architecture to use DuckDB as in process OLAP engine? How would it achieve storage of data (large data volume)

vladsanchez 3y ago
With Apache Arrow. https://duckdb.org/2023/08/04/adbc.html https://duckdb.org/2023/08/04/adbc.html
chrisjc 3y ago
> One thing I couldn't wrap my head around, why there is no standalone version? Standalone version of what? DuckDB? If you mean DuckDB, there is: https://duckdb.org/docs/archive/0.8.1/installation/index https://duckdb.org/docs/archive/0.8.1/installation/index > How would it achieve storage of data (large data volume) - There are so many ways you can do this that you end up going down many rabbit holes... - Locally on your own machine's local storage. - On AWS you could use EBS, EFS, expandable /tmp, S3... - On a Data Lake you could use parquet/json/csv/etc, hive, and in version 0.9.0 (Sept 11th) you could use Apache Iceberg. And probably so many other options I haven't even thought or read about...