3 ms·
I tried DuckDb once some months ago after reading that it can handle json files. I don’t remember the exact failure details, but it crapped out parsing a json f
by radicality 3y ago
I tried DuckDb once some months ago after reading that it can handle json files. I don’t remember the exact failure details, but it crapped out parsing a json file only few MB large, with some buffer size errors.
What’s the overall opinion of duckdb here?
- wenc 3y agoDuckDB is ok at JSON data but it hasn’t worked around all the corner cases like Pandas has. Converting from a hierarchical format like JSON to a tabular format is always fraught. I find Pandas to be slightly more robust and faster. However for any kind of tabular data, DuckDB blows everything out of the water. I’ve mostly been using it with Parquet and TSV files and the performance is just unbelievable. SQL that even Redshift struggles with can be done in DuckDB in no time. If the data is parquet, there is no need to reach for Spark. DuckDB is just way faster. The level of SQL supported by duckdb is also outstanding, from pivot/unpivot, and friendly SQL constructs like group by all, order by all, it makes writing complicated SQL a pleasure.