I’ve mostly replaced pd.read_csv and pd.read_parquet with duckdb.query(“select * from ‘x.csv’) or duckdb.query(“select * from ‘y/*.parquet’). It’s much faster

I’ve mostly replaced pd.read_csv and pd.read_parquet with duckdb.query(“select * from ‘x.csv’) or duckdb.query(“select * from ‘y/*.parquet’). It’s much faster because DuckDB is vectorized. The result is a Pandas dataframe. Querying the Pandas dataframe from DuckDB is faster than querying it with Pandas itself.