4 ms·
do you have more details on how you're doing this? i'd love to implement something like this
by swasheck 3y ago
do you have more details on how you're doing this? i'd love to implement something like this
- Talusmaximus 3y agoseconded.. very interested in this approach as well!
- meitham 3y agoWe have a restful server that accepts odata query. We translate that to sql using sqlalchemy (it’s a python stack). The application is a financial risk system with billions of rows. The query usually fetches data from sqlserver, does some manipulation using pandas dataframes, then serve it as either json or csv. We added duckdb as a cache distributed across many files (a request cannot return data from more than one file) then that very same odata query goes into duckdb. Applies the standard select, filter, group by or pivot and return a dataframe. In most cases duckdb was twice faster than sqlserver. Apologies about any bad grammar/spelling errors, typing from tiny phone in bed
- elmolino89 3y agoYou may shave some time replacing pandas with polars.