3 ms·

I've moved from python/polars/pandas to DuckDB and have not looked back

• ismailmaj 2mo ago
  even for just in-memory quick data analysis?
  • krapht 2mo ago
    spinning up a venv with duckdb: 5 minutes using sql everywhere; never having to remember dataframe syntax: timeless
    • rtpg 2mo ago
      what is the visualisation strategy here? I find post-hoc grouping and plotting in pandas pretty satisfying.
      • mrtimo 1mo ago
        Check out malloyyo on github
• viccis 2mo ago
  Same. Almost every time I would use its streaming interfaces in Python, it would STILL materialize everything into memory. That was like 6 months ago. Maybe streaming interfaces actually work, but I found them to be leaky abstractions that required a ton of hand holding to make sure they didn't build a bunch of memory pressure, if you're lucky enough to even have a way to do it. For example, last time I used it, you couldn't do NDJSON streaming scans from S3 (looks like fixed with PR #26563).
• sirfz 2mo ago
  Yep, and chdb as well. Said this before, chdb's DatStore is a pretty neat pandas replacement too.
• nojito 2mo ago
  SQL is a terrible api for data transformation. https://duckdb.org/docs/lts/sql/statements/pivot#limitations https://duckdb.org/docs/lts/sql/statements/pivot#limitations vs https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.pivot.html https://docs.pola.rs/api/python/stable/reference/dataframe/a...