4 ms·
I wanted to use ibis with Duck as the backend. But I am afraid the chicken and egg problem of using existing ML libraries defaulting to use pandas as the datafr
by mrbonner 2y ago
I wanted to use ibis with Duck as the backend. But I am afraid the chicken and egg problem of using existing ML libraries defaulting to use pandas as the dataframe API. Does anyone have a workaround?
- datadrivenangel 2y agoIt's super easy to switch back and forth between dataframes and duckdb, so there's no lock in?
- leventov 2y agoIt's not super easy to port a library with domain-specific ML/DS algorithms built around Pandas, such as accepting Pandas dataframes as arguments to its functions and class constructors.
- wenc 2y agoDuckDB understands Pandas and Polars dataframes so you can move back and forth seamlessly between them in-memory (with Polars, it's nearly zero-copy via an in-memory data structure). df = pd.DataFrame({"a": [1,2,3], "b": [4,5,6]}) # New Pandas dataframe df2 = duckdb.query("select 2*a as c, b from df") # DuckDB df3 = df2.df() # materialize as Pandas, .pl() for Polars ml_df = ML.func(df3) # pass dataframe into ML function df4 = duckdb.query("select count(distinct x) from ml_df")