codyvoda

6 ms·

31.
by codyvoda 2y ago
were you using the pandas backend in Ibis and pitching that to your company? you'd have much better performance and efficiency using the default backend, even if you used pandas for data input and output (though the DuckDB data readers
32.
by codyvoda 2y ago
> If Ibis moves away from pandas, then it just means that I am less likely to try Ibis because there is no bridge. the bridge is that Ibis accepts pandas as input and has a `to_pandas()` method as output Ibis also still depends on pandas
33.
by codyvoda 2y ago
fun fact if you weren't aware, Ibis was also originally created by Wes McKinney and takes a lot of inspiration from the tidyverse
34.
by codyvoda 2y ago
Polars is a great choice but, like pandas, locks you into its execution engine(s). Polars is a supported backend for Ibis, though depending on your use case DuckDB or DataFusion may scale better. we did some benchmarking recently: https:&#
35.
by codyvoda 2y ago
Great question, a bunch of reasons: 1. The value prop of Ibis is to avoid vendor lock-in and allow users to more easily swap out execution engines that meet their needs. So it's valuable to the project to have "competing" bac
36.
by codyvoda 2y ago
Ibis still supports over a dozen backends, including DuckDB and DataFusion and Polars (all great single-node OLAP query engine options that are often faster than trying something distributed) and PySpark, Snowflake, BigQuery, Trino, ClickHo
37.
by codyvoda 3y ago
the "check back in a month" soon. I have versions of it that work but I just haven't been satisfied with. also, the major underlying dependency (Marvin) is going through a large refactor for v2. once that stabilizes a bit, I&
38.
by codyvoda 3y ago
note that Ibis Birdbrain is very much work-in-progress, but should provide an open-source solution to do this w/ 20+ backends old demo here: https://gist.github.com/lostmygithubaccount/08ddf29898732101... planning
39.
by codyvoda 3y ago
Wes also literally created another Python dataframe project, Ibis, to overcome many of the issues with pandas https://ibis-project.org most data engines and dataframe tools these days use Apache Arrow, it's a bit orthogonal
40.
by codyvoda 3y ago
fairly distinct, there is of course some overlap. you could technically do (mostly?) everything with tensors that you could with dataframes, but generally dataframes are for analyzing and transforming data for ETL/analytic workloads. t
41.
by codyvoda 3y ago
the creator of pandas created Ibis, which has a Postgres backend, for reasons like this: https://ibis-project.org/backends/postgresql this is a better approach to Python dataframes and SQL
42.
by codyvoda 3y ago
Ibis, a Python dataframe created by the creator of pandas, uses DuckDB as the default backend and generally beats Polars on these benchmarks (with exceptions on some queries)