10 ms·

Farewell Pandas, and thanks for all the fish

228 points by nojito 2y ago

joelschw 2y ago
Huge fan of Ibis, the value isn't that you can now use DuckDB... it's that your syntax will work when the next cool thing arrives too
log4shell 2y ago
Its great to have a single entrypoint for multiple backends. What I am trying to understand and couldn't find much information related to: How does the use of multiple engines in Ibis impact the consistency of results for the same input and query, particularly in relation to semantic differences among the engines?
softwaredoug 2y ago
One thing I do like about pandas is it’s pretty extensible to columns of new types. Maybe I’m missing something, but does Polars allow this? Last time I checked there wasn’t a clear path forward.
- datadrivenangel 2y ago
  Ibis allows this with custom datatypes.
- porker 2y ago
  I'm curious, when do you want to add a new column type? It's not a situation I've been in.
  - softwaredoug 2y ago
    GeoPandas is one popular library (https://geopandas.org/en/stable/ https://geopandas.org/en/stable/) I added a column type for full text search, searching tokenized text (https://github.com/softwaredoug/searcharray https://github.com/softwaredoug/searcharray) The pandas support for extensions is very robust with extensive testing to make sure you can do all the data aggregations functionality you expect in and pandas column.
- deadlypointer 2y ago
  OTOH the list and struct types allow lots of complex operations, e.g with `.eval`, which run blazingly fast.
maxdo 2y ago
curious to know what they use instead of Dask ?
- datadrivenangel 2y ago
  DuckDB. It's faster on a single computer than Dask.
  - codyvoda 2y ago
    Ibis still supports over a dozen backends, including DuckDB and DataFusion and Polars (all great single-node OLAP query engine options that are often faster than trying something distributed) and PySpark, Snowflake, BigQuery, Trino, ClickHouse, Druid, and more plenty of options for fast and scalable computation. in the fullness of time, it would be great to have a Dask backend but it'd need to be rewritten to use dask-expr, which isn't something the core Ibis team has the capacity for
  - lopatin 2y ago
    As a potential user of Ibis, how do I know they won't eventually drop support for other backends until it just becomes a frontend for DuckDB?
    - codyvoda 2y ago
      Great question, a bunch of reasons: 1. The value prop of Ibis is to avoid vendor lock-in and allow users to more easily swap out execution engines that meet their needs. So it's valuable to the project to have "competing" backends supported (e.g. DuckDB, DataFusion, and Polars for local, and Snowflake, ClickHouse, PySpark, Databricks, etc. for distributed) -- the idea is a standard like Apache Arrow that everyone adopts to stop this cycle of tightly coupling a dataframe interface to an execution engine 2. Ibis is primarily supported by Voltron Data (my employer) which is itself a vendor, so the real worry would be that Ibis drops all backends until it only support Voltron Data's engine (Theseus)... 3. ... but Voltron Data cannot do that for a number of reasons. First, Ibis is an independently governed open source project. Second, it would not make any business sense. You can read some more on that here: https://ibis-project.org/posts/why-voda-supports-ibis https://ibis-project.org/posts/why-voda-supports-ibis. The general idea is to add more backends, not remove them, but pandas is a special case where it introduces a lot of technical complexity with (arguably) negative benefit to potential users -- the DuckDB backend can do everything the pandas backend can do, but better
deleted 2y ago
[deleted]
spywaregorilla 2y ago
What is this exactly? The front page doesn't make it clear to make what ibis is besides being an alternative to pandas/polars.
- prepend 2y ago
  Check the project’s left nav, https://ibis-project.org/why https://ibis-project.org/why I didn’t know either. It’s a python dataframe library/api.
- smu3l 2y ago
  Never heard of ibis before but the front page give a pretty good overview of what it is to me at least. Looks like a dataframes api that can be executed on a variety of backend engines, both distributed and local. Somewhat analogous to how the pandas api can be used in pyspark via pyspark pandas, but from the api -> implementation direction rather than (pandas) implementation -> api -> implementation maybe?
- giovannibonetti 2y ago
  As far as I can tell, it is an ORM for data engineers. They can write Python code that gets translated to either SQL or some other language understood by the DB engine that actually runs it.
  - cpcloud 2y ago
    Ibis is not an ORM. It has no opinion of or knowledge about your data model, or relationships between tables and their columns. Check out SQLAlchemy's ORM (https://docs.sqlalchemy.org/en/20/orm/ https://docs.sqlalchemy.org/en/20/orm/) for what that looks like.
infecto 2y ago
I will try it out but honestly the pain-point for me is the library api to pandas is not always intuitive/natural to how things should be in Python. The NaN/None bit is annoying but I find that to be a minor annoyance.
mananaysiempre 2y ago
> NULL indicates a missing value, and NaN is Not a Number. That’s actually less true than it sounds. One of the primary functions of NaN is to be the result of 0/0, so there it means that there could be a value but we don’t know what it is because we didn’t take the limit properly. One of the primary functions of NULL is to say that a tuple satisfies a predicate except we don’t know what this one position is—it’s certainly is something out in the real world, we just don’t know what. These ideas are what motivates the comparison shenanigans both NaN and NULL are known for. There’s certainly an argument to be made that the actual implementation of both of these ideas is half-baked in the respective standards, and that they are half-baked differently so we shouldn’t confuse them. But I don’t think it’s fair to say that they are just completely unrelated. If anything, it’s Python’s None that’s doesn’t belong.
- cpcloud 2y ago
  100% this. We'd love to have a docs contribution that lays this out in detail if you'd be up for it! Disclaimer: (I lead the project and work on it full time).
- aeonik 2y ago
  Why don't we build a portable numeric system that just has these mathematical constants and definitions built in, in a portable and performance manner?
  - verandaguy 2y ago
    We just don’t have the technology for that, else there’d be libraries floating around that let you do that.
- miohtama 2y ago
  Related to this, below someone posted a link to the blog post of Wes McKinney where he discussed Pandas limitations and how PyArrow works around these > 4. Doing missing data right > All missing data in Arrow is represented as a packed bit array, separate from the rest of the data. This makes missing data handling simple and consistent across all data types. You can also do analytics on the null bits (AND-ing bitmaps, or counting set bits) using fast bit-wise built-in hardware operators and SIMD. > The null count in an array is also explicitly stored in its metadata, so if data does not have nulls, we can choose faster code paths that skip null checking. With pandas, we cannot assume that arrays do not have null sentinel values and so most analytics has extra null checking which hurts performance. If you have no nulls, you don’t even need to allocate the bit array. > Because missing data is not natively supported in NumPy, over time we have had to implement our own null-friendly versions of most key performance-critical algorithms. It would be better to have null-handling built into all algorithms and memory management from the ground up. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/
ironmagma 2y ago
Man, these frontend people are constantly churning through dependencies, a new framework every year. Don't reinvent the wheel, just be like backend folks and pick a dependency and stick with it. Oh wait...
- vundercind 2y ago
  ETL and data analysis “notebook” world been warming up for years and now it’s red-hot due to every company down to Grandma’s Cookie Bakery LLC wanting to “do AI”. And most of the standard tools are surprisingly terrible so churn is increasing for reasons similar to why that happened in frontend (the underlying tools are ass and the existing interfaces bad and almost all of it’s being overcomplicated for bad reasons and a million more developers just jumped into the pool)
  - ironmagma 2y ago
    Yes, and so really what we are seeing is just the effects of a high quality bar. If you want high-quality software, you do more iterations. And iterations produce breaking changes.
    - vundercind 2y ago
      Mmm. A lot of it’s motivated by pain, with the output filtered through a web of organizational friction, principal agent problems, self-promotional nonsense, and other harmful factors. If the web’s any indication, there’s no guarantee this activity is efficient at getting us closer to meeting a high quality bar.
- draw_down 2y ago
  [dead]
- dr_mee6 2y ago
  Just as an FYI Ibis is being around for 10 years approximately...
frakt0x90 2y ago
Curious if you considered Polars. That's become the defacto standard in my group as we all dislike pandas.
- cpcloud 2y ago
  We support Polars as a backend. DuckDB was chosen as the default because Polars was too unstable to be the default backend two years ago.
- amelius 2y ago
  Good riddance. Pandas turns 10x programmers into 1x programmers.
  - data-ottawa 2y ago
    I've been using pandas for over a decade and I'm very efficient with it. The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it. I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style. It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing. I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.
    - codyvoda 2y ago
      fun fact if you weren't aware, Ibis was also originally created by Wes McKinney and takes a lot of inspiration from the tidyverse
      - data-ottawa 2y ago
        I was not aware of that, the influence from the tidyverse is very clear I’ve tried to make mockups of similarly inspired API and I think they did a great job with clever tricks like the `_` reference (does that clash with IPython/Jupyter’s use?). I do wonder how they manage compatibility with so many backends, it seems like many features will be directly tied to your backend (e.g. Trino using Java Pattern for regex vs BigQuery using Re2) in hard to explain ways. But maybe that’s not a big concern, because very often you’re only going to be using a couple of backends. I’ll have to try Ibis out for myself, it looks like it can unify a lot of the work I have to do. I’ve moved away from pandas for all but the last mile of computation, so this might be a direct good SQL replacement.
mrbluecoat 2y ago
Dask was really cool back in the day. Farewell, friend.
- lmc 2y ago
  Back in the day? What are people using now instead?
  - ZeroCool2u 2y ago
    Ray seems to be a popular choice now if you need multi node. But Polars is really nice if you're okay with just vertically scaling on one machine.
  - cpcloud 2y ago
    I've been working with pencil and paper recently. Thinking about starting an Ibis backend for it.
  - dr_mee6 2y ago
    Abacuses?
thsgtu 2y ago
About time. It always surprises me how long pandas has been able to hold on. Wes McKinney talked about pandas' limitations back in 2017 https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/
__mharrison__ 2y ago
Not surprising. There are much better compute engines than pandas. Folks then ask why not jump from pandas to [insert favorite tool]? - Existing codebases. Lots of legacy pandas floating about. - Third party integration. Everyone supports pandas. Lots of libraries work with tools like Polars, but everything works with pandas. - YAGNI - For lots of small data tasks, pandas is perfectly fine.
- cpcloud 2y ago
  All great reasons to stick with Pandas. If a thing works for you, then that's one less thing to manage. Fully agree that if pandas is working for you, then you're better off sticking with it!
- whimsicalism 2y ago
  If your task is too big for pandas, you should probably skip right over dask and polars for a better compute engine.
  - __mharrison__ 2y ago
    Jump straight to cluster and skip Dask? Not sure what "big" means here, but a combination of .pipe, pyarrow, and polars can speed up many slow Pandas operations. Polars streaming is surprisingly good for larger than RAM. I get that clusters are cool, but I prefer to keep it on a single machine if possible. Also, libraries like cudf can greatly speed up Pandas code on a single machine, while Snowpark can scale Pandas code to Snowflake scale.
    - cpcloud 2y ago
      In my experience, Polars streaming runs out of memory at much smaller scales than both DuckDB and DataFusion and tends to use much more memory for the same workload when it doesn't outright segfault. Polars is faster than those two once you get to less than a few GB, but beyond that you're better off with DuckDB or DataFusion. I would love for this to improve in Polars, and I'm sure it will!
mrguyorama 2y ago
Pandas made me think I hated python.
- e10v_me 2y ago
  Funny, the same) When I was switching from R (data.table) to Python, it was painful. Not only because it was slow, but because of the API. At that time, I thought that maybe it's because of switching to something new. Several years later, switching from Pandas to Polars API was a real joy (Ibis is also good is that sense). So, I learned that it had been Pandas fault all along))
whimsicalism 2y ago
Frankly, pandas/dask/polars - all are trying to recreate something that has existed for years (ie. sql, spark) and with a terrible API. I truly thought I was terrible at python for a long time because of pandas - turns out it just has an absolutely terrible API surface
- tech_ken 2y ago
  I would not say pandas is a recreation of SQL/Spark, they have very different use-cases in my experience. SQL/Spark is like a bulk data management tool: I use it if I need to load massive data from some remote store, perform light preprocessing, join up a couple dimension tables, etc. Having normalized and joined my data, then pandas enters as a 'last-mile' processing engine, particularly when paired with ex. SKLearn or whatever other inference lib you're using. Pandas is awesome if you need to ex. apply string manipulation to a data table or daisy-chain some complicated computations together. Honestly in my opinion the API is really nice, I came over from R tidyverse and the 'chained methods' approach to pandas let's me carry over all my old patterns and paradigms. I find it far easier to use that approach than having to write like 20 dependent subqueries or staging tables
- cmollis 2y ago
  true spark has existed for years and is a great toolset.. i use it ever day. it's also a huge hassle spinning clusters up and down and configuration is complex. I can execute some pretty hairy scans against a huge s3 parquet dataset in Duckdb that I would typically have to run in either spark or athena.. it's a little slower, but not ridiculously slower. And, it does all of that from my desktop.. no clusters, no mem or task configs.. just run the query. Being able to integrate all of the expensive historical scanning and knitting that back into an ML pipeline with desktop python is pretty nice.
dammaj 2y ago
Personnally, I tend to use Pandas because it is integrated everywhere, because of the ecosystem that uses it. Let's say I want to read data from json file (csv file, python dict, etc.) and I want to plot it using plotly. If Ibis is compatible with whatever Pands dataframe is compatible with, then for most of my usage I don't really care much about the "backend".
kremi 2y ago
Pandas has been working fine for me. The most powerful feature that makes me stick to it is the multi-index (hierarchical indexes) [1]. Can be used for columns too. Not sure how the cool new kids like polars or ibis would fare in that category. [1] https://pandas.pydata.org/docs/user_guide/advanced.html#advanced-indexing-with-hierarchical-index https://pandas.pydata.org/docs/user_guide/advanced.html#adva...
- cpcloud 2y ago
  Multi-indexes definitely have their place. In fact, I got involved in pandas development in 2013 as part of some work I was doing in graduate school, and I was a heavy user of multi-indexed columns. I loved them. Over time, and after working on a variety of use cases, I personally have come to believe the baggage introduced by these data structures wasn't worth it. Take a look at the indexing code in pandas, and the staggering complexity of what's possible to put inside square brackets and how to decipher its meaning. The maintenance cost alone is quite high. We don't plan to ever support multi-indexed rows or columns in Ibis. I don't think we'd fare well _at all_ there, intentionally so.
  - kremi 2y ago
    > Take a look at the indexing code in pandas As the end-user, not quite my concern. > and the staggering complexity of what's possible to put inside square brackets and how to decipher its meaning I might not be aware of everything that's possible -- the usage I have of it doesn't give me an impression of staggering complexity. In fact I've found the functionality quite basic, and have been using pd.MultiIndex.from_* quite extensively for anything slightly more advanced than selecting a bunch of values at some level of the index.
    - hansvm 2y ago
      > As the end-user, not quite my concern. Complicated code is (probabilistically) slow, buggy, infrequently updated code. By all means, if it looks like a good enough tool for the job (especially if the alternatives don't) then use it anyway, but that's slightly different from it not being your concern. I've seen enough projects need "surprise" major revisions because some team tried to sneak a dataframe into a 10M QPS service that my default is keeping pandas far away from anything close to a user-facing product. I've also seen costs balloon as the data's scale grows beyond what pandas can handle, but basically all the alternatives suck for myriad reasons, so I don't try to push "not pandas" in the data backend. People can figure out what works for themselves, and I kind of like just writing it from scratch in a performant language when I personally hit that bottleneck.
UncleOxidant 2y ago
Related to this pandas?: https://pandas.pydata.org/ https://pandas.pydata.org/
- cpcloud 2y ago
  On the contrary, very related to that pandas!
Kalanos 2y ago
I've only heard about Ibis maybe three times in the past two years and I pay pretty close attention to the space. If Ibis moves away from pandas, then it just means that I am less likely to try Ibis because there is no bridge. Sure, hip new frameworks are moving away from pandas/numpy, but I'll wait 5 years for the dust to settle here while the compatibility and edge cases sort themselves out. The pydata/numfocus ecosystem is extensive. It's just tabular data. So what if I have to wait a few more milliseconds to get my result.
- cpcloud 2y ago
  Sounds like a solid plan. No reason to incur switching costs if things are working! I think you're misunderstanding what "removing pandas" means. You can still compute on DataFrames, it'll just be with something other than Pandas itself, probably DuckDB, Polars, or DataFusion. So, the bridge was there, is there, and isn't going anywhere.
  - deleted 2y ago
    [deleted]
  - halfcat 2y ago
    > I think you're misunderstanding what "removing pandas" means. Maybe it’s in the messaging, like the “Farewell Pandas” part?
    - cpcloud 2y ago
      Feel free to put up a pull request with an alternate title that you feel communicates the message in a more accurate way!
- codyvoda 2y ago
  > If Ibis moves away from pandas, then it just means that I am less likely to try Ibis because there is no bridge. the bridge is that Ibis accepts pandas as input and has a `to_pandas()` method as output Ibis also still depends on pandas (and thus numpy) internally. also, Ibis was created by the creator of pandas, and the lead developer (and other contributors) have also worked on pandas. the Ibis team understands pandas and the broader Python data ecosystem very well > It's just tabular data. So what if I have to wait a few more milliseconds to get my result. usually as scale grows, milliseconds -> seconds -> minutes -> hours -> days. at some point along the way having a dataframe library that can scale up without rewriting your code might be useful. but if you're dealing with small tabular data and pandas meets your needs, it's a great library to use and stick with!
riezebos 2y ago
Nice to see, over the past months I've replaces pandas with ibis in all new projects and I am a huge fan! - Syntax in general feels more fluid than pandas - Chaining operations with deferred expressions makes code snippets very portable - Duckdb backend is super fast - Community is very active, friendly and responsive I'm trying to promote it to all my peers but it's not a very well known project in my circles. (Unlike Polars which seems to be the subject of 10% of the talks at all Python conferences)
glial 2y ago
In my experience, the best thing about Pandas is how much it made me appreciate using dplyr and the tidyverse. If it wasn't for Pandas, I may not be the avid R user I am today.
antman 2y ago
"TL; DR: we are deprecating the pandas and dask backends and will be removing them in version 10.0. There is no feature gap between the pandas backend and our default DuckDB backend" There is an important feature gap, pandas has all the users and no one knows you guys. Was trying to pitch this to my company but I guess its goodbye Ibis and thanks for all the fish.
- codyvoda 2y ago
  were you using the pandas backend in Ibis and pitching that to your company? you'd have much better performance and efficiency using the default backend, even if you used pandas for data input and output (though the DuckDB data readers are generally more efficient too). I'd be happy to help you on the pitch! and if you were using the pandas backend or know of many users doing so, please reach out! we love feedback from our users and would be sad to see y'all go, we're open to reconsidering any decisions based on feedback
  - antman 2y ago
    Yes I agree and have tested it, my point is that marketing wise it is a shame. I will even think that when looking into new tooling "pandae" compatibility as a search term and incentive would be far ahead the word "dataframe"
    - codyvoda 2y ago
      we will take this excellent marketing advise under advisement, thank you for your consideration
    - cpcloud 2y ago
      Pandas compatibility as input and output will remain. Not sure what's in doubt here.
zhangyt26 2y ago
How does ibis manage ordering of the data frame for SQL based engine? One big gap that I noticed is that pandas retain ordering between operations, bust most SQL based engine doesn’t give you that guarantee. Looking at duckdb, it looks they only do that for csv.
- codyvoda 2y ago
  this is a huge difference between Ibis (and really, any dataframe library that works with larger-than-RAM or distributed engines) and pandas. PySpark, Polars, and Snowpark all to some extent also do not implicitly order your data so in Ibis, you need to explicitly manage order with an `order_by` call at the end. there are a lot of technical reasons for why maintaining order is inefficient as you scale up
rldjbpin 2y ago
you could completely respect the decision to deprecate some support, but the marketing speak comes across distasteful. from a user's perspective, selecting a tool should be about what's "best" for you and not what is the "best" out there (from some arbitrary measure nonetheless). pandas made sense and might still make sense for many, especially who are migrating from R and are already comfortable with dataframe paradigm. using the same paradigm and just saying "we run faster" does not solve the ergonomics or the reason why people adopted this way of processing data. there are plenty players in the space going about this from the stack. would be nice to see more discussion on alternative generic data processing paradigms instead.
- codyvoda 2y ago
  this is a blog for the Ibis project, directly from one of the main developers/maintainers of the project, about deprecating two (of 20+) backends the project supports and giving the justification as a project, we're big on using the right tool for the job -- of course we often think that's Ibis. a lot of effort has been put into the ergonomics, taking learnings from pandas and dplyr and SQL and various other Python data projects the team has been instrumental in. if you weren't aware, the creator of pandas also created Ibis and the current lead developer (Phillip Cloud) was also very involved in pandas. we have plenty of other blogs and things on the documentation about the project's ergonomics and thoughts on data processing paradigms this specific blog isn't intended as "marketing speak" or to somehow be distasteful, it's intended to inform the Ibis community that we're deprecating a backend and justifying why
- cpcloud 2y ago
  Pandas support as input and output isn't going anywhere. People can continue to bring their pandas DataFrames to Ibis and run Ibis expressions against them. > pandas made sense and might still make sense for many, especially who are migrating from R and are already comfortable with dataframe paradigm. Yes, and nothing is different about the paradigm after the changes outlined in the blog post, it's still a dataframe paradigm. > using the same paradigm and just saying "we run faster" does not solve the ergonomics or the reason why people adopted this way of processing data. This seems to indicate you _do_ understand that the paradigm isn't changing but then does a complete 180 into complaining about Ibis not solving pandas ergonomics issues. Which is it?

228 points by nojito 2y ago

Huge fan of Ibis, the value isn't that you can now use DuckDB... it's that your syntax will work when the next cool thing arrives too

log4shell 2y ago

Its great to have a single entrypoint for multiple backends. What I am trying to understand and couldn't find much information related to: How does the use of multiple engines in Ibis impact the consistency of results for the same input and query, particularly in relation to semantic differences among the engines?

softwaredoug 2y ago

One thing I do like about pandas is it’s pretty extensible to columns of new types. Maybe I’m missing something, but does Polars allow this? Last time I checked there wasn’t a clear path forward.

datadrivenangel 2y ago
Ibis allows this with custom datatypes.
porker 2y ago
I'm curious, when do you want to add a new column type? It's not a situation I've been in.
- softwaredoug 2y ago
  GeoPandas is one popular library (https://geopandas.org/en/stable/ https://geopandas.org/en/stable/) I added a column type for full text search, searching tokenized text (https://github.com/softwaredoug/searcharray https://github.com/softwaredoug/searcharray) The pandas support for extensions is very robust with extensive testing to make sure you can do all the data aggregations functionality you expect in and pandas column.
deadlypointer 2y ago
OTOH the list and struct types allow lots of complex operations, e.g with `.eval`, which run blazingly fast.

maxdo 2y ago

curious to know what they use instead of Dask ?

datadrivenangel 2y ago
DuckDB. It's faster on a single computer than Dask.
- codyvoda 2y ago
  Ibis still supports over a dozen backends, including DuckDB and DataFusion and Polars (all great single-node OLAP query engine options that are often faster than trying something distributed) and PySpark, Snowflake, BigQuery, Trino, ClickHouse, Druid, and more plenty of options for fast and scalable computation. in the fullness of time, it would be great to have a Dask backend but it'd need to be rewritten to use dask-expr, which isn't something the core Ibis team has the capacity for
- lopatin 2y ago
  As a potential user of Ibis, how do I know they won't eventually drop support for other backends until it just becomes a frontend for DuckDB?
  - codyvoda 2y ago
    Great question, a bunch of reasons: 1. The value prop of Ibis is to avoid vendor lock-in and allow users to more easily swap out execution engines that meet their needs. So it's valuable to the project to have "competing" backends supported (e.g. DuckDB, DataFusion, and Polars for local, and Snowflake, ClickHouse, PySpark, Databricks, etc. for distributed) -- the idea is a standard like Apache Arrow that everyone adopts to stop this cycle of tightly coupling a dataframe interface to an execution engine 2. Ibis is primarily supported by Voltron Data (my employer) which is itself a vendor, so the real worry would be that Ibis drops all backends until it only support Voltron Data's engine (Theseus)... 3. ... but Voltron Data cannot do that for a number of reasons. First, Ibis is an independently governed open source project. Second, it would not make any business sense. You can read some more on that here: https://ibis-project.org/posts/why-voda-supports-ibis https://ibis-project.org/posts/why-voda-supports-ibis. The general idea is to add more backends, not remove them, but pandas is a special case where it introduces a lot of technical complexity with (arguably) negative benefit to potential users -- the DuckDB backend can do everything the pandas backend can do, but better

deleted 2y ago

[deleted]

spywaregorilla 2y ago

What is this exactly? The front page doesn't make it clear to make what ibis is besides being an alternative to pandas/polars.

prepend 2y ago
Check the project’s left nav, https://ibis-project.org/why https://ibis-project.org/why I didn’t know either. It’s a python dataframe library/api.
smu3l 2y ago
Never heard of ibis before but the front page give a pretty good overview of what it is to me at least. Looks like a dataframes api that can be executed on a variety of backend engines, both distributed and local. Somewhat analogous to how the pandas api can be used in pyspark via pyspark pandas, but from the api -> implementation direction rather than (pandas) implementation -> api -> implementation maybe?
giovannibonetti 2y ago
As far as I can tell, it is an ORM for data engineers. They can write Python code that gets translated to either SQL or some other language understood by the DB engine that actually runs it.
- cpcloud 2y ago
  Ibis is not an ORM. It has no opinion of or knowledge about your data model, or relationships between tables and their columns. Check out SQLAlchemy's ORM (https://docs.sqlalchemy.org/en/20/orm/ https://docs.sqlalchemy.org/en/20/orm/) for what that looks like.

infecto 2y ago

I will try it out but honestly the pain-point for me is the library api to pandas is not always intuitive/natural to how things should be in Python. The NaN/None bit is annoying but I find that to be a minor annoyance.

mananaysiempre 2y ago

> NULL indicates a missing value, and NaN is Not a Number. That’s actually less true than it sounds. One of the primary functions of NaN is to be the result of 0/0, so there it means that there could be a value but we don’t know what it is because we didn’t take the limit properly. One of the primary functions of NULL is to say that a tuple satisfies a predicate except we don’t know what this one position is—it’s certainly is something out in the real world, we just don’t know what. These ideas are what motivates the comparison shenanigans both NaN and NULL are known for. There’s certainly an argument to be made that the actual implementation of both of these ideas is half-baked in the respective standards, and that they are half-baked differently so we shouldn’t confuse them. But I don’t think it’s fair to say that they are just completely unrelated. If anything, it’s Python’s None that’s doesn’t belong.

cpcloud 2y ago
100% this. We'd love to have a docs contribution that lays this out in detail if you'd be up for it! Disclaimer: (I lead the project and work on it full time).
aeonik 2y ago
Why don't we build a portable numeric system that just has these mathematical constants and definitions built in, in a portable and performance manner?
- verandaguy 2y ago
  We just don’t have the technology for that, else there’d be libraries floating around that let you do that.
miohtama 2y ago
Related to this, below someone posted a link to the blog post of Wes McKinney where he discussed Pandas limitations and how PyArrow works around these > 4. Doing missing data right > All missing data in Arrow is represented as a packed bit array, separate from the rest of the data. This makes missing data handling simple and consistent across all data types. You can also do analytics on the null bits (AND-ing bitmaps, or counting set bits) using fast bit-wise built-in hardware operators and SIMD. > The null count in an array is also explicitly stored in its metadata, so if data does not have nulls, we can choose faster code paths that skip null checking. With pandas, we cannot assume that arrays do not have null sentinel values and so most analytics has extra null checking which hurts performance. If you have no nulls, you don’t even need to allocate the bit array. > Because missing data is not natively supported in NumPy, over time we have had to implement our own null-friendly versions of most key performance-critical algorithms. It would be better to have null-handling built into all algorithms and memory management from the ground up. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/

ironmagma 2y ago

Man, these frontend people are constantly churning through dependencies, a new framework every year. Don't reinvent the wheel, just be like backend folks and pick a dependency and stick with it. Oh wait...

vundercind 2y ago
ETL and data analysis “notebook” world been warming up for years and now it’s red-hot due to every company down to Grandma’s Cookie Bakery LLC wanting to “do AI”. And most of the standard tools are surprisingly terrible so churn is increasing for reasons similar to why that happened in frontend (the underlying tools are ass and the existing interfaces bad and almost all of it’s being overcomplicated for bad reasons and a million more developers just jumped into the pool)
- ironmagma 2y ago
  Yes, and so really what we are seeing is just the effects of a high quality bar. If you want high-quality software, you do more iterations. And iterations produce breaking changes.
  - vundercind 2y ago
    Mmm. A lot of it’s motivated by pain, with the output filtered through a web of organizational friction, principal agent problems, self-promotional nonsense, and other harmful factors. If the web’s any indication, there’s no guarantee this activity is efficient at getting us closer to meeting a high quality bar.
draw_down 2y ago
[dead]
dr_mee6 2y ago
Just as an FYI Ibis is being around for 10 years approximately...

frakt0x90 2y ago

Curious if you considered Polars. That's become the defacto standard in my group as we all dislike pandas.

cpcloud 2y ago
We support Polars as a backend. DuckDB was chosen as the default because Polars was too unstable to be the default backend two years ago.
amelius 2y ago
Good riddance. Pandas turns 10x programmers into 1x programmers.
- data-ottawa 2y ago
  I've been using pandas for over a decade and I'm very efficient with it. The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it. I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style. It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing. I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.
  - codyvoda 2y ago
    fun fact if you weren't aware, Ibis was also originally created by Wes McKinney and takes a lot of inspiration from the tidyverse
    - data-ottawa 2y ago
      I was not aware of that, the influence from the tidyverse is very clear I’ve tried to make mockups of similarly inspired API and I think they did a great job with clever tricks like the `_` reference (does that clash with IPython/Jupyter’s use?). I do wonder how they manage compatibility with so many backends, it seems like many features will be directly tied to your backend (e.g. Trino using Java Pattern for regex vs BigQuery using Re2) in hard to explain ways. But maybe that’s not a big concern, because very often you’re only going to be using a couple of backends. I’ll have to try Ibis out for myself, it looks like it can unify a lot of the work I have to do. I’ve moved away from pandas for all but the last mile of computation, so this might be a direct good SQL replacement.

mrbluecoat 2y ago

Dask was really cool back in the day. Farewell, friend.

lmc 2y ago
Back in the day? What are people using now instead?
- ZeroCool2u 2y ago
  Ray seems to be a popular choice now if you need multi node. But Polars is really nice if you're okay with just vertically scaling on one machine.
- cpcloud 2y ago
  I've been working with pencil and paper recently. Thinking about starting an Ibis backend for it.
- dr_mee6 2y ago
  Abacuses?

thsgtu 2y ago

About time. It always surprises me how long pandas has been able to hold on. Wes McKinney talked about pandas' limitations back in 2017 https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/

__mharrison__ 2y ago

Not surprising. There are much better compute engines than pandas. Folks then ask why not jump from pandas to [insert favorite tool]? - Existing codebases. Lots of legacy pandas floating about. - Third party integration. Everyone supports pandas. Lots of libraries work with tools like Polars, but everything works with pandas. - YAGNI - For lots of small data tasks, pandas is perfectly fine.

cpcloud 2y ago
All great reasons to stick with Pandas. If a thing works for you, then that's one less thing to manage. Fully agree that if pandas is working for you, then you're better off sticking with it!
whimsicalism 2y ago
If your task is too big for pandas, you should probably skip right over dask and polars for a better compute engine.
- __mharrison__ 2y ago
  Jump straight to cluster and skip Dask? Not sure what "big" means here, but a combination of .pipe, pyarrow, and polars can speed up many slow Pandas operations. Polars streaming is surprisingly good for larger than RAM. I get that clusters are cool, but I prefer to keep it on a single machine if possible. Also, libraries like cudf can greatly speed up Pandas code on a single machine, while Snowpark can scale Pandas code to Snowflake scale.
  - cpcloud 2y ago
    In my experience, Polars streaming runs out of memory at much smaller scales than both DuckDB and DataFusion and tends to use much more memory for the same workload when it doesn't outright segfault. Polars is faster than those two once you get to less than a few GB, but beyond that you're better off with DuckDB or DataFusion. I would love for this to improve in Polars, and I'm sure it will!

mrguyorama 2y ago

Pandas made me think I hated python.

e10v_me 2y ago
Funny, the same) When I was switching from R (data.table) to Python, it was painful. Not only because it was slow, but because of the API. At that time, I thought that maybe it's because of switching to something new. Several years later, switching from Pandas to Polars API was a real joy (Ibis is also good is that sense). So, I learned that it had been Pandas fault all along))

whimsicalism 2y ago

Frankly, pandas/dask/polars - all are trying to recreate something that has existed for years (ie. sql, spark) and with a terrible API. I truly thought I was terrible at python for a long time because of pandas - turns out it just has an absolutely terrible API surface

tech_ken 2y ago
I would not say pandas is a recreation of SQL/Spark, they have very different use-cases in my experience. SQL/Spark is like a bulk data management tool: I use it if I need to load massive data from some remote store, perform light preprocessing, join up a couple dimension tables, etc. Having normalized and joined my data, then pandas enters as a 'last-mile' processing engine, particularly when paired with ex. SKLearn or whatever other inference lib you're using. Pandas is awesome if you need to ex. apply string manipulation to a data table or daisy-chain some complicated computations together. Honestly in my opinion the API is really nice, I came over from R tidyverse and the 'chained methods' approach to pandas let's me carry over all my old patterns and paradigms. I find it far easier to use that approach than having to write like 20 dependent subqueries or staging tables
cmollis 2y ago
true spark has existed for years and is a great toolset.. i use it ever day. it's also a huge hassle spinning clusters up and down and configuration is complex. I can execute some pretty hairy scans against a huge s3 parquet dataset in Duckdb that I would typically have to run in either spark or athena.. it's a little slower, but not ridiculously slower. And, it does all of that from my desktop.. no clusters, no mem or task configs.. just run the query. Being able to integrate all of the expensive historical scanning and knitting that back into an ML pipeline with desktop python is pretty nice.

dammaj 2y ago

Personnally, I tend to use Pandas because it is integrated everywhere, because of the ecosystem that uses it. Let's say I want to read data from json file (csv file, python dict, etc.) and I want to plot it using plotly. If Ibis is compatible with whatever Pands dataframe is compatible with, then for most of my usage I don't really care much about the "backend".

kremi 2y ago

Pandas has been working fine for me. The most powerful feature that makes me stick to it is the multi-index (hierarchical indexes) [1]. Can be used for columns too. Not sure how the cool new kids like polars or ibis would fare in that category. [1] https://pandas.pydata.org/docs/user_guide/advanced.html#advanced-indexing-with-hierarchical-index https://pandas.pydata.org/docs/user_guide/advanced.html#adva...

cpcloud 2y ago
Multi-indexes definitely have their place. In fact, I got involved in pandas development in 2013 as part of some work I was doing in graduate school, and I was a heavy user of multi-indexed columns. I loved them. Over time, and after working on a variety of use cases, I personally have come to believe the baggage introduced by these data structures wasn't worth it. Take a look at the indexing code in pandas, and the staggering complexity of what's possible to put inside square brackets and how to decipher its meaning. The maintenance cost alone is quite high. We don't plan to ever support multi-indexed rows or columns in Ibis. I don't think we'd fare well _at all_ there, intentionally so.
- kremi 2y ago
  > Take a look at the indexing code in pandas As the end-user, not quite my concern. > and the staggering complexity of what's possible to put inside square brackets and how to decipher its meaning I might not be aware of everything that's possible -- the usage I have of it doesn't give me an impression of staggering complexity. In fact I've found the functionality quite basic, and have been using pd.MultiIndex.from_* quite extensively for anything slightly more advanced than selecting a bunch of values at some level of the index.
  - hansvm 2y ago
    > As the end-user, not quite my concern. Complicated code is (probabilistically) slow, buggy, infrequently updated code. By all means, if it looks like a good enough tool for the job (especially if the alternatives don't) then use it anyway, but that's slightly different from it not being your concern. I've seen enough projects need "surprise" major revisions because some team tried to sneak a dataframe into a 10M QPS service that my default is keeping pandas far away from anything close to a user-facing product. I've also seen costs balloon as the data's scale grows beyond what pandas can handle, but basically all the alternatives suck for myriad reasons, so I don't try to push "not pandas" in the data backend. People can figure out what works for themselves, and I kind of like just writing it from scratch in a performant language when I personally hit that bottleneck.

UncleOxidant 2y ago

Related to this pandas?: https://pandas.pydata.org/ https://pandas.pydata.org/

cpcloud 2y ago
On the contrary, very related to that pandas!

Kalanos 2y ago

I've only heard about Ibis maybe three times in the past two years and I pay pretty close attention to the space. If Ibis moves away from pandas, then it just means that I am less likely to try Ibis because there is no bridge. Sure, hip new frameworks are moving away from pandas/numpy, but I'll wait 5 years for the dust to settle here while the compatibility and edge cases sort themselves out. The pydata/numfocus ecosystem is extensive. It's just tabular data. So what if I have to wait a few more milliseconds to get my result.

cpcloud 2y ago
Sounds like a solid plan. No reason to incur switching costs if things are working! I think you're misunderstanding what "removing pandas" means. You can still compute on DataFrames, it'll just be with something other than Pandas itself, probably DuckDB, Polars, or DataFusion. So, the bridge was there, is there, and isn't going anywhere.
- deleted 2y ago
  [deleted]
- halfcat 2y ago
  > I think you're misunderstanding what "removing pandas" means. Maybe it’s in the messaging, like the “Farewell Pandas” part?
  - cpcloud 2y ago
    Feel free to put up a pull request with an alternate title that you feel communicates the message in a more accurate way!
codyvoda 2y ago
> If Ibis moves away from pandas, then it just means that I am less likely to try Ibis because there is no bridge. the bridge is that Ibis accepts pandas as input and has a `to_pandas()` method as output Ibis also still depends on pandas (and thus numpy) internally. also, Ibis was created by the creator of pandas, and the lead developer (and other contributors) have also worked on pandas. the Ibis team understands pandas and the broader Python data ecosystem very well > It's just tabular data. So what if I have to wait a few more milliseconds to get my result. usually as scale grows, milliseconds -> seconds -> minutes -> hours -> days. at some point along the way having a dataframe library that can scale up without rewriting your code might be useful. but if you're dealing with small tabular data and pandas meets your needs, it's a great library to use and stick with!

riezebos 2y ago

Nice to see, over the past months I've replaces pandas with ibis in all new projects and I am a huge fan! - Syntax in general feels more fluid than pandas - Chaining operations with deferred expressions makes code snippets very portable - Duckdb backend is super fast - Community is very active, friendly and responsive I'm trying to promote it to all my peers but it's not a very well known project in my circles. (Unlike Polars which seems to be the subject of 10% of the talks at all Python conferences)

glial 2y ago

In my experience, the best thing about Pandas is how much it made me appreciate using dplyr and the tidyverse. If it wasn't for Pandas, I may not be the avid R user I am today.

antman 2y ago

"TL; DR: we are deprecating the pandas and dask backends and will be removing them in version 10.0. There is no feature gap between the pandas backend and our default DuckDB backend" There is an important feature gap, pandas has all the users and no one knows you guys. Was trying to pitch this to my company but I guess its goodbye Ibis and thanks for all the fish.

codyvoda 2y ago
were you using the pandas backend in Ibis and pitching that to your company? you'd have much better performance and efficiency using the default backend, even if you used pandas for data input and output (though the DuckDB data readers are generally more efficient too). I'd be happy to help you on the pitch! and if you were using the pandas backend or know of many users doing so, please reach out! we love feedback from our users and would be sad to see y'all go, we're open to reconsidering any decisions based on feedback
- antman 2y ago
  Yes I agree and have tested it, my point is that marketing wise it is a shame. I will even think that when looking into new tooling "pandae" compatibility as a search term and incentive would be far ahead the word "dataframe"
  - codyvoda 2y ago
    we will take this excellent marketing advise under advisement, thank you for your consideration
  - cpcloud 2y ago
    Pandas compatibility as input and output will remain. Not sure what's in doubt here.

zhangyt26 2y ago

How does ibis manage ordering of the data frame for SQL based engine? One big gap that I noticed is that pandas retain ordering between operations, bust most SQL based engine doesn’t give you that guarantee. Looking at duckdb, it looks they only do that for csv.

codyvoda 2y ago
this is a huge difference between Ibis (and really, any dataframe library that works with larger-than-RAM or distributed engines) and pandas. PySpark, Polars, and Snowpark all to some extent also do not implicitly order your data so in Ibis, you need to explicitly manage order with an `order_by` call at the end. there are a lot of technical reasons for why maintaining order is inefficient as you scale up

rldjbpin 2y ago

you could completely respect the decision to deprecate some support, but the marketing speak comes across distasteful. from a user's perspective, selecting a tool should be about what's "best" for you and not what is the "best" out there (from some arbitrary measure nonetheless). pandas made sense and might still make sense for many, especially who are migrating from R and are already comfortable with dataframe paradigm. using the same paradigm and just saying "we run faster" does not solve the ergonomics or the reason why people adopted this way of processing data. there are plenty players in the space going about this from the stack. would be nice to see more discussion on alternative generic data processing paradigms instead.

codyvoda 2y ago
this is a blog for the Ibis project, directly from one of the main developers/maintainers of the project, about deprecating two (of 20+) backends the project supports and giving the justification as a project, we're big on using the right tool for the job -- of course we often think that's Ibis. a lot of effort has been put into the ergonomics, taking learnings from pandas and dplyr and SQL and various other Python data projects the team has been instrumental in. if you weren't aware, the creator of pandas also created Ibis and the current lead developer (Phillip Cloud) was also very involved in pandas. we have plenty of other blogs and things on the documentation about the project's ergonomics and thoughts on data processing paradigms this specific blog isn't intended as "marketing speak" or to somehow be distasteful, it's intended to inform the Ibis community that we're deprecating a backend and justifying why
cpcloud 2y ago
Pandas support as input and output isn't going anywhere. People can continue to bring their pandas DataFrames to Ibis and run Ibis expressions against them. > pandas made sense and might still make sense for many, especially who are migrating from R and are already comfortable with dataframe paradigm. Yes, and nothing is different about the paradigm after the changes outlined in the blog post, it's still a dataframe paradigm. > using the same paradigm and just saying "we run faster" does not solve the ergonomics or the reason why people adopted this way of processing data. This seems to indicate you _do_ understand that the paradigm isn't changing but then does a complete 180 into complaining about Ibis not solving pandas ergonomics issues. Which is it?