4 ms·

why? It's the industry standard as far as my reach goes. What other framework would you replace it with? No, polars or spark is not a good answer, those are o

• porker 10mo ago
  > No, polars or spark is not a good answer, those are optimized for data engineering performance, not a holistic approach to data science. Can you expand on why Polars isn't optimised for a holistic approach to data science?
  • fifilura 10mo ago
    I have not work with Polars, but I would imagine any incompatibility with existing libraries (e.g. plotting libraries like plotnine, bokeh) would quickly put me off. It is a curse I know. I would also choose a better interface. Performance is meh to me, I use SQL if i want to do something at scale that involves row/column data.
    • rbartelme 10mo ago
      This is a non-issue with Polars dataframes to_pandas() method. You get all the performance of Polars for cleaning large datasets, and to_pandas() gives you backwards compatibility with other libraries. However, plotnine is completely compatible with Polars dataframe objects.
    • maleldil 10mo ago
      You can always convert from Polars to Pandas. Plotnine will do it automatically for you, even.
• crystal_revenge 10mo ago
  You can assert whatever you want, but Polars is a great answer. The performance improvements are secondary to me compared to the dramatic improvement in interface. Today all serious DS work will ultimately become data engineering work anyway. The time when DS can just fiddle around in notebooks all day has passed.
  • this_user 10mo ago
    Pandas is widely adopted and deeply integrated into the Python ecosystem. Meanwhile, Polars remains a small niche, and it's one of those hype technologies that will likely be dead in 3 years once most of its users realise that it offers them no actual practical advantages over Pandas. If you are dealing with huge data sets, you are probably using Spark or something like Dask already where jobs can run in the cloud. If you need speed and efficiency on your local machine, you use NumPy outright. And if you really, really need speed, you rewrite it in C/C++. Polars is trying to solve an issue that just doesn't exist for the vast majority of users.
    • stdbrouw 10mo ago
      Arguably Spark solves a problem that does not exist anymore: single node performance with tools like DuckDB and Polars is so good that there’s no need for more complex orchestration anymore, and these tools are sufficiently user-friendly that there is little point to switching to Pandas for smaller datasets.
    • minimaxir 10mo ago
      > once most of its users realise that it offers them no actual practical advantages over Pandas What? Speed and better nested data support (arrays/JSON) alone are extremely useful to every data scientist. My produtivity skyrocketed after switching from pandas to polars.
    • crystal_revenge 10mo ago
      > Pandas is widely adopted and deeply integrated into the Python ecosystem. This is pretty laughable. Yes there are very DS specific tools that make good use of Pandas, but `to_pandas` in Polars trivially solves this. The fact that Pandas always feels like injecting some weird DSL into existing Python code bases is one of the major reasons why I really don't like it. > If you are dealing with huge data sets, you are probably using Spark or something like Dask already where jobs can run in the cloud. If you need speed and efficiency on your local machine, you use NumPy outright. And if you really, really need speed, you rewrite it in C/C++. Have you used Polars at all? Or for that matter written significant Pandas outside of a notebook? The number one benefit of Polars, imho, is that Polars works using Expressions that allow you to trivially compose and reuse fundamental logic when working with data in a way the works well with other Python code. This solves the biggest problem with Pandas is that it does not abstract well. Not to mention that Pandas is really poor dataframe experience outside of it's original use case which was financial time series. The entire multi-index experience is awful and I know that either you are calling 'reset_index' multiple times in your Pandas logic or you have bugs.
• minimaxir 10mo ago
  What can you do in more easily in pandas than polars?