3 ms·
For me, the superpower of polars is production stability. Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly aroun
by perrygeo 23d ago
For me, the superpower of polars is production stability.
Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float.
Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild.
I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely!
Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.
- sigseg1v 23d agoDo people actually use languages where it doesn't do any compile time checks on the API in 2026? Why would developers put up with the lack of that. I'm not in this ecosystem but what you are describing sounds like the bare minimum to me that should be table stakes.
- perrygeo 23d agoI agree. But the data science/engineering space is enamored with Python. Makes good sense when doing interactive work. Makes no sense in production. But by that point, most developers consider it a sunk cost and just keep their Python apps limping along rather than rewriting. That's why polars is a great option. Start prototyping in python, then a relatively easy port to a Rust app when the need hits.
- maleldil 23d agoYou can still have extensive "compile-time" checking with linters and static type checkers. This makes Polars even better in typed Python, as it type-checks much better than the dynamic mess that is Pandas.
- perrygeo 23d agoPolars is much better than pandas in this regard. But using it via Python still carries the risk of runtime errors. Python's linters leave many holes unfilled - it catches some stuff but still leaves everything else to the runtime. Literally every Python application that I've worked on in the last 3 years a) is 100% type checked yet b) still hits massive numbers of novel runtime errors on the regular. Python's anemic type system is in no way a substitute for an advanced compiler.
- paddy_m 22d agoWhen I deal with datascience, I rarely have type problems in my programs that python typecheckers can't catch. I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading.
- perrygeo 22d ago> I rarely have type problems in my programs that python typecheckers can't catch. > I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading. Something about those two statements makes my brain explode. You rarely have type problems, except when dealing with data? Isn't the whole point of a program to deal with data?
- what 22d agoIf it’s 100% type checked and you’re still hitting massive numbers of type related runtime errors, it probably isn’t 100% checked.
- perrygeo 21d agoEmpirically false. Python's type system is not sufficient to describe runtime variants, period. I have worked on at least three system that used mypy in strict mode yet hit runtime errors constantly. I get where you're coming from. A good type system should provide invariant guarantees about runtime behavior. But Python's does not.