3 ms·

You can still have extensive "compile-time" checking with linters and static type checkers. This makes Polars even better in typed Python, as it type-checks muc

You can still have extensive "compile-time" checking with linters and static type checkers. This makes Polars even better in typed Python, as it type-checks much better than the dynamic mess that is Pandas.

perrygeo 23d ago
Polars is much better than pandas in this regard. But using it via Python still carries the risk of runtime errors. Python's linters leave many holes unfilled - it catches some stuff but still leaves everything else to the runtime. Literally every Python application that I've worked on in the last 3 years a) is 100% type checked yet b) still hits massive numbers of novel runtime errors on the regular. Python's anemic type system is in no way a substitute for an advanced compiler.
- paddy_m 22d ago
  When I deal with datascience, I rarely have type problems in my programs that python typecheckers can't catch. I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading.
  - perrygeo 22d ago
    > I rarely have type problems in my programs that python typecheckers can't catch. > I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading. Something about those two statements makes my brain explode. You rarely have type problems, except when dealing with data? Isn't the whole point of a program to deal with data?
- what 22d ago
  If it’s 100% type checked and you’re still hitting massive numbers of type related runtime errors, it probably isn’t 100% checked.
  - perrygeo 21d ago
    Empirically false. Python's type system is not sufficient to describe runtime variants, period. I have worked on at least three system that used mypy in strict mode yet hit runtime errors constantly. I get where you're coming from. A good type system should provide invariant guarantees about runtime behavior. But Python's does not.

3 ms·

You can still have extensive "compile-time" checking with linters and static type checkers. This makes Polars even better in typed Python, as it type-checks muc

by maleldil 23d ago

perrygeo 23d ago
Polars is much better than pandas in this regard. But using it via Python still carries the risk of runtime errors. Python's linters leave many holes unfilled - it catches some stuff but still leaves everything else to the runtime. Literally every Python application that I've worked on in the last 3 years a) is 100% type checked yet b) still hits massive numbers of novel runtime errors on the regular. Python's anemic type system is in no way a substitute for an advanced compiler.
- paddy_m 22d ago
  When I deal with datascience, I rarely have type problems in my programs that python typecheckers can't catch. I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading.
  - perrygeo 22d ago
    > I rarely have type problems in my programs that python typecheckers can't catch. > I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading. Something about those two statements makes my brain explode. You rarely have type problems, except when dealing with data? Isn't the whole point of a program to deal with data?
- what 22d ago
  If it’s 100% type checked and you’re still hitting massive numbers of type related runtime errors, it probably isn’t 100% checked.
  - perrygeo 21d ago
    Empirically false. Python's type system is not sufficient to describe runtime variants, period. I have worked on at least three system that used mypy in strict mode yet hit runtime errors constantly. I get where you're coming from. A good type system should provide invariant guarantees about runtime behavior. But Python's does not.