4 ms·

Python's typing ecosystem has come leaps and bounds, quickly. About 5 years ago I had to do an evaluation of Python+mypy and found that while mypy was a great i

by mjr00 3y ago

Python's typing ecosystem has come leaps and bounds, quickly. About 5 years ago I had to do an evaluation of Python+mypy and found that while mypy was a great idea, support by major libraries was abysmal and mypy itself was immature, so it wasn't suitable for real use. 2 years later, I took another look and the ecosystem had evolved massively, most libraries I cared about had type annotations added, PyCharm/VSCode had added support for type checking in the IDE, etc. These days all of my use of Python is with type annotations that are validated by mypy as part of CI. It's very rare that I run into a Python library that doesn't have type annotations available. It's massively improved the stability of the Python code I'm working on. One indirect side effect is that static typing actively discourages people from poor design patterns such as **kwargs or passing around dicts/dataframes of data, since there's an obvious downside to doing so now.

bjornasm 3y ago
I agree with avoiding *kwargs, but I wonder what we should do rather than to pass around dataframes if we have a program that works with dataframes? I get that there is a downside as the actual types are hidden inside the dataframes, but I am unsure if the tradeoff is worth it by unpacking the dataframe into arrays or objects for every function call and return, if the functions require them to be dataframes?
- sgarland 3y ago
  You can define nested types for args and return values: def foo(bar: dict[int, dict[int, str]]) -> list[tuple[str]] In older (< 3.10 ?) versions you have to import the classes from the typing module, but other than that it’s the same. You can also define types with the Generic class if you’d rather. With live checks while you write, it’s pretty easy to do this as you go.
- mjr00 3y ago
  Sadly I haven't found any good ways to strongly type dataframes, so my solution is mainly cultural: just make people aware that they shouldn't use dataframes unless it makes sense, as opposed to Python codebases where dataframes are the default data type that gets passed around. I've tried to limit dataframe usage only to things like reading data from CSVs, then converting them to typed dataclasses before further processing. It also depends on your use case. pandas is obviously very flexible and easy to use, so it might be worth the tradeoff to keep using it if your main use case is interactive Jupyter notebooks run by data scientists, or dealing with highly variable input data. But most of my Python is backend data pipelines that work on predictable input data, where reliability is important, so I default to dataclasses, only using pandas in rare situations with a lot of validation code surrounding it.

by mjr00 3y ago

bjornasm 3y ago

I agree with avoiding *kwargs, but I wonder what we should do rather than to pass around dataframes if we have a program that works with dataframes? I get that there is a downside as the actual types are hidden inside the dataframes, but I am unsure if the tradeoff is worth it by unpacking the dataframe into arrays or objects for every function call and return, if the functions require them to be dataframes?

sgarland 3y ago
You can define nested types for args and return values: def foo(bar: dict[int, dict[int, str]]) -> list[tuple[str]] In older (< 3.10 ?) versions you have to import the classes from the typing module, but other than that it’s the same. You can also define types with the Generic class if you’d rather. With live checks while you write, it’s pretty easy to do this as you go.
mjr00 3y ago
Sadly I haven't found any good ways to strongly type dataframes, so my solution is mainly cultural: just make people aware that they shouldn't use dataframes unless it makes sense, as opposed to Python codebases where dataframes are the default data type that gets passed around. I've tried to limit dataframe usage only to things like reading data from CSVs, then converting them to typed dataclasses before further processing. It also depends on your use case. pandas is obviously very flexible and easy to use, so it might be worth the tradeoff to keep using it if your main use case is interactive Jupyter notebooks run by data scientists, or dealing with highly variable input data. But most of my Python is backend data pipelines that work on predictable input data, where reliability is important, so I default to dataclasses, only using pandas in rare situations with a lot of validation code surrounding it.