4 ms·
You seem to be hung up on typing here. But, with care, you can make any code solid without necessarily relying on types, which, in Python's case, AFAIK, is a bu
by bradwood 4y ago
You seem to be hung up on typing here. But, with care, you can make any code solid without necessarily relying on types, which, in Python's case, AFAIK, is a build-time safety net, rather than run-time protection.
I think a lot of this comes down to how your construct your code, which is often some kind of data pipeline. Wherever possible, I try to do my pandas code in a functional style -- lots of pure functions that operate on a dataframe and return a a derivative dataframe with only a simple transformation (ideally immutably, if this is not too expensive).
Chaining these together can be done with a functional library, or a simple implementation of `pipe()` or `pipeeither()`[1].
The main advantage here is that testing all the functions that go into this pipeline, becomes much easier. No side effects, no mocking, and very easy to test with `pytest` or similar.
Spending time writing the tests might seem laborious but it's worth it, and you can make these robustly able to exercise all kinds of conditions, including things like empty dataframes, weird data (NaNs, etc), in the inputs.
Finally, keep all charting completely away from the data manipulation functions and treat these separately. They can be a PITA to write automated tests for, and probably just need eyeball manual testing, so make them as simple as possible, without any data manipulation in these.
Not a silver bullet, I'm afraid, but the approach does work but you need to be thorough and structured in your approach.
[1]: https://news.ycombinator.com/item?id=34406420 https://news.ycombinator.com/item?id=34406420
- TheAlchemist 4y agoThanks for feedback. Yes, it's mostly about typing and hinting in the IDE, but also about maintenability. For instance, if you use string as column names, they will be everywhere - it will be very difficult to refactor the code (for example - change the name of the column) pipe is interesting indeed. However, I would still like to make specify some minimum requirement a dataframe must meet (for ex, having a column 'Date', and 'City') so that the function can perform it's action. I've played a bit with the available tools, but none seem to work well - pandera is nice but for some reason PyCharm seem unable to leverage that, so there is no automcomplete nor warnings if the type is not appropriate.
- bradwood 4y agoWhat about just liberal use of ‘assert‘ throughout the codebase. These are good to use in production code (not just in tests). And give you a means of defensively coding and forcing errors if, for example column names are missing?
- TheAlchemist 4y agoThat wouldn't solve the problems described, would it ? If a column is missing, the code would crash anyway - that's not my issue. My issue is more about understanding and maintainability of the code, as well as ease of coding. I've found this, which seems quite close to be usable: https://strictly-typed-pandas.readthedocs.io/en/latest/ https://strictly-typed-pandas.readthedocs.io/en/latest/
- bradwood 4y agoYou may find lots of ide tools, type hinting systems and so on. But these are a bit like crutches IMHO. You don't need them to make maintainable code. If your functions are small, pure, well-tested then you are good. Small functions are much more maintainable, testable,easy to reason about, etc. So rather than worrying about code completion and type hints. Just try to work in a more functional style. It will pay big dividends.
- TheAlchemist 4y agoAgree on small functions, although it still doesn't really solve my perceived problem. Would you know some good python applications using pandas in that way that I could look at to get a better idea ?