4 ms·
This looks really nice! It feels like in the last few years (probably with the increase in data science and analytics) dataframe engines have become really exci
by benrutter 3y ago
This looks really nice! It feels like in the last few years (probably with the increase in data science and analytics) dataframe engines have become really exciting, its very cool to see so much innovation in the field.
One thing i haven't seen yet and I'd be really interested in knowing if column level type hinting is on the plan at all for this (or any similar tools)?
I'm a data engineer and my team use mypy pretty heavily, but that can only really tell us if something is a "dataframe" type. It always strikes me that 90% of data issues during development (incorrect joins, column misspellings or non-type appropriate operations, not factoring in nulls) would be eliminated if mypy had information of the columns and their data types.
- jaychia 3y agoInteresting. Daft currently does validation on types/names only at runtime. The flow looks like: 1. Construct a dataframe (performs schema inference) 2. Access (now well-typed) columns and operations on those columns in the dataframe, with associated validations. Unfortunately step (1) can only happen at runtime and not at type-checking-time since it requires running some schema inference logic, and step (2) relies on step (1) because the expressions of computation are "resolved" against those inferred types. However, if we can fix (1) to happen at type-checking time using user-provided type-hints in place of the schema inference, we can maybe figure out a way to propagate this information through to mypy. Would love to continue the discussion further as an Issue/Discussion on our Github!
- benrutter 3y agoThanks for getting back, thats really interesting, for sure I will do!