4 ms·
Show HN: Scaling Dataflows with Hamilton
- sterlinm 5y agoI still think this is an awesome project. Two questions: 1) What’s the story on type hints? I see pd.Series in the examples but have you done anything to offer more specific typing? 2) Have you considered pluggable extensions for data lineage platforms or supporting OpenLineage?
- mlplat 5y agoThanks! 1) Not limited to pd.Series -- any type works. So if you have a dataset or a way of expressing types it should just work. That said, pd.Series/DataFrames are convenient, and we have some special tooling around them. We're looking at tooling to add more expressiveness to datasets. Something like koalas type-hinting (there are a few approaches) might be helpful in this: https://koalas.readthedocs.io/en/latest/user_guide/typehints.html https://koalas.readthedocs.io/en/latest/user_guide/typehints.... With the type-equivalency check we added this should be fairly extensible. 2) Yes! Although not too extensively. I think OpenLineage would be a pretty simple thing to add -- just have to get the abstractions right.
- mlplat 5y agoAlso here's the ticket we have on lineage -- would love input if you have any! https://github.com/stitchfix/hamilton/issues/42 https://github.com/stitchfix/hamilton/issues/42