3 ms·
Thanks! 1) Not limited to pd.Series -- any type works. So if you have a dataset or a way of expressing types it should just work. That said, pd.Series/DataFram
by mlplat 5y ago
Thanks!
1) Not limited to pd.Series -- any type works. So if you have a dataset or a way of expressing types it should just work. That said, pd.Series/DataFrames are convenient, and we have some special tooling around them. We're looking at tooling to add more expressiveness to datasets. Something like koalas type-hinting (there are a few approaches) might be helpful in this: https://koalas.readthedocs.io/en/latest/user_guide/typehints.html https://koalas.readthedocs.io/en/latest/user_guide/typehints.... With the type-equivalency check we added this should be fairly extensible.
2) Yes! Although not too extensively. I think OpenLineage would be a pretty simple thing to add -- just have to get the abstractions right.