3 ms·
That's a reasonable summary. Thanks! One note: we do have ideas to extend Hamilton to "compile" the DAG onto Spark/Dask/etc.
by krawczstef 5y ago
That's a reasonable summary. Thanks!
One note: we do have ideas to extend Hamilton to "compile" the DAG onto Spark/Dask/etc.
- sterlinm 5y agoI should add I really like the general idea of what you're trying to do. I actually worked on some similar ideas for similar reasons (I think) a few years back. It was in Matlab* though and not as slick as what you've created. Is the idea with Hamilton to limit it specifically to columnar operations? e.g. no functions that filter the DataFrame? I think that's one thing that I attempted to do that ended up overcomplicating the framework I created. *womp womp