3 ms·
I haven't evaluated how Hamilton is implemented specifically but: * It's solving a really different problem than Spark/Dask/etc. It could definitely use those
by sterlinm 5y ago
I haven't evaluated how Hamilton is implemented specifically but:
* It's solving a really different problem than Spark/Dask/etc. It could definitely use those tools but it's just not the same thing.
* If you're looking at this and thinking it's useless even if you're familiar with Pandas/dataframes it's probably just that you haven't had to work on the types of problems that this particular tool is intended to help with.
- krawczstef 5y agoThat's a reasonable summary. Thanks! One note: we do have ideas to extend Hamilton to "compile" the DAG onto Spark/Dask/etc.
- sterlinm 5y agoI should add I really like the general idea of what you're trying to do. I actually worked on some similar ideas for similar reasons (I think) a few years back. It was in Matlab* though and not as slick as what you've created. Is the idea with Hamilton to limit it specifically to columnar operations? e.g. no functions that filter the DataFrame? I think that's one thing that I attempted to do that ended up overcomplicating the framework I created. *womp womp