2 ms·

> So it's like spark for pandas? I didn't get that at all. Spark is a data-processing engine. Hamilton seems to be a framework for expressing all of your fea

> So it's like spark for pandas? I didn't get that at all. Spark is a data-processing engine. Hamilton seems to be a framework for expressing all of your feature extractions and transformations in such a way that they can be encapsulated and organized into functions and modules. These functions can then be composed into a transformation pipeline instructions. Hamilton can then inspect this composition of functions and understand all of the inter-dependencies, effectively constructing a DAG that can be used to visualize what exactly the pipeline is doing while also creating some feature/column lineage. This DAG can also be used to figure out what functionality can be pruned from your modules if it is never used. Unit tests written easily. There is nothing within Hamilton that dictates which data-processing engine should be used to do the work. In fact they mention it in their blog that Spark could indeed be used. But I imagine anyone could come along and write an integration for Pandas (if it doesn't already exist), BigQuery, Snowflake, Materialize, etc... This creates an abstraction between what you're trying to do, and what data processing engine is used to do it. Not to mention being able to create some fantastic visualizations (i assume) of what is actually going on in your transformation or feature-extraction pipeline.

zmmmmm 5y ago
> These functions can then be composed into a transformation pipeline instructions. Hamilton can then inspect this composition of functions and understand all of the inter-dependencies, effectively constructing a DAG that can be used to visualize what exactly the pipeline is doing It isn't immediately obvious but that is exactly what Spark does - it builds a DAG out of your operations and only lazily executes what it has to when asked for a result. But on top of that, a tremendous amount of work has gone into auto-optimising that DAG execution so it can be very very smart while allowing relatively natural expression of queries and transformations at the top level. This is all going to have to be re-invented if you try and replace the DAG front end and just use spark's execution back end.
- chrisjc 5y ago
  > This is all going to have to be re-invented if you try and replace the DAG front end and just use spark's execution back end. Again, I don't know enough about Hamilton, but I don't think it's a matter of replacing anything Spark does, or any other data execution runtime for that matter. It's just another layer, another abstraction. Hamilton -> Spark SQL/API -> Spark artifact -> Spark execution Hamilton -> Snowflake SQL -> Snowflake execution etc...

zmmmmm 5y ago

> These functions can then be composed into a transformation pipeline instructions. Hamilton can then inspect this composition of functions and understand all of the inter-dependencies, effectively constructing a DAG that can be used to visualize what exactly the pipeline is doing It isn't immediately obvious but that is exactly what Spark does - it builds a DAG out of your operations and only lazily executes what it has to when asked for a result. But on top of that, a tremendous amount of work has gone into auto-optimising that DAG execution so it can be very very smart while allowing relatively natural expression of queries and transformations at the top level. This is all going to have to be re-invented if you try and replace the DAG front end and just use spark's execution back end.

chrisjc 5y ago
> This is all going to have to be re-invented if you try and replace the DAG front end and just use spark's execution back end. Again, I don't know enough about Hamilton, but I don't think it's a matter of replacing anything Spark does, or any other data execution runtime for that matter. It's just another layer, another abstraction. Hamilton -> Spark SQL/API -> Spark artifact -> Spark execution Hamilton -> Snowflake SQL -> Snowflake execution etc...