4 ms·

I think the point is that it doesn't do anything (data processing). Instead it's a way to create a set of directives about how to do something, and at the same

I think the point is that it doesn't do anything (data processing). Instead it's a way to create a set of directives about how to do something, and at the same time understand the inter-dependencies of those directives. The blog post gets into the problems created by doing "something simple". https://multithreaded.stitchfix.com/blog/2021/10/14/functions-dags-hamilton/ https://multithreaded.stitchfix.com/blog/2021/10/14/function...

jstx1 5y ago
Thanks, the more I understand this, the less I like it. I'm still not seeing the usefulness but I see a lot of potential problems and complications from redesigning your codebase like this. I work with pandas dataframes all the time. It really looks like they're forcing a functional paradigm in a place where it doesn't fit well.
- chrisjc 5y ago
  You'll have to excuse my ignorance about Pandas... But I believe what this would enable you to do is continue writing your data processing pipelines and then have them execute in Pandas (assuming there is a Pandas implementation) while you're working locally. When it comes time to deploy your transformations to a production environment that uses Spark, Snowflake, etc... for its data processing engine (assuming integrations exist), it should work just the same way. Keep in mind, that these are instructions that are being deployed. The power here is that the instructions are pushed down into the data processing engine. Data is not exhumed into the dataframe runtime! Hamilton functions can be "compiled" into data-engine runtime code (pandas, SparkSQL/SparkDF, SQL, etc). Then on top of all of this, you (and more importantly your teammates and/or replacements) will have a better visualization of what your pipeline is doing. You'll also have a better idea of what unused functionality you can prune out of your code-base. Better unit testing. etc...

jstx1 5y ago

Thanks, the more I understand this, the less I like it. I'm still not seeing the usefulness but I see a lot of potential problems and complications from redesigning your codebase like this. I work with pandas dataframes all the time. It really looks like they're forcing a functional paradigm in a place where it doesn't fit well.

chrisjc 5y ago
You'll have to excuse my ignorance about Pandas... But I believe what this would enable you to do is continue writing your data processing pipelines and then have them execute in Pandas (assuming there is a Pandas implementation) while you're working locally. When it comes time to deploy your transformations to a production environment that uses Spark, Snowflake, etc... for its data processing engine (assuming integrations exist), it should work just the same way. Keep in mind, that these are instructions that are being deployed. The power here is that the instructions are pushed down into the data processing engine. Data is not exhumed into the dataframe runtime! Hamilton functions can be "compiled" into data-engine runtime code (pandas, SparkSQL/SparkDF, SQL, etc). Then on top of all of this, you (and more importantly your teammates and/or replacements) will have a better visualization of what your pipeline is doing. You'll also have a better idea of what unused functionality you can prune out of your code-base. Better unit testing. etc...