6 ms·

Hamilton: A Microframework for Dataframe Generation

56 points by gammarator 5y ago

wxnx 5y ago
This is awesome to see! I had a very similar idea recently (representing "big" preprocessing pipelines in Python as DAGs), but for research-scale projects. Funnily enough, I was also motivated by a time series project that has resulted in several thousand lines of gnarly preprocessing code, and growing - mostly in Pandas. I'm not sure Hamilton would totally work for our use case, but I'll be following closely either way. Thank you for open sourcing this!
- elijahbenizzy 5y ago
  Author here -- that's awesome! Great minds think alike. Always interested in use-cases (that both fit and don't) -- feel free to open up an issue with something representative/what you want and I'm happy to contribute my opinion.
spratzt 5y ago
Looks great and I shall definitely try it. I've always thought that Stitchfix were the most interesting ecommerce retail operation. It's a shame that none of their data science roles are available in London.
imcoconut 5y ago
This looks amazing, and very timely. Any chance you have a mirror on gitlab (or github)?
- zxexz 5y ago
  This[0] was linked in the article. [0] https://github.com/stitchfix/hamilton https://github.com/stitchfix/hamilton
dash2 5y ago
Nice. I've really felt the pain point here. In R I use Drake to run a big pipeline on my laptop. In theory that provides the same "only run the computations you need" DAG, but because outputs are defined as whole data frames, changing or adding a single column often forces recomputation of everything. Column level isolation like this is the smart way to go.
__mharrison__ 5y ago
I would love to see more real-life examples. My take is that most people don't know about the .assign method [0] or severely underutilize it. 0 - https://www.metasnake.com/blog/pydata-assign.html https://www.metasnake.com/blog/pydata-assign.html
- pvitz 5y ago
  I don't understand the advantage of the assign-method in the article you have linked to. If I would like to alter data in a column of a large dataframe, why should I use the assign-method (and copy data) instead of the mentioned loc-method and just overwrite the data (without a copy)? I am assuming, of course, that I will not need the original dataframe anymore.
  - lmeyerov 5y ago
    Yep, we regularly use assign & pipe to avoid the 80% case I think the article is about: ``` df2 = df.assign(new_col_1=f(df)) ``` or ``` def add_new_col_1(df): return df.assign(new_col_1=...) df2 = df.pipe(add_new_col_1) ``` We do use reified compute DAGs in some places, but by that point, we're using dask anyways, and that kind of code ends up more annoying than if we could avoid. Speaking as someone who has done years of FRP/streams/etc., if users can stick with direct control flow or things that look like it (async/await), help them do it :) RE:Immutability, it's a convention that eliminates some classes of bugs, so quite nice when a team does it. Code written by senior / reviewed teams are nice b/c these things add up to either a pleasant experience or perpetual paranoia for the day-to-day: * A lot of our code is in notebook envs, where the ability to move cells up/down matters, so we try to only do single assignments to avoid non-reproducibility bugs * Similar in production, but more about when someone comes back and wants to edit or log. having to undo the name reuse is annoying, and may lead to bugs when you don't notice it
lr1970 5y ago
An earlier thread 23 days ago [0] [0] https://news.ycombinator.com/item?id=29158021 https://news.ycombinator.com/item?id=29158021
akdor1154 5y ago
Very interesting. In general i'm defaulting to "ahh not another syntax/grammar to learn!", but i can absolutely appreciate that a code base written with this would be far better than the messes of opaque pandas wrangling it would otherwise involve. I really appreciate API design that consciously encourages clean code, and on that basis this looks fantastic.
jcmontx 5y ago
Thanks I'll stick to Verstappen
feoren 5y ago
It always amazes me how readily people assume that their problem is unique and special. This article can be summarized as: standard software engineering problem gets solved with standard software engineering practices. The problem they describe of a giant function that gets touched every time something changes is called a "God Function" or a "God Class" and it's one of the most common problems in software engineering. Their solution is called "Dependency Inversion" and it was designed to solve literally that exact problem in the exact way they're doing it. That doesn't mean it's easy! Anyone who applies Software Engineering principles to make their code better deserves congratulations. So congratulations, Stitchfix! But you could have found this solution a lot quicker if you didn't fall into that age old trap of thinking your data-science code was different, and excluded from standard software engineering. Everyone does this, and I don't know why. They read all these blogs and books on software engineering best practices, but then they don't apply them to anything. You can't apply it to your SQL because it's just different. Your database organization is different. Your I/O code is different. Your API surface is different. Your UI is different. Your feature engineering is different. No! It's software. Stop thinking that you need a unique solution to your unique snowflake problem. Apply well-known and good software engineering practices to everything you ever do. They even say in the article "It hasn’t grown convoluted out of ... bad software engineering practices". Of course it has! What do you think software engineering practices are supposed to apply to!?
- deleted 5y ago
  [deleted]
- mtVessel 5y ago
  In my experience, many data engineers and data scientists know nothing about software engineering, unless they've had a previous life as a coder. Most data engineers who claim they know python really only know pandas, and many couldn't implement fizzbuzz without a library. I'm a data guy, but I started out as a developer, and eventually choose to specialize in data systems. I'm continually amazed to find that this path is the exception.
  - inadequatespace 5y ago

56 points by gammarator 5y ago

wxnx 5y ago

This is awesome to see! I had a very similar idea recently (representing "big" preprocessing pipelines in Python as DAGs), but for research-scale projects. Funnily enough, I was also motivated by a time series project that has resulted in several thousand lines of gnarly preprocessing code, and growing - mostly in Pandas. I'm not sure Hamilton would totally work for our use case, but I'll be following closely either way. Thank you for open sourcing this!

elijahbenizzy 5y ago
Author here -- that's awesome! Great minds think alike. Always interested in use-cases (that both fit and don't) -- feel free to open up an issue with something representative/what you want and I'm happy to contribute my opinion.

spratzt 5y ago

Looks great and I shall definitely try it. I've always thought that Stitchfix were the most interesting ecommerce retail operation. It's a shame that none of their data science roles are available in London.

imcoconut 5y ago

This looks amazing, and very timely. Any chance you have a mirror on gitlab (or github)?

zxexz 5y ago
This[0] was linked in the article. [0] https://github.com/stitchfix/hamilton https://github.com/stitchfix/hamilton

dash2 5y ago

Nice. I've really felt the pain point here. In R I use Drake to run a big pipeline on my laptop. In theory that provides the same "only run the computations you need" DAG, but because outputs are defined as whole data frames, changing or adding a single column often forces recomputation of everything. Column level isolation like this is the smart way to go.

__mharrison__ 5y ago

I would love to see more real-life examples. My take is that most people don't know about the .assign method [0] or severely underutilize it. 0 - https://www.metasnake.com/blog/pydata-assign.html https://www.metasnake.com/blog/pydata-assign.html

pvitz 5y ago
I don't understand the advantage of the assign-method in the article you have linked to. If I would like to alter data in a column of a large dataframe, why should I use the assign-method (and copy data) instead of the mentioned loc-method and just overwrite the data (without a copy)? I am assuming, of course, that I will not need the original dataframe anymore.
- lmeyerov 5y ago
  Yep, we regularly use assign & pipe to avoid the 80% case I think the article is about: ``` df2 = df.assign(new_col_1=f(df)) ``` or ``` def add_new_col_1(df): return df.assign(new_col_1=...) df2 = df.pipe(add_new_col_1) ``` We do use reified compute DAGs in some places, but by that point, we're using dask anyways, and that kind of code ends up more annoying than if we could avoid. Speaking as someone who has done years of FRP/streams/etc., if users can stick with direct control flow or things that look like it (async/await), help them do it :) RE:Immutability, it's a convention that eliminates some classes of bugs, so quite nice when a team does it. Code written by senior / reviewed teams are nice b/c these things add up to either a pleasant experience or perpetual paranoia for the day-to-day: * A lot of our code is in notebook envs, where the ability to move cells up/down matters, so we try to only do single assignments to avoid non-reproducibility bugs * Similar in production, but more about when someone comes back and wants to edit or log. having to undo the name reuse is annoying, and may lead to bugs when you don't notice it

lr1970 5y ago

An earlier thread 23 days ago [0] [0] https://news.ycombinator.com/item?id=29158021 https://news.ycombinator.com/item?id=29158021

akdor1154 5y ago

Very interesting. In general i'm defaulting to "ahh not another syntax/grammar to learn!", but i can absolutely appreciate that a code base written with this would be far better than the messes of opaque pandas wrangling it would otherwise involve. I really appreciate API design that consciously encourages clean code, and on that basis this looks fantastic.

jcmontx 5y ago

Thanks I'll stick to Verstappen

feoren 5y ago

It always amazes me how readily people assume that their problem is unique and special. This article can be summarized as: standard software engineering problem gets solved with standard software engineering practices. The problem they describe of a giant function that gets touched every time something changes is called a "God Function" or a "God Class" and it's one of the most common problems in software engineering. Their solution is called "Dependency Inversion" and it was designed to solve literally that exact problem in the exact way they're doing it. That doesn't mean it's easy! Anyone who applies Software Engineering principles to make their code better deserves congratulations. So congratulations, Stitchfix! But you could have found this solution a lot quicker if you didn't fall into that age old trap of thinking your data-science code was different, and excluded from standard software engineering. Everyone does this, and I don't know why. They read all these blogs and books on software engineering best practices, but then they don't apply them to anything. You can't apply it to your SQL because it's just different. Your database organization is different. Your I/O code is different. Your API surface is different. Your UI is different. Your feature engineering is different. No! It's software. Stop thinking that you need a unique solution to your unique snowflake problem. Apply well-known and good software engineering practices to everything you ever do. They even say in the article "It hasn’t grown convoluted out of ... bad software engineering practices". Of course it has! What do you think software engineering practices are supposed to apply to!?

deleted 5y ago
[deleted]
mtVessel 5y ago
In my experience, many data engineers and data scientists know nothing about software engineering, unless they've had a previous life as a coder. Most data engineers who claim they know python really only know pandas, and many couldn't implement fizzbuzz without a library. I'm a data guy, but I started out as a developer, and eventually choose to specialize in data systems. I'm continually amazed to find that this path is the exception.
- inadequatespace 5y ago