3 ms·

Yeah! So we actually have an integration with polars. See https://github.com/DAGWorks-Inc/hamilton/blob/5c8e564d19ff2386b00a1ef71de43d11205c5273/examples/polars

Yeah! So we actually have an integration with polars. See https://github.com/DAGWorks-Inc/hamilton/blob/5c8e564d19ff2386b00a1ef71de43d11205c5273/examples/polars/README.md https://github.com/DAGWorks-Inc/hamilton/blob/5c8e564d19ff23.... To be clear, the specific paradigm we're referring to is this way of writing transforms as functions where the parameter name is the upstream dependency -- not the notion of delayed execution. I think there are two different concepts here though: 1. How the transforms are executed 2. How the transforms are organized Hamilton cares about (2) and delegates to Polars/pandas for (1). The problem we're trying to solve is the code getting messy and transforms being poorly documented/hard to own -- Hamilton isn't going to solve the problem of optimizing compute as tooling like polars, pandas, and pyspark can handle that quite well.

krawczstef 4y ago
Yep, we'd love more feedback on how to make the declarative syntax with Polars more natural with Hamilton so you can get the benefits of unit testing, documentation, visualization, swapping out implementations easily, etc.
brahbrah 4y ago
Not to take away from anything you’ve done here, you guys have put a lot of great effort into this, but this paradigm is not “new”. It’s a common modeling paradigm in banks and hedge funds at least. I’ve built/worked on frameworks based on exactly this concept at 2 previous firms. Here’s some open source examples of the same concept: pyungo, fn_graph, Loman
- elijahbenizzy 4y ago
  Oh this is great! I knew about fn_graph (claimed "new" as we created this before fn_graph, but OS'd afterwards) -- I think we're talking with the author soon. But the others are awesome. I used to work at a hedge fund and I think this way of thinking came pretty naturally to me...
- krawczstef 4y ago
  Yep, I heard that when we open sourced Hamilton initially -- it was right around the time there was a "Bank Python" post floating around too. When I chatted with Travis O. at about the same time, he pointed that fact out, but he said something like: "oh cool, you can do column level *and* row level computation. Nice." So I interpreted that some places don't have the flexibility that Hamilton has? Otherwise yeah, those other libraries use the same concept, but it's interesting to see the very different UXs with them.

krawczstef 4y ago

Yep, we'd love more feedback on how to make the declarative syntax with Polars more natural with Hamilton so you can get the benefits of unit testing, documentation, visualization, swapping out implementations easily, etc.

brahbrah 4y ago

Not to take away from anything you’ve done here, you guys have put a lot of great effort into this, but this paradigm is not “new”. It’s a common modeling paradigm in banks and hedge funds at least. I’ve built/worked on frameworks based on exactly this concept at 2 previous firms. Here’s some open source examples of the same concept: pyungo, fn_graph, Loman

elijahbenizzy 4y ago
Oh this is great! I knew about fn_graph (claimed "new" as we created this before fn_graph, but OS'd afterwards) -- I think we're talking with the author soon. But the others are awesome. I used to work at a hedge fund and I think this way of thinking came pretty naturally to me...
krawczstef 4y ago
Yep, I heard that when we open sourced Hamilton initially -- it was right around the time there was a "Bank Python" post floating around too. When I chatted with Travis O. at about the same time, he pointed that fact out, but he said something like: "oh cool, you can do column level *and* row level computation. Nice." So I interpreted that some places don't have the flexibility that Hamilton has? Otherwise yeah, those other libraries use the same concept, but it's interesting to see the very different UXs with them.