elijahbenizzy

15 ms·

1. 151.▲
   by elijahbenizzy 4y ago

   Interesting... So I'm not sure I agree. While Hamilton does support this kind of thing (and we're likely going to build out more support), the assumption above is that the unit of work at the task-level is natural to map to indivi
2. 152.▲
   by elijahbenizzy 4y ago

   Think you got it! Re factoring projects well, its interesting, but I think that there's some good strategies here. What's worked for us is working backwards -- starting with the artifact you want and progressively defining how you
3. 153.▲
   by elijahbenizzy 4y ago

   Thank you! We're not hiring just yet, but feel free to log onto the Hamilton OS slack and send us your resume -- happy to chat about future possibilities or connect you with people that are hiring in a similar space!
4. 154.▲
   by elijahbenizzy 4y ago

   Thank you! This is awesome to hear from someone so knowledgable on the space. Really great feedback :) > It's important to remember that DAG tools exist to solve two primary problems, that arise from one underlying cause. Those prob
5. 155.▲
   by elijahbenizzy 4y ago

   First, worth jumping on tryhamilton.dev -- we made it to make it easy to get started. And, as always, it depends on your use case! If it’s all analytics and DWH manipulations — that’s not Hamilton’s expertise. If you want to do any DS work
6. 156.▲
   by elijahbenizzy 4y ago

   Yeah! So I'm not an expert with domino, but looking at it, it serves a slightly different purpose. DAGWorks is an opinionated way to write code that allows you to abstract it away from the infrastructure, whereas Domino is more about m
7. 157.▲
   by elijahbenizzy 4y ago

   Yeah! So we actually have an integration with polars. See https://github.com/DAGWorks-Inc/hamilton/blob/5c8e564d19ff23... . To be clear, the specific paradigm we're referring to is this way of writing tra
8. 158.▲
   by elijahbenizzy 4y ago

   Thanks! Yeah -- love the example we built with the metaflow team. I even think Savin spoke right before me at PyData NYC in the same room!
9. 159.▲
   by elijahbenizzy 4y ago

   TL;DR -- ML platforms solve for a million different problems, and most people don't have all (or any) of them. Hamilton is a pretty simple way of organizing code, so it can plug in with a bunch of different approaches (and that's
10. 160.▲
    by elijahbenizzy 4y ago

    Thanks! Lots of plans in the works. Two directions we're moving: 1. Integrate with orchestration systems (E.G. run Hamilton pipelines on different orchestration platforms). You could imagine compiling to airflow pipelines, running hami
11. 161.▲
    by elijahbenizzy 4y ago

    Thanks! Appreciate it. I'm finding this space is massive and there's still more problems managing ML code than there are good solutions for it :) So lots of room for everyone.
12. 162.▲
    by elijahbenizzy 4y ago

    Thanks for the question! Clarification -- by "yield" do you mean "return a single function's result" or "use a function that's a generator"? In the former case, yes, it's pretty easy: driver.
13. 163.▲
    by elijahbenizzy 4y ago

    Thank you! Recognize your username -- two of our newer decorators are your design/suggestion :) https://hamilton.readthedocs.io/en/latest/reference/api-refe...
14. 164.▲
    by elijahbenizzy 4y ago

    Thanks for your question! Good points -- those are not easy problems! I'd be really curious about what the community thinks, but here are my opinions: While Hamilton/DAGWorks is mainly for expressing the pipeline/abstracting
15. 165.▲
    Show HN: The perks of creating dataflows with Hamilton(medium.com)

    4 points by elijahbenizzy 4y ago|0 comments
16. 166.▲
    Show HN: Validating Your Data with Hamilton(multithreaded.stitchfix.com)

    2 points by elijahbenizzy 4y ago|0 comments
17. 167.▲
    by elijahbenizzy 5y ago

    Author here -- that's awesome! Great minds think alike. Always interested in use-cases (that both fit and don't) -- feel free to open up an issue with something representative/what you want and I'm happy to contribute my
18. 168.▲
    by elijahbenizzy 5y ago

    Speaking as an author (so some bias), but you certainly make an interesting point about the cost of adoption. Its a high barrier to entry (requires a rewrite) for migration, but in our experience it actually is worth it. So yeah, not for ev
19. 169.▲
    by elijahbenizzy 5y ago

    One of the authors here — I think that systems that produce things magically often run into this burden. That doesn’t mean that the magic is bad, per se, rather that the thought put into operationalizing it (the user experience) is often &l