krawczstef

10 ms·

61.
The Intersection of DAGs, ML, & Complex Code Bases: An Elegant Solution Unveiled(podcasts.apple.com)
3 points by krawczstef 3y ago|2 comments
62.
by krawczstef 3y ago
I had been hearing these pains from Langchain users for quite a while. Suffice to say I think: 1. too many layers of OO abstractions are a liability in production contexts. I'm biased, but a more functional approach is a better way to
63.
by krawczstef 3y ago
Totally! As a person driving a project like https://github.com/DAGWorks-Inc/hamilton I couldn't agree more!
64.
by krawczstef 3y ago
From a nuts and bolts perspective, I've been thinking of building some reactivity on top of https://github.com/dagworks-inc/hamilton (author here) that could get at this. (If you have a use case that could be docu
65.
by krawczstef 3y ago
Reminds me of this project from Stitch Fix from years ago -- https://github.com/stitchfix/nodebook . Ahead of it's time I guess..
66.
by krawczstef 4y ago
Yep, I heard that when we open sourced Hamilton initially -- it was right around the time there was a "Bank Python" post floating around too. When I chatted with Travis O. at about the same time, he pointed that fact out, but he s
67.
by krawczstef 4y ago
> Even if you don't think this is a great use case, or your sweet spot, it's important to realize that it is very, very common, and when your users hit it they will look up how to deal with it. When the answer is "we can&#
68.
by krawczstef 4y ago
> There is no abstract, platonic solution that works equally well with 100 rows or 100 billion rows. Very true! One reason why navigating the options available is tough, since it's pretty dependent on your problem space.
69.
by krawczstef 4y ago
Yep - it's on my TODO to get https://github.com/flyteorg/flyte/issues/2627 over the line!
70.
by krawczstef 4y ago
> Ah, it seems like you're imagining Hamilton to be used to structure sort of "small" pieces, which might then be orchestrated into a Big Picture thing by another tool, something in the Airflow/Dagster/Argo/
71.
by krawczstef 4y ago
I'll let someone with more polars & duckdb experience to weigh in. But in short yes. Especially if you take the perspective they're both trying to help you do operations over tabular data, where the result is also something ta
72.
by krawczstef 4y ago
Would love! Yeah I think how we did the PySpark Map UDF support should enable us to do something similar with Quokka.
73.
by krawczstef 4y ago
> It's also interesting because this style of API is portable more or less unchanged to just about any programming language, from C# to Idris. Yep I think a declarative syntax is quite portable and can be reimplemented easily in oth
74.
by krawczstef 4y ago
Nice! Yep that's part of our vision to "compile onto", i.e. generate code for, frameworks such as yours. Happy to sketch something out - want to join our slack to chat https://join.slack.com/t/hamilton-op
75.
by krawczstef 4y ago
ah yep -- good catch -- d'oh -- and it's too late to update. But yes, the data scientists vs engineers should be flipped.
76.
by krawczstef 4y ago
Nice! We had a similar abstraction at Stitch Fix.
77.
by krawczstef 4y ago
Yep, we'd love more feedback on how to make the declarative syntax with Polars more natural with Hamilton so you can get the benefits of unit testing, documentation, visualization, swapping out implementations easily, etc.
78.
by krawczstef 4y ago
yep some of these ML platform tools only make sense once you reach a certain scale/set of problems. For flyte, it was created because Lyft had a very heterogeneous set of tools one could build a ML pipeline with. So they made it really
79.
by krawczstef 4y ago
Great questions! > "spicy" take: allowing users to write imperative code (e.g. using loops) that dynamically generates DAGs are never a good idea. Can you give some examples of when it was a bad idea? Otherwise to clarify, with
80.
by krawczstef 4y ago
To give you another take on Elijah's answer: > acquiring and moving data around, yep with Hamilton we provide the ability to cleanly separate bits of logic that's required to change and update this. For example you'd write
81.
by krawczstef 4y ago
Thanks Josh! I actually think we're pretty complementary, less competitive. For example, it'd be very conceivable for users to use Hamilton, DAGWorks, and Databricks together!
82.
Launch HN: DAGWorks – ML platform for data science teams
182 points by krawczstef 4y ago|65 comments
83.
by krawczstef 4y ago
Yes! As one of the creators of https://github.com/stitchfix/hamilton this was one of the aims. Simplifying the cognitive burden for those developing and managing data transforms over the course of years, and in particu
84.
Show HN: Lightweight Data Quality checks with Hamilton & Pandera(github.com)
1 points by krawczstef 4y ago|0 comments
85.
by krawczstef 5y ago
Author here. Yep, definitely don't think conceptually what we've done is unique, but the implementation/use is. The "trick" here, is that the end user doesn't need to know about these great software engineering
86.
by krawczstef 5y ago
Looks like an interesting (& powerful) library - what problem is it trying to solve exactly? That wasn't clear from the README (the greek's library doesn't shed any more light on things either).
87.
by krawczstef 5y ago
Yes I can see how you can make that leap. Right now Hamilton isn't about scaling computation per se, more so about scaling a code base to handle featurization and people using that code base. Hamilton wouldn't replace Dask. Instea
88.
by krawczstef 5y ago
Yeah we can make the links to the talk and blog post a bit more prominent -- but as to why? This should help answer that: https://multithreaded.stitchfix.com/blog/2021/10/14/function...
89.
by krawczstef 5y ago
That's good feedback thanks. I can make a section about problems, solution, & benefits. Would that work?
90.
by krawczstef 5y ago
That's a reasonable summary. Thanks! One note: we do have ideas to extend Hamilton to "compile" the DAG onto Spark/Dask/etc.

More ›