4 ms·
Sure! We wrote a little bit about the origins in an announce post a couple weeks ago (https://news.ycombinator.com/item?id=35366838 https://news.ycombinator.com
by jlowin 4y ago
Sure! We wrote a little bit about the origins in an announce post a couple weeks ago (https://news.ycombinator.com/item?id=35366838 https://news.ycombinator.com/item?id=35366838).
Marvin (https://www.github.com/prefecthq/marvin https://www.github.com/prefecthq/marvin) powers our AI efforts at Prefect (https://www.github.com/prefecthq/prefect https://www.github.com/prefecthq/prefect).
The first version of Marvin was an internal framework that powered our Slackbot. There are close to 30,000 members of our open-source community and we rely heavily on automation to deliver support. Then, as more of our customers started building AI stacks, we began to view Marvin as a platform to experiment with high-level UX for deploying AI. We have a few internal use cases, but it was the diversity of customer objectives that gave us confidence.
Historically, we've always focused on data engineering, but the more we worked with LLMs, the more we saw the same set of issues, basically driven by the need to integrate brittle, non-deterministic APIs that are heavily influenced by external state into well-structured traditional engineering and pipelines. We started using Marvin to codify the high-level patterns we were repeatedly deploying, including getting structured outputs from the LLM and building effective conversational agents for B2B use.
The lightbulb moment was when we designed AI functions, which have no source code and essentially use the LLM as a runtime. It's one of those ideas that feels too simple to actually work... but it actually works incredibly well. It was the first time we felt like we weren't building tools to use AI, but rather using AI to build our tools. We open-sourced with AI functions as the headline and the response has been amazing! Now we're focused on releasing the "core" of Marvin -- the bots, plugins, and knowledge handling -- with a similar focus on usability.
Hope that's what you were looking for!
- darkteflon 4y agoSuperb, that’s very helpful, thanks! I’m going to add this to our toolkit and use it alongside our langchain experiments. The space has a real “thousand flowers” feel to it atm. At risk of running out my question quota: we use dagster at the moment but have found that it has some unexpected rough edges. Would you be able to point me in the direction of your preferred comparison against Prefect?
- jlowin 4y agoNo quota here! If you like Marvin, you’ll probably like Prefect. Both are designed to be clean, Pythonic interfaces to a complex and hard-to-observe system (a data stack; an AI stack). I think one of the key differences between Prefect and Dagster is that Prefect views orchestration as coordination, while Dagster views orchestration as reconciliation. The data stack is a complex system whose state is frequently mutated by forces outside our users’ control. Therefore, our product is focused on letting our users understand and react to those events, no matter where they come from. That could include everything from scheduling fully-orchestrated Prefect pipelines, to setting an SLA for database maintenance that Prefect doesn’t have anything else to do with. Reconciliation, in contrast, requires users to define a digital twin of their stack, in order to serve as ground truth and become the reconciliation target. Philosophically, we view Prefect as one piece of an ever-changing stack. We are focused on being as flexible as possible to fit into the stack, rather than the other way around.
- darkteflon 4y agoExcellent! Thanks so much for your time, it’s very helpful and I really appreciate it.
- aaauaucuggaa 4y agoHop on the slack, there are plenty of folks (myself included) who are happy to help get you started. I found prefect after slogging through a year of painful development work on an Airflow project, and I have been a rabid fan of it ever since. Been using it at my current gig for 2+ years, on an old version of the open source offering. Despite all that it's been a dream to work with.
- jlowin 4y agoAnytime! Thanks for checking out our work!