3 ms·

Superb, that’s very helpful, thanks! I’m going to add this to our toolkit and use it alongside our langchain experiments. The space has a real “thousand flowers

Superb, that’s very helpful, thanks! I’m going to add this to our toolkit and use it alongside our langchain experiments. The space has a real “thousand flowers” feel to it atm. At risk of running out my question quota: we use dagster at the moment but have found that it has some unexpected rough edges. Would you be able to point me in the direction of your preferred comparison against Prefect?

jlowin 4y ago
No quota here! If you like Marvin, you’ll probably like Prefect. Both are designed to be clean, Pythonic interfaces to a complex and hard-to-observe system (a data stack; an AI stack). I think one of the key differences between Prefect and Dagster is that Prefect views orchestration as coordination, while Dagster views orchestration as reconciliation. The data stack is a complex system whose state is frequently mutated by forces outside our users’ control. Therefore, our product is focused on letting our users understand and react to those events, no matter where they come from. That could include everything from scheduling fully-orchestrated Prefect pipelines, to setting an SLA for database maintenance that Prefect doesn’t have anything else to do with. Reconciliation, in contrast, requires users to define a digital twin of their stack, in order to serve as ground truth and become the reconciliation target. Philosophically, we view Prefect as one piece of an ever-changing stack. We are focused on being as flexible as possible to fit into the stack, rather than the other way around.
- darkteflon 4y ago
  Excellent! Thanks so much for your time, it’s very helpful and I really appreciate it.
  - aaauaucuggaa 4y ago
    Hop on the slack, there are plenty of folks (myself included) who are happy to help get you started. I found prefect after slogging through a year of painful development work on an Airflow project, and I have been a rabid fan of it ever since. Been using it at my current gig for 2+ years, on an old version of the open source offering. Despite all that it's been a dream to work with.
  - jlowin 4y ago
    Anytime! Thanks for checking out our work!

jlowin 4y ago

No quota here! If you like Marvin, you’ll probably like Prefect. Both are designed to be clean, Pythonic interfaces to a complex and hard-to-observe system (a data stack; an AI stack). I think one of the key differences between Prefect and Dagster is that Prefect views orchestration as coordination, while Dagster views orchestration as reconciliation. The data stack is a complex system whose state is frequently mutated by forces outside our users’ control. Therefore, our product is focused on letting our users understand and react to those events, no matter where they come from. That could include everything from scheduling fully-orchestrated Prefect pipelines, to setting an SLA for database maintenance that Prefect doesn’t have anything else to do with. Reconciliation, in contrast, requires users to define a digital twin of their stack, in order to serve as ground truth and become the reconciliation target. Philosophically, we view Prefect as one piece of an ever-changing stack. We are focused on being as flexible as possible to fit into the stack, rather than the other way around.

darkteflon 4y ago
Excellent! Thanks so much for your time, it’s very helpful and I really appreciate it.
- aaauaucuggaa 4y ago
  Hop on the slack, there are plenty of folks (myself included) who are happy to help get you started. I found prefect after slogging through a year of painful development work on an Airflow project, and I have been a rabid fan of it ever since. Been using it at my current gig for 2+ years, on an old version of the open source offering. Despite all that it's been a dream to work with.
- jlowin 4y ago
  Anytime! Thanks for checking out our work!