3 ms·
I have started using prefect.io as a workflow runner. It helps me define tasks and deeply define what should happen on failure of a task including cleaning up a
by faizshah 7y ago
I have started using prefect.io as a workflow runner. It helps me define tasks and deeply define what should happen on failure of a task including cleaning up a cluster.
Because Tasks are just functions you could test your tasks with regular unit testing and mocking methods.
But I agree a lot of data people seem to think interactive is good enough.
- edraferi 7y agoHow do you think Prefect compares to Airflow / Dagster?
- faizshah 7y agoWhat makes prefect stand out is the highly pythonic API and the powerful primitives for defining failure scenarios. Additionally, for me, integrating with dask is key as dask has become one of my main tools. Heres a rewrite of the airflow tutorial in prefect: https://docs.prefect.io/core/examples/airflow_tutorial_dag.html https://docs.prefect.io/core/examples/airflow_tutorial_dag.h... My advice is try it out and see if you like the api. I think that the airflow ui is still a killer feature in 2019 though. Theres also this post from the creator of prefect on why not airflow: https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232... I haven’t tried dagster yet, how does it compare to airflow?
- kfk 7y agoBy the way prefect is mostly a wrapper around dask.delayed