4 ms·

Curious what you like better?

by mchusma 6y ago

kingkonz1 6y ago
not OP, but I prefer https://github.com/azkaban/azkaban https://github.com/azkaban/azkaban to airflow
- zmmmmm 6y ago
  so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well. Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
  - ttymck 6y ago
    You raise a good point, there is an apparent need for an application that retains the dags-as-code but radically simplifies the airflow architecture.
    - rywalker 6y ago
      if you like YAML-based DAGs, you can do that in Airflow with the dag-factory extension https://github.com/ajbosco/dag-factory https://github.com/ajbosco/dag-factory
dfsegoat 6y ago
Luigi from Spotify: https://github.com/spotify/luigi https://github.com/spotify/luigi We’ve been using it for complex update workflows for about 5 yrs now, and it just works. It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.
- kumare3 6y ago
  Spotify is moving to https://Flyte.org https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned
six2seven 6y ago
Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow. I would be also curious to hear more about alternative and mature open-source solutions to Airflow. [1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...
- dimberman 6y ago
  I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably. Disclosure: I'm on the Airflow PMC.
  - mchusma 6y ago
    Thanks for your work on airflow. 2.0 is exciting and we are looking forward to it. Love airflow 1 though too, it has helped us a ton.
  - ElevenPhonons 6y ago
    Is there a list of backward incompatible changes? Also, how would you describe the overall health of the code base with all of these new features added?
x87678r 6y ago
https://dagster.io/ https://dagster.io/
mfateev 6y ago
temporal.io
- ttsda 6y ago
  This is it, the DAG model is a thing of the past

Curious what you like better?

kingkonz1 6y ago

not OP, but I prefer https://github.com/azkaban/azkaban https://github.com/azkaban/azkaban to airflow

zmmmmm 6y ago
so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well. Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
- ttymck 6y ago
  You raise a good point, there is an apparent need for an application that retains the dags-as-code but radically simplifies the airflow architecture.
  - rywalker 6y ago
    if you like YAML-based DAGs, you can do that in Airflow with the dag-factory extension https://github.com/ajbosco/dag-factory https://github.com/ajbosco/dag-factory

dfsegoat 6y ago

Luigi from Spotify: https://github.com/spotify/luigi https://github.com/spotify/luigi We’ve been using it for complex update workflows for about 5 yrs now, and it just works. It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.

kumare3 6y ago
Spotify is moving to https://Flyte.org https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned

six2seven 6y ago

Although this article [1] compares Airflow to Prefect it outlines pretty interestingly limitations of Airflow. I would be also curious to hear more about alternative and mature open-source solutions to Airflow. [1] https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

dimberman 6y ago
I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick Prefect over Airflow until they tried 2.0. We put a lot of work into it, including extensive QA time to ensure that it runs reliably. Disclosure: I'm on the Airflow PMC.
- mchusma 6y ago
  Thanks for your work on airflow. 2.0 is exciting and we are looking forward to it. Love airflow 1 though too, it has helped us a ton.
- ElevenPhonons 6y ago
  Is there a list of backward incompatible changes? Also, how would you describe the overall health of the code base with all of these new features added?

x87678r 6y ago

https://dagster.io/ https://dagster.io/

mfateev 6y ago

temporal.io

ttsda 6y ago
This is it, the DAG model is a thing of the past