5 ms·

But why Airflow, it has so many weird things. I hope it is dethroned soon.

rywalker 6y ago
fyi airflow 2.0 ships in the next few weeks, we're made a lot of improvements https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0tXCMgzh5c_Pj https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0...
- numbsafari 6y ago
  It'd be really helpful if there were an article that folks could read. Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video. This trend towards posting video content instead of, rather than along-side, written content is pretty anti-accessbility.
  - manigandham 6y ago
    Here's an article from the same source: https://www.astronomer.io/blog/introducing-airflow-2-0/ https://www.astronomer.io/blog/introducing-airflow-2-0/
- dividedbyzero 6y ago
  Do you have an article or something I can skim/scan more quickly? Not going to invest over an hour to watch those videos.
riyadparvez 6y ago
I have read Airflow 2.0 release addresses lot of long-term cirticisms of Airflow. Hope anyone here experience with Airflow can illuminate us.
soamv 6y ago
Because people use it. It's been around a while and has an ecosystem around it. (This is the interesting thing about AWS, they keep the focus on what people do use, rather than some opinionated idea of what they should use.)
mchusma 6y ago
Curious what you like better?
- kingkonz1 6y ago
  not OP, but I prefer https://github.com/azkaban/azkaban https://github.com/azkaban/azkaban to airflow
  - zmmmmm 6y ago
    so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well. Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
    - ttymck 6y ago
      You raise a good point, there is an apparent need for an application that retains the dags-as-code but radically simplifies the airflow architecture.
      - rywalker 6y ago
        if you like YAML-based DAGs, you can do that in Airflow with the dag-factory extension https://github.com/ajbosco/dag-factory https://github.com/ajbosco/dag-factory
- dfsegoat 6y ago
  Luigi from Spotify: https://github.com/spotify/luigi https://github.com/spotify/luigi We’ve been using it for complex update workflows for about 5 yrs now, and it just works. It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.
  - kumare3 6y ago
    Spotify is moving to https://Flyte.org https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned
seddonm1 6y ago
I am hoping Argo Workflows (https://argoproj.github.io/projects/argo https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.
- drewda 6y ago
  Agreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-workflows/ https://www.interline.io/blog/scaling-openstreetmap-data-wor...
- vanpythonista 6y ago
  I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.
  - seddonm1 6y ago
    Agree that YAML is not really ideal for this case. For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results https://argoproj.github.io/argo/examples/#scripts-results Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines/ https://www.kubeflow.org/docs/components/pipelines/pipelines...
  - kumare3 6y ago
    Have you tried Flyte - flyte.org?
- lxglv 6y ago
  Running something on Kubernetes does not make this something highly available. It makes this something randomly restarting.
blakeburch 6y ago
Agreed. It's the golden standard for data workflows right now, but it's pretty cumbersome to work with. I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.
emehex 6y ago
Me too. It's so complicated! If you just need to schedule something without any dependencies, maybe try: https://github.com/maxhumber/hickory https://github.com/maxhumber/hickory Disclaimer: I'm the creator.
arthurcolle 6y ago
Lmao this is a great comment - I have found so many bizarre eccentricities with Airflow, but it is still, in general, quite nice to build with. What are you thinking about?

fyi airflow 2.0 ships in the next few weeks, we're made a lot of improvements https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0tXCMgzh5c_Pj https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0...

numbsafari 6y ago
It'd be really helpful if there were an article that folks could read. Consuming video instead of text assumes that (1) I can hear and (2) I have the time to sit and listen at your speaking pace and (3) I'm in a place where I can sit with the audio turned up (4) I really want to expend the extra bandwidth for your video. This trend towards posting video content instead of, rather than along-side, written content is pretty anti-accessbility.
- manigandham 6y ago
  Here's an article from the same source: https://www.astronomer.io/blog/introducing-airflow-2-0/ https://www.astronomer.io/blog/introducing-airflow-2-0/
dividedbyzero 6y ago
Do you have an article or something I can skim/scan more quickly? Not going to invest over an hour to watch those videos.

riyadparvez 6y ago

I have read Airflow 2.0 release addresses lot of long-term cirticisms of Airflow. Hope anyone here experience with Airflow can illuminate us.

soamv 6y ago

Because people use it. It's been around a while and has an ecosystem around it. (This is the interesting thing about AWS, they keep the focus on what people do use, rather than some opinionated idea of what they should use.)

mchusma 6y ago

Curious what you like better?

kingkonz1 6y ago
not OP, but I prefer https://github.com/azkaban/azkaban https://github.com/azkaban/azkaban to airflow
- zmmmmm 6y ago
  so, workflows described in yaml? I also am a bit ambivalent about Airflow and the way workflows are built up in it, but YAML seems really weak and unlikely to handle complexity well. Also - it's entirely built around Hadoop from what I can see? Seems a limited use case compared to Airflow.
  - ttymck 6y ago
    You raise a good point, there is an apparent need for an application that retains the dags-as-code but radically simplifies the airflow architecture.
    - rywalker 6y ago
      if you like YAML-based DAGs, you can do that in Airflow with the dag-factory extension https://github.com/ajbosco/dag-factory https://github.com/ajbosco/dag-factory
dfsegoat 6y ago
Luigi from Spotify: https://github.com/spotify/luigi https://github.com/spotify/luigi We’ve been using it for complex update workflows for about 5 yrs now, and it just works. It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.
- kumare3 6y ago
  Spotify is moving to https://Flyte.org https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned

seddonm1 6y ago

I am hoping Argo Workflows (https://argoproj.github.io/projects/argo https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.

drewda 6y ago
Agreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-workflows/ https://www.interline.io/blog/scaling-openstreetmap-data-wor...
vanpythonista 6y ago
I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.
- seddonm1 6y ago
  Agree that YAML is not really ideal for this case. For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results https://argoproj.github.io/argo/examples/#scripts-results Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines/ https://www.kubeflow.org/docs/components/pipelines/pipelines...
- kumare3 6y ago
  Have you tried Flyte - flyte.org?
lxglv 6y ago
Running something on Kubernetes does not make this something highly available. It makes this something randomly restarting.

blakeburch 6y ago

Agreed. It's the golden standard for data workflows right now, but it's pretty cumbersome to work with. I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.

emehex 6y ago

Me too. It's so complicated! If you just need to schedule something without any dependencies, maybe try: https://github.com/maxhumber/hickory https://github.com/maxhumber/hickory Disclaimer: I'm the creator.

arthurcolle 6y ago

Lmao this is a great comment - I have found so many bizarre eccentricities with Airflow, but it is still, in general, quite nice to build with. What are you thinking about?