3 ms·
I am hoping Argo Workflows (https://argoproj.github.io/projects/argo https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration
by seddonm1 6y ago
I am hoping Argo Workflows (https://argoproj.github.io/projects/argo https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.
- drewda 6y agoAgreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-workflows/ https://www.interline.io/blog/scaling-openstreetmap-data-wor...
- vanpythonista 6y agoI like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.
- seddonm1 6y agoAgree that YAML is not really ideal for this case. For other readers Argo does support 'Scripts' which means you can implement complex logic similar to Airflow if required but can be a bit clumsy: https://argoproj.github.io/argo/examples/#scripts-results https://argoproj.github.io/argo/examples/#scripts-results Additionally, I think we may see other layers on top of Argo in future that may abstract from the YAML similar to how Kubeflow uses Argo for the Kubeflow Pipelines capability: https://www.kubeflow.org/docs/components/pipelines/pipelines/ https://www.kubeflow.org/docs/components/pipelines/pipelines...
- kumare3 6y agoHave you tried Flyte - flyte.org?
- lxglv 6y agoRunning something on Kubernetes does not make this something highly available. It makes this something randomly restarting.
- seddonm1 6y agoMaybe. In this case Argo uses the Kubernetes distributed state store (etcd or equivalent) and is a stateless service so can deal with failures quite well.