3 ms·
It is one of the few workflow managers that support cyclic graphs (e.g. ecflow, cylc, prefect, covalent more or less. ), instead of working only with DAGs like
by kinow 3y ago
It is one of the few workflow managers that support cyclic graphs (e.g. ecflow, cylc, prefect, covalent more or less.
), instead of working only with DAGs like Airflow.
It is a feature common for weather and climate workflows, and I think other HPC and machine learning workflows too.
https://kinoshita.eti.br/2021/10/01/cyclic-workflows-with-cylc-and-stackstorm.html https://kinoshita.eti.br/2021/10/01/cyclic-workflows-with-cy...
- __MatrixMan__ 3y agoFYI if you need a cycle in Airflow you can stitch DAGs together with datasets. The simplest would be to just have a DAG update the same dataset that it consumes. Whether it actually corresponds with a dataset out there is immaterial, it can just be an event trigger.