30 ms·
In one of my previous companies, they used Airflow(by airbnb) to schedule and manage workflows. Previously they were using nothing but CRON. It turned out to b
by kumaranvpl 8y ago
In one of my previous companies, they used Airflow(by airbnb) to schedule and manage workflows.
Previously they were using nothing but CRON. It turned out to be not so efficient to retry failed workflows and cancel the execution of following dependent jobs.
Airflow turned out to be a great fit for our case. I highly recommend checking it out.
- tedmiston 8y agoWe're using Apache Airflow [2] internally as well. It's pretty featureful and addresses most concerns mentioned by OP, such as orchestration, retries, open source code reuse, and dependency management. It has primitive monitoring and alerting, but one needs to bring something external for that today. Shameless plug - My startup [2] offers Airflow as a SaaS as well as an enterprise distribution with monitoring tools to build upon core Airflow. [1]: https://github.com/apache/incubator-airflow https://github.com/apache/incubator-airflow [2]: https://www.astronomer.io/ https://www.astronomer.io/