Here is, a slightly outdated, article that compares several ETL workflow tools http://bytepawn.com/luigi-airflow-pinball.html http://bytepawn.com/luigi-airflow-

Here is, a slightly outdated, article that compares several ETL workflow tools http://bytepawn.com/luigi-airflow-pinball.html http://bytepawn.com/luigi-airflow-pinball.html . Why we choose Airflow was because of the following reasons: * Scheduler that knows how to handle retries, skipped tasks, failing tasks * Great UI * Horizontal scaleable * Great community * Extensible; we could make it work in an enterprise context (kerberos, ldap etc) * No XML * Testable and debug-able workflows