4 ms·
True, but what if the "grow complex" step doesn't happen? YAGNI. If you are in a domain that needs complex work-flows, I can see selecting a work-flow framework
by tabtab 8y ago
True, but what if the "grow complex" step doesn't happen? YAGNI. If you are in a domain that needs complex work-flows, I can see selecting a work-flow framework. But I've seen some really ugly frameworks where one put in Cadillacs for every part when Chevys would do just fine 98% of the time. Staff have to learn, understand, maintain, and tune complex frameworks.
- tedmiston 8y agoI understand the concern. It's kind of like using a web framework — you probably wouldn't write your own form processing and ORM code to avoid injection attacks when it's already been built. Airflow has analogous features and protections for tasks/workflows. Most companies adopting Airflow already have workflow requirements like this, even if it's just a single transform or moving data from one system to another. Even if you have just a two-step workflow with Task B dependent upon success of Task A, Airflow offers protection, historical stats, email alerting, etc over trying to schedule successive cron jobs with built-in assumptions, hacking together a dependency system, etc. To me, Airflow is the Honda of this domain. Overall it's a relatively small and simple framework from the DAG author's perspective.