3 ms·
Airflow owning the scheduling, retry, task branching/dependencies is fine. But a few issues: to take advantage of Airflow's core features (XCOMs, task mapping
by annexrichmond 2y ago
Airflow owning the scheduling, retry, task branching/dependencies is fine. But a few issues: to take advantage of Airflow's core features (XCOMs, task mapping, etc) your biz logic/tasks must be in the same venv, which is not sustainable as teams grow. Once you have external tasks, you now have a more complex system to operate and test, and your Airflow installation is now very overkill (you need a pool of workers just to monitor external... workers?).
- notatallshaw 2y agoYou don't need to have your biz tasks in the same venv as your airflow worker to take advantage of task mapping or XCOM. You just need a well defined interface on how to call your biz tasks, and then you can use any of ExternalPythonOperator, DockerOperator, DockerSwarmOperator, KubernetesPodOperator, etc. etc. or write your own to pass in values or data to your task however you want. Airflow is quite complex, and I don't recommend it as people's go to, but IMO that's in large part because it is so unopinionated about how you call and run your tasks and leaves the configuration up to you. But this also means it ends up being a lot of people's choice because they are able to get it to fit what they need.