5 ms·
> I expect a framework to fit nicely with existing biz logic. I wouldn't, that's just describing a custom codebase. Airflow comes with bells and whistles that
by politelemon 2y ago
> I expect a framework to fit nicely with existing biz logic.
I wouldn't, that's just describing a custom codebase. Airflow comes with bells and whistles that can be taken advantage of if you fit your execution in their DAG model, that's all. Those bells and whistles can be an excellent pattern to work with. You can go all in on operators, or you can just have an operator call out to an external task that does all the work and only rely on Airflow for its retry/alerting mechanism while keeping the external task in language of your choice.
- annexrichmond 2y agoAirflow owning the scheduling, retry, task branching/dependencies is fine. But a few issues: to take advantage of Airflow's core features (XCOMs, task mapping, etc) your biz logic/tasks must be in the same venv, which is not sustainable as teams grow. Once you have external tasks, you now have a more complex system to operate and test, and your Airflow installation is now very overkill (you need a pool of workers just to monitor external... workers?).
- notatallshaw 2y agoYou don't need to have your biz tasks in the same venv as your airflow worker to take advantage of task mapping or XCOM. You just need a well defined interface on how to call your biz tasks, and then you can use any of ExternalPythonOperator, DockerOperator, DockerSwarmOperator, KubernetesPodOperator, etc. etc. or write your own to pass in values or data to your task however you want. Airflow is quite complex, and I don't recommend it as people's go to, but IMO that's in large part because it is so unopinionated about how you call and run your tasks and leaves the configuration up to you. But this also means it ends up being a lot of people's choice because they are able to get it to fit what they need.