4 ms·

That might be fine for small teams, but any 100+ person company would already have their own abstractions and don't need to reinvent those wheels just for Airfl

That might be fine for small teams, but any 100+ person company would already have their own abstractions and don't need to reinvent those wheels just for Airflow. But even if you're smaller I still think you're setting up for failure if all your biz logic is within Airflow unless you are careful about making it reusable/shareable for other contexts. Eg, what if you want to convert some scheduled pipeline to some event-driven architecture with other systems? That means needing to refactor everything out. It's not interoperable or modular and that's why it should be avoided.

nooorofe 2y ago
> if you want to convert some scheduled pipeline to some event-driven architecture Airflow has sensors and triggers. https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html https://airflow.apache.org/docs/apache-airflow/stable/author... But in the core it is built around data pipeline concept, event driven pipeline will much more fragile. Airflow intentionally doesn't manage business logic, it works with "tasks".
- annexrichmond 2y ago
  Yes, but that means you are forced to build EDA on top of Airflow, which may not be ideal for many cases. You are stuck managing your pools/workers within Airflow's paradigm, which means all workload must (a) be written in Python and (b) have Airflow installed on the venv (very heavy pkg) and (c) be k8s pod or Celery (unless you write your own).
  - nyrikki 2y ago
    Only because you have chosen to introduce configuration and maintenance complexity by using airflow as enterprise wide middleware. In a modern even based SOA, products like airflow are a sometimes food while pub/sub is the default. Perhaps a search for images of the zachman framework would help conceptualize how you are tightly coupling to the implementation. But also research SOA 2.0, or event based SOA, the Enterprise Service Bus concept of the original SOA is as dead as COBRA. ETA: the minimal package load for airflow isn't bad, are you installing all of the plugins and their dependencies?
    - annexrichmond 2y ago
      We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless. There are a number of blog posts echoing a similar critique[1]. Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages. According to their README: https://github.com/apache/airflow#approach-to-dependencies-of-airflow https://github.com/apache/airflow#approach-to-dependencies-o... > Airflow has a lot of dependencies - direct and transitive > The important dependencies are: SQLAlchemy, Alembic, Flask, werkzeug, celery, kubernetes Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server? [1] Anecdotes from various posts - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl... - https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-comparing-flyte-and-airflow-72c40d143aad https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-co... - https://dagster.io/blog/dagster-airflow https://dagster.io/blog/dagster-airflow > Airflow, in its design, made the incorrect abstraction by having Operators actually implement functional work instead of spinning up developer work. > By simply moving to using a Kubernetes Operator, Airflow developers can develop more quickly, debug more confidently, and not worry about conflicting package requirements. > Airflow lacks proper library isolation. It becomes hard or impossible to do if any team requires a specific library version for a given workflow > There is no way to separate DAGs to development, staging, and production using out-of-the-box Airflow features. That makes Airflow harder to use for mission-critical applications that require proper testing and the ability to roll back > Data pipelines written for Airflow are typically bound to a particular environment. To avoid dependency hell, most guides recommend defining Airflow tasks with operators like the KubernetesPodOperator, which dictates that the task gets executed in Kubernetes. When a DAG is written in this way, it’s nigh-impossible to run it locally or as part of CI. And it requires opting out of all of the integrations that come out-of-the-box with Airflow.
adammarples 2y ago
If you need to convert a scheduled pipeline into some event driven architecture then yes, it will need a rewrite. Is there any case in which this wouldn't be true? What does it mean to be "interoperable"? Airflow drags can be triggered by events or they can trigger events if needs be. I admit it is not designed to do event streaming though.
nyrikki 2y ago
It sounds like you are tightly coupling to implementation and also expecting this to be the rightfully maligned ESB. Airflow, aws step functions, etc are complex event processors and shouldn't typically be used as wide as you are suggesting. It is a common pitfall to accidentally build distributed monoliths, and in fact often requires active architectural governance to avoid it. Airflow isn't perfect, but I highly recommend considering a review on why modernization efforts fail. The what and the how shouldn't be tightly coupled, especially across capabilities. But yes. Airflow makes a poor ESB/DTC, but that is not the target for the project.

nooorofe 2y ago

> if you want to convert some scheduled pipeline to some event-driven architecture Airflow has sensors and triggers. https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html https://airflow.apache.org/docs/apache-airflow/stable/author... But in the core it is built around data pipeline concept, event driven pipeline will much more fragile. Airflow intentionally doesn't manage business logic, it works with "tasks".

annexrichmond 2y ago
Yes, but that means you are forced to build EDA on top of Airflow, which may not be ideal for many cases. You are stuck managing your pools/workers within Airflow's paradigm, which means all workload must (a) be written in Python and (b) have Airflow installed on the venv (very heavy pkg) and (c) be k8s pod or Celery (unless you write your own).
- nyrikki 2y ago
  Only because you have chosen to introduce configuration and maintenance complexity by using airflow as enterprise wide middleware. In a modern even based SOA, products like airflow are a sometimes food while pub/sub is the default. Perhaps a search for images of the zachman framework would help conceptualize how you are tightly coupling to the implementation. But also research SOA 2.0, or event based SOA, the Enterprise Service Bus concept of the original SOA is as dead as COBRA. ETA: the minimal package load for airflow isn't bad, are you installing all of the plugins and their dependencies?
  - annexrichmond 2y ago
    We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless. There are a number of blog posts echoing a similar critique[1]. Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it easier to support multiple languages. According to their README: https://github.com/apache/airflow#approach-to-dependencies-of-airflow https://github.com/apache/airflow#approach-to-dependencies-o... > Airflow has a lot of dependencies - direct and transitive > The important dependencies are: SQLAlchemy, Alembic, Flask, werkzeug, celery, kubernetes Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server? [1] Anecdotes from various posts - https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl... - https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-comparing-flyte-and-airflow-72c40d143aad https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-co... - https://dagster.io/blog/dagster-airflow https://dagster.io/blog/dagster-airflow > Airflow, in its design, made the incorrect abstraction by having Operators actually implement functional work instead of spinning up developer work. > By simply moving to using a Kubernetes Operator, Airflow developers can develop more quickly, debug more confidently, and not worry about conflicting package requirements. > Airflow lacks proper library isolation. It becomes hard or impossible to do if any team requires a specific library version for a given workflow > There is no way to separate DAGs to development, staging, and production using out-of-the-box Airflow features. That makes Airflow harder to use for mission-critical applications that require proper testing and the ability to roll back > Data pipelines written for Airflow are typically bound to a particular environment. To avoid dependency hell, most guides recommend defining Airflow tasks with operators like the KubernetesPodOperator, which dictates that the task gets executed in Kubernetes. When a DAG is written in this way, it’s nigh-impossible to run it locally or as part of CI. And it requires opting out of all of the integrations that come out-of-the-box with Airflow.

adammarples 2y ago

If you need to convert a scheduled pipeline into some event driven architecture then yes, it will need a rewrite. Is there any case in which this wouldn't be true? What does it mean to be "interoperable"? Airflow drags can be triggered by events or they can trigger events if needs be. I admit it is not designed to do event streaming though.

nyrikki 2y ago

It sounds like you are tightly coupling to implementation and also expecting this to be the rightfully maligned ESB. Airflow, aws step functions, etc are complex event processors and shouldn't typically be used as wide as you are suggesting. It is a common pitfall to accidentally build distributed monoliths, and in fact often requires active architectural governance to avoid it. Airflow isn't perfect, but I highly recommend considering a review on why modernization efforts fail. The what and the how shouldn't be tightly coupled, especially across capabilities. But yes. Airflow makes a poor ESB/DTC, but that is not the target for the project.