9 ms·

This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there

This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.

samdjstephens 5y ago
Interesting, I wouldn’t say that I’ve found it difficult to run in even a small team. The problem I’ve always had with Airflow has been with non-cron-like use cases, for example data pipelines kicked off when some event occurs. Sensors were often an awkward fit and the HTTP API was quite immature back when I was using it
- the_af 5y ago
  Agreed about sensors. We still have some trouble figuring them out and understanding why they sometimes don't trigger when they should.
ajoseps 5y ago
do you have recommendations on alternatives that are not tied to a cloud provider?
- jonpon 5y ago
  We are trying to build something like this at https://www.magniv.app/ https://www.magniv.app/. Would love to have you join our beta if you are interested!
- liveoneggs 5y ago
  jenkins
- blakeburch 5y ago
  Shipyard, Prefect, Dagster are all good options. Lots of newcomers in the orchestration space.
DevKoala 5y ago
I run Airflow even for my local trading setup. For large teams, I often go with managed solutions like Astronomer.
zukzuk 5y ago
Everyone's context is different, but I've found the exact opposite to be true. Airflow is simple and dumb enough that it can be easily understood and managed by a small team, but it's also flexible and powerful enough that we can't come up with a good enough reason to switch to anything else.* *We are, however, becoming more and more reliant on dbt, and the article makes a good point about Airflow providing no visibility for what's going on in a dbt node. So we're ending up with an increasingly simpler Airflow dag, with most of the complexity hidden inside a single dbt node.
- contravariant 5y ago
  I'm not too familiar with the use of dbt but what was the reason you chose to have a single dbt node rather than translating the dependencies into an airflow dag?
- hulahoof 5y ago
  This reflects how I often deploy Airflow as well (usually on GCP as Composer) We use DBT to manage the DAG for the BQ transformations, put this in a container and deploy it into the kubernetes cluster that airflow is running on as a single node. Airflow can then handle the scheduling and DAG nodes for non DWH dependencies such as loading/checking for files, kicking off tasks that need to run after the DWH refresh and the like. I find once it is set up it is extremely easy for small teams to follow the pattern, and the single view of all the pipelines running is a great benefit - as well as handling the logic around last successful runs etc., that would need to be implemented manually if using simple cron jobs.
gorkemyurt 5y ago
Hi, I am the author of the post which parts did you find hard to follow?
rr808 5y ago
There really aren't many alternatives out there after cron. Maybe lambda jobs count? What are you thinking of as alternatives?
superyesh 5y ago
I understand it is subjective. But I use a forked version of https://github.com/puckel/docker-airflow https://github.com/puckel/docker-airflow on our managed K8s cluster and it points to a cloud managed Postgres. It has worked pretty well for over 3 years with no-one actually managing it from an infra POV. YMMV. This is driving a product whose ARR is well in the 100s of Millions. If you have simple needs that are more or less set, I agree Airflow is overkill and a simple Jenkins instance is all you need.
shmoogy 5y ago
I manage and run our airflow instance - outside of migrating from 1.X to 2.x I haven't really had any problems. Learning curve was a bit higher than I hoped, but being able to set tasks downstream and backfill is so much nicer than regular cron / windows task manager script running.
otabdeveloper4 5y ago
> there are far better tools out there for scheduling Really? Which ones? The only thing vaguely fitting this case is Jenkins, but using Jenkins to run ETL/ELT is a serious impedance mismatch.
- urthor 5y ago
  Dagster/Prefect are the alternatives. But yes, I'm confused. Triggering a dag and having it exit based on complex logic is a perfectly normal pattern.

samdjstephens 5y ago

Interesting, I wouldn’t say that I’ve found it difficult to run in even a small team. The problem I’ve always had with Airflow has been with non-cron-like use cases, for example data pipelines kicked off when some event occurs. Sensors were often an awkward fit and the HTTP API was quite immature back when I was using it

the_af 5y ago
Agreed about sensors. We still have some trouble figuring them out and understanding why they sometimes don't trigger when they should.

ajoseps 5y ago

do you have recommendations on alternatives that are not tied to a cloud provider?

jonpon 5y ago
We are trying to build something like this at https://www.magniv.app/ https://www.magniv.app/. Would love to have you join our beta if you are interested!
liveoneggs 5y ago
jenkins
blakeburch 5y ago
Shipyard, Prefect, Dagster are all good options. Lots of newcomers in the orchestration space.

DevKoala 5y ago

I run Airflow even for my local trading setup. For large teams, I often go with managed solutions like Astronomer.

zukzuk 5y ago

Everyone's context is different, but I've found the exact opposite to be true. Airflow is simple and dumb enough that it can be easily understood and managed by a small team, but it's also flexible and powerful enough that we can't come up with a good enough reason to switch to anything else.* *We are, however, becoming more and more reliant on dbt, and the article makes a good point about Airflow providing no visibility for what's going on in a dbt node. So we're ending up with an increasingly simpler Airflow dag, with most of the complexity hidden inside a single dbt node.

contravariant 5y ago
I'm not too familiar with the use of dbt but what was the reason you chose to have a single dbt node rather than translating the dependencies into an airflow dag?
hulahoof 5y ago
This reflects how I often deploy Airflow as well (usually on GCP as Composer) We use DBT to manage the DAG for the BQ transformations, put this in a container and deploy it into the kubernetes cluster that airflow is running on as a single node. Airflow can then handle the scheduling and DAG nodes for non DWH dependencies such as loading/checking for files, kicking off tasks that need to run after the DWH refresh and the like. I find once it is set up it is extremely easy for small teams to follow the pattern, and the single view of all the pipelines running is a great benefit - as well as handling the logic around last successful runs etc., that would need to be implemented manually if using simple cron jobs.

gorkemyurt 5y ago

Hi, I am the author of the post which parts did you find hard to follow?

rr808 5y ago

There really aren't many alternatives out there after cron. Maybe lambda jobs count? What are you thinking of as alternatives?

superyesh 5y ago

I understand it is subjective. But I use a forked version of https://github.com/puckel/docker-airflow https://github.com/puckel/docker-airflow on our managed K8s cluster and it points to a cloud managed Postgres. It has worked pretty well for over 3 years with no-one actually managing it from an infra POV. YMMV. This is driving a product whose ARR is well in the 100s of Millions. If you have simple needs that are more or less set, I agree Airflow is overkill and a simple Jenkins instance is all you need.

shmoogy 5y ago

I manage and run our airflow instance - outside of migrating from 1.X to 2.x I haven't really had any problems. Learning curve was a bit higher than I hoped, but being able to set tasks downstream and backfill is so much nicer than regular cron / windows task manager script running.

otabdeveloper4 5y ago

> there are far better tools out there for scheduling Really? Which ones? The only thing vaguely fitting this case is Jenkins, but using Jenkins to run ETL/ELT is a serious impedance mismatch.

urthor 5y ago
Dagster/Prefect are the alternatives. But yes, I'm confused. Triggering a dag and having it exit based on complex logic is a perfectly normal pattern.