3 ms·
I've been using Airflow for nearly a year in production and I'm surprised by all the positive commentary in this thread about the tool. To be fair, I've been us
by anordin95 6y ago
I've been using Airflow for nearly a year in production and I'm surprised by all the positive commentary in this thread about the tool. To be fair, I've been using the GCP offered version of Airflow - Composer. I've found various components to be flaky and frustrating.
For instance, large scale backfills don't seem to be well supported. I find various components of the system break when trying to large scale backfills, for instance 1M DAG runs.
As another note, the scheduler also seems rather fragile and prone to crashing.
My team has generally found the system good, but not rock-solid nor something to be confident in.
- chrisjc 6y agoThis mirrors my experience with Airflow by another SaaS. Ended up getting rid of it when our Data Warehouse introduced tasks and stored procedures, since most of our work was ELT.
- rywalker 6y agoYes, we need to improve the backfills in Airflow. We're working on making the scheduler HA and more performant, reach out to me if you'd like to collaborate on your use case (ry at astronomer dot io)