3 ms·
> I can't imagine why people use Airflow these days. You lose out on a lot of other Airflow functionality because it all assumes you have your business logic e
by braza 2y ago
> I can't imagine why people use Airflow these days.
You lose out on a lot of other Airflow functionality because it all assumes you have your business logic embedded within DAGs.
As an old Airflow user, I can relate with that.
The contributors are great and the speed of the new features and fixes are great also.
I see two main issues with Airflow: (i) the lack of good interoperability with a cloud native stack, specifically with the k8s operators, that sounds quite hackish and (ii) lack of a better version control in the DAGs in way that we can have actually real lineage.
The point is that for most of the veterans in ~ETL~ Data Engineering in Cloud Native stacks, we just want to send a cli command or a Make command to a container to execute the thing that we want.
Apache Airflow does that, but as most of the things in the Python ecosystem at the time that you have a great UI (not confuse with UX) become way easier to embed business logic in the DAGs and have your application embedded with Airflow logic.
I will never bash Airflow since I got a lot of money in consulting migrating from and to it, but the general feeling of working on a daily basis for almost 7 years made me realize the sad reality around orquestrators. In the past, those used to be kind of the bedrock technology that you would never want to migrate, but today, due to this feature bloat and lack of vision, I started to see orquestrators as disposable as Java Script frameworks.