I've had the pleasure of working with a world class infra team that managed our Airflow setup on Kubernetes and this made working with it a wonderful experience

I've had the pleasure of working with a world class infra team that managed our Airflow setup on Kubernetes and this made working with it a wonderful experience. At the same time, early in my career, I used Airflow with a team that didn't understand the technology (myself included) and it was a nightmare. I'm now at a tiny company without the infra to support Airflow, so I tested all the managed airflow providers (Composer, MWAA, and Astronomer) and ultimately settled on Astronomer. I also spent some time creating a framework that generates Dags from configuration and now 90% of my new etls just involve creating a config and a sql query. Pretty happy with my current setup and my only complaint is that Astronomer won't let you exec into a Task pod, which is actually kind of a big deal, but still better than having to manage a k8s cluster.