4 ms·

Well written article. One question I always have when reading such an article. Is it really worth it for these kinds of companies to run Airflow on Kubernetes.

Well written article. One question I always have when reading such an article. Is it really worth it for these kinds of companies to run Airflow on Kubernetes. You could also run it for example on AWS Batch with Spot instances.

beckingz 4y ago
Running Airflow on Kubernetes has been one of the most painful data engineering challenges I've worked on.
- marcinzm 4y ago
  How so? Did you have any existing Kubernetes knowledge? We found it fairly easy to deploy using the community Helm chart (official chart wasn't out yet).
- ricklamers 4y ago
  We kept hearing this from our users. We’ve just released our k8s operator based deployment of Orchest that should give you a good experience running an orchestration tool on k8s without much trouble. https://github.com/orchest/orchest https://github.com/orchest/orchest (We extended Argo, works fantastically well by the way!)
- mrbungie 4y ago
  Did you have any previous experience running workloads in k8s before? Running the Airflow Helm is pretty straightforward, even with more "complex" use cases like heterogenous pods for different task sizes.
  - beckingz 4y ago
    One of our engineers had to make 2 contributions to the Bitnami Helm chart for airflow for some reason.