4 ms·
I am using Apache Airflow since a couple of years now and the biggest improvement was the addition of the Kubernetes Operator. You basically keep a vanilla Airf
by james_woods 6y ago
I am using Apache Airflow since a couple of years now and the biggest improvement was the addition of the Kubernetes Operator. You basically keep a vanilla Airflow installation and the custom code is encapsulated and tested in containers. This simplifies it a lot.
- recov 6y agoThis is what we do - it's great for decoupling the two. Any heavy work is run in GKE with some custom operators. It also makes on boarding non engineers much easier as they don't have to worry about connections/credentials/etc.