I've been quite happy with the following pattern: - Encapsulate your business logic in microservices and expose ETL actions with APIs. - Call your microservic

I've been quite happy with the following pattern: - Encapsulate your business logic in microservices and expose ETL actions with APIs. - Call your microservices using Airflow. That way my Airflow jobs are very lightweight (they only call https APIs) and only contain logic of when doing things, not how. All the core business logic for a specific domain lives in a single container that can be tested and deployed independently. Doing ETL using Airflow jobs exclusively or Lambda would spread business logic and make it a nightmare to test and reason about.