3 ms·
> Airflow is an ETL(Extract, Transform, Load) workflow orchestration tool, used in data transformation pipelines. Apologies if this is pedantic, but the orches
by slap_shot 6y ago
> Airflow is an ETL(Extract, Transform, Load) workflow orchestration tool, used in data transformation pipelines.
Apologies if this is pedantic, but the orchestration of jobs transcends ETL workflows. There's countless usecases of scheduling dependent jobs that aren't ETL workloads.
- carlosf 6y agoI've been quite happy with the following pattern: - Encapsulate your business logic in microservices and expose ETL actions with APIs. - Call your microservices using Airflow. That way my Airflow jobs are very lightweight (they only call https APIs) and only contain logic of when doing things, not how. All the core business logic for a specific domain lives in a single container that can be tested and deployed independently. Doing ETL using Airflow jobs exclusively or Lambda would spread business logic and make it a nightmare to test and reason about.