3 ms·
Lambdas have resource constraints that Airflow DAGs don't. Most notably, Airflow DAGs can run for any arbitrary length of time. And the local storage attached
by mywittyname 4y ago
Lambdas have resource constraints that Airflow DAGs don't. Most notably, Airflow DAGs can run for any arbitrary length of time. And the local storage attached to the Airflow cluster is actual disk space, and not just a fake in-memory disk, making it possible process files larger than the amount of memory allocated to the DAG.
There's certainly some functionality overlap, but I don't see Lambda and Airflow as competitors. Each has capabilities that the other doesn't.
- xtracto 4y agoI remember reading that you can attach EFS to lambdas. That would solve some of the storage issues.