Airflow uses Celery to horizontally scale its execution. The Airflow scheduler takes care of what tasks to run in what order, but also what to do when they fail

Airflow uses Celery to horizontally scale its execution. The Airflow scheduler takes care of what tasks to run in what order, but also what to do when they fail, need to retry, don't need to run at all, backfill the past etc. Spark for Airflow is just one of the engines where a transformation of data can happen.