4 ms·
Airflow is only a DAG task executor, which hardly scratches the surface of what is needed for managing experiment tracking, telemetry for model training, and ad
by mlthoughts2018 6y ago
Airflow is only a DAG task executor, which hardly scratches the surface of what is needed for managing experiment tracking, telemetry for model training, and ad hoc vs scheduled ML workloads.
Airflow is useful as a component of an ML platform, but it is only in principle capable of addressing a really really tiny part of the requirements.
You also need to ensure Airflow can easily provision the required execution environment (eg. distributed training, multi-gpu training, heavily custom runtime environments).
Overall Airflow isn’t a big part of ML workflows, just a small side tool for a small subset of cases.