3 ms·
Having to setup and run Airflow on K8s is a hell of a prerequisite step to getting cost-efficient and fast access to GPU training. Airflow is also absolutely n
by thundergolfer 4y ago
Having to setup and run Airflow on K8s is a hell of a prerequisite step to getting cost-efficient and fast access to GPU training.
Airflow is also absolutely not built for that purpose. It's ~10yr old Hadoop-era technology.
- __MatrixMan__ 4y agoAs for getting airflow on k8s in the first place, the apache airflow helm chart pretty much just handles things, doesn't it? It might be a pain to manage many deployments for many teams, but going from 0 to 1 isn't so bad. As for configuring the kubernetes pod operator to ask for pods with GPU's, it exposes the k8s python API in the dag definition. I haven't done it myself, but I think that it's not really airflow that's going to be a pain there. Getting the pod spec right is gonna have to happen whatever does the orchestration. (Full disclosure: my employer offers airflow as a service)