I agree, but you're looking at everything through an automated ETL workflow which sorta invoke the idea of using a tool like airflow. But generally you don't ne

I agree, but you're looking at everything through an automated ETL workflow which sorta invoke the idea of using a tool like airflow. But generally you don't need that. What you need is a specific set of functionality (training, tuning, testing) so you can either kick off manually when you wanna do things interactively or hook up into a pipeline when you wanna automate things. And yes you can track everything using MLFlow on databricks. Matter of fact Delta Lake is one of the most powerful features as you track not just migrations but data changes, so you can track end-to-end lineage and so far it's the ONLY platform (that I've tested) that allows this with ease.