2 ms·

Also there's a level of automation for model tuning (experiments) especially if this is an update to an existing model (you still need to do model validataion).

Also there's a level of automation for model tuning (experiments) especially if this is an update to an existing model (you still need to do model validataion). For the initial model there's gonna be automation but a lot of time will be spent validating the model with business KPI, not just some model metric like accuracy. Databricks makes this automation and tracking pretty simple. You don't have to do this in notebooks. Once you're done with exploration, build out the pipeline in your "framework" and kick off jobs to execute it on databricks. Yes you can use ETL tools here to, I dunno, train every x hours...whatever

mlthoughts2018 6y ago
Each round trip of model training (such as when optimizing hyperparameters) should be a distinct run through the ETL stages, with dedicated artifact tracking. Observability and monitoring for ML model training ETLs should always include whatever end criteria the business uses to judge the model, such as an ETL step for real acceptance testing, not just domain specific accuracy like F1 score or ROC curves.
- bertomart 6y ago
  I agree, but you're looking at everything through an automated ETL workflow which sorta invoke the idea of using a tool like airflow. But generally you don't need that. What you need is a specific set of functionality (training, tuning, testing) so you can either kick off manually when you wanna do things interactively or hook up into a pipeline when you wanna automate things. And yes you can track everything using MLFlow on databricks. Matter of fact Delta Lake is one of the most powerful features as you track not just migrations but data changes, so you can track end-to-end lineage and so far it's the ONLY platform (that I've tested) that allows this with ease.
  - mlthoughts2018 6y ago
    > “ But generally you don't need that.” I think this is wrong. This is what ML researchers tend to think when they are ignorant of why DevOps best practices exist and what other teams need in order to provide underlying infrastructure support. You’re only thinking of “what you need” from the point of view of the developer experience of the ML engineer, which frankly is usually the least important part by a wide margin.

mlthoughts2018 6y ago

Each round trip of model training (such as when optimizing hyperparameters) should be a distinct run through the ETL stages, with dedicated artifact tracking. Observability and monitoring for ML model training ETLs should always include whatever end criteria the business uses to judge the model, such as an ETL step for real acceptance testing, not just domain specific accuracy like F1 score or ROC curves.

bertomart 6y ago
I agree, but you're looking at everything through an automated ETL workflow which sorta invoke the idea of using a tool like airflow. But generally you don't need that. What you need is a specific set of functionality (training, tuning, testing) so you can either kick off manually when you wanna do things interactively or hook up into a pipeline when you wanna automate things. And yes you can track everything using MLFlow on databricks. Matter of fact Delta Lake is one of the most powerful features as you track not just migrations but data changes, so you can track end-to-end lineage and so far it's the ONLY platform (that I've tested) that allows this with ease.
- mlthoughts2018 6y ago
  > “ But generally you don't need that.” I think this is wrong. This is what ML researchers tend to think when they are ignorant of why DevOps best practices exist and what other teams need in order to provide underlying infrastructure support. You’re only thinking of “what you need” from the point of view of the developer experience of the ML engineer, which frankly is usually the least important part by a wide margin.