4 ms·

I actually posted on a somewhat related topic here: - https://managingml.substack.com/p/ml-model-training-is-an-etl https://managingml.substack.com/p/ml-model-

by mlthoughts2018 6y ago

I actually posted on a somewhat related topic here: - https://managingml.substack.com/p/ml-model-training-is-an-etl https://managingml.substack.com/p/ml-model-training-is-an-et... As an ML manager one of the things that I dislike about the Databricks model is the reliance on notebooks and bespoke interactive cluster workflows. ML workloads are stuck in the stone age because there’s no common pattern to plug them into existing ETL frameworks, but really that is what they are. Model training is an ETL that just happens to need unique domain tools to visualize how it’s succeeding / track observability metrics. But apart from that, you really need to automate model training and experimentation. It is a total antipattern to ever use notebooks, or even non-notebook code developed interactively. Rather you should be submitting tasks to a task queue that maps your ML exploratory workload or training workload to a scheduled compute resource, runs it with observability baked in, and treats outputs like schema’d outputs of more traditional ETLs. Instead of that, Databricks is much more of an early 2000s MATLAB model. Make it addictive and easy for unpragmatic researchers detached from real production use cases, then figure employers will have to adopt it since all the expensive-to-hire researchers can only be productive with it. Long term I think it’s a very bad gamble. Just consider how much open source Python tools have eaten MATLAB’s lunch in the past 10 years.

Jugurtha 6y ago
This is the direction we're going towards with our internal platform[0] that we built to make our life easier. We do have real-time collaborative notebooks. However, we also have long-running notebook scheduling where notebooks get picked by a runners and executed against certain environments. One cool feature is that you can watch the notebook's output as it runs, even if you had closed the browser or logged in from another computer. Anothr cool feature is that we do automatic experiment tracking, so we detect parameters, models, metrics, and the notebook automatically, removing the cognitive load from the use to "remember" doing that. People rarely remember clerical things, but even when they do, we didn't want to pollute the notebooks with tracking code of say MLflow. This happens automatically. These long-running notebooks are simply jobs. There are other types of jobs: building a Docker image for the model so that you can push it to a registry is also a job, so is deploying a model to get a "REST" endpoint and a live dashboard to monitor model performance. One of our guidelines is: "Anything that is possible with point and click must be possible with an API call", so we make sure to make everything programmable. For example, scheduling a long-running notebook can be done sending in a request. We also are working to add in webhooks and use cloudevents spec so that people can take information from our system, such as a job having finished, and trigger things in other systems.. Another is workload portability. Again, it's still early and we're prioritizing the issues that have frustrated us the most for the past years in our projects. There's a long way to go, especially that there are many such as yourself who may have worked on problems we've never seen, or a scale/data we haven't handled before. - [0]: https://iko.ai https://iko.ai
bertomart 6y ago
Can't disagree with you more here. Databricks is a tool. It's on you to create your frameworks or whatever to optimize your workflow. I've been using Databricks and it's such a great platform, at least compared to what I've used in the past (EMR/Sagemaker). Simply build a framework where you offload the notebooks and exploratory stuff to databricks and build out productionable pipelines locally using sample data from dbricks, then test on databricks. That's what we've been doing and it's working very nicely. Furthermore you can remotely kick off jobs on databricks, so once you have your pipeline worked out you really don't need to interact with notebooks if you don't want...just use it as a scalable backend, dashboard, model tracker, whatever fits in your workflow. I feel like instead of using the tool a lot of folks have specific expectations and if the tool doesn't fit exactly into how they work then it's just written off.
- mlthoughts2018 6y ago
  The trouble is that those expectations exist for a reason, and there’s a whole world of best practices and efficient patterns that have developed specifically to solve the challenges of orchestrating and monitoring ETLs. What feels convenient for researchers using a notebook is just fool’s gold (for known, legit reasons), and it damages credibility with other domains of engineering when their widely vetted best practices are ignored just for the sake of convenience tools like notebooks or manual job kickoff interfaces through a vendor tool like Databricks.
  - bertomart 6y ago
    They (Databricks) are not advertising themselves as a best practices way of doing anything, they're just a platform for doing data science and analytics. It's up to the data scientist to use the tool properly and to have proper methodologies. A platform is simply a set of tools that helps a target audience, and that's what databricks is. Now the issue here is that a lot of datascience either cannot code/do engineering or just don't want to. But in my view people like that are just glorified analysts. Data Science is built on the foundation of software engineering (+ stats, viz, math, etc...); this is why it's so complicated. If you cannot code or don't understand the SDLC or best practices you're basically a carpenter who can't hammer or saw.
bertomart 6y ago
Also there's a level of automation for model tuning (experiments) especially if this is an update to an existing model (you still need to do model validataion). For the initial model there's gonna be automation but a lot of time will be spent validating the model with business KPI, not just some model metric like accuracy. Databricks makes this automation and tracking pretty simple. You don't have to do this in notebooks. Once you're done with exploration, build out the pipeline in your "framework" and kick off jobs to execute it on databricks. Yes you can use ETL tools here to, I dunno, train every x hours...whatever
- mlthoughts2018 6y ago
  Each round trip of model training (such as when optimizing hyperparameters) should be a distinct run through the ETL stages, with dedicated artifact tracking. Observability and monitoring for ML model training ETLs should always include whatever end criteria the business uses to judge the model, such as an ETL step for real acceptance testing, not just domain specific accuracy like F1 score or ROC curves.
  - bertomart 6y ago
    I agree, but you're looking at everything through an automated ETL workflow which sorta invoke the idea of using a tool like airflow. But generally you don't need that. What you need is a specific set of functionality (training, tuning, testing) so you can either kick off manually when you wanna do things interactively or hook up into a pipeline when you wanna automate things. And yes you can track everything using MLFlow on databricks. Matter of fact Delta Lake is one of the most powerful features as you track not just migrations but data changes, so you can track end-to-end lineage and so far it's the ONLY platform (that I've tested) that allows this with ease.
    - mlthoughts2018 6y ago
      > “ But generally you don't need that.” I think this is wrong. This is what ML researchers tend to think when they are ignorant of why DevOps best practices exist and what other teams need in order to provide underlying infrastructure support. You’re only thinking of “what you need” from the point of view of the developer experience of the ML engineer, which frankly is usually the least important part by a wide margin.

Jugurtha 6y ago

This is the direction we're going towards with our internal platform[0] that we built to make our life easier. We do have real-time collaborative notebooks. However, we also have long-running notebook scheduling where notebooks get picked by a runners and executed against certain environments. One cool feature is that you can watch the notebook's output as it runs, even if you had closed the browser or logged in from another computer. Anothr cool feature is that we do automatic experiment tracking, so we detect parameters, models, metrics, and the notebook automatically, removing the cognitive load from the use to "remember" doing that. People rarely remember clerical things, but even when they do, we didn't want to pollute the notebooks with tracking code of say MLflow. This happens automatically. These long-running notebooks are simply jobs. There are other types of jobs: building a Docker image for the model so that you can push it to a registry is also a job, so is deploying a model to get a "REST" endpoint and a live dashboard to monitor model performance. One of our guidelines is: "Anything that is possible with point and click must be possible with an API call", so we make sure to make everything programmable. For example, scheduling a long-running notebook can be done sending in a request. We also are working to add in webhooks and use cloudevents spec so that people can take information from our system, such as a job having finished, and trigger things in other systems.. Another is workload portability. Again, it's still early and we're prioritizing the issues that have frustrated us the most for the past years in our projects. There's a long way to go, especially that there are many such as yourself who may have worked on problems we've never seen, or a scale/data we haven't handled before. - [0]: https://iko.ai https://iko.ai

bertomart 6y ago

Can't disagree with you more here. Databricks is a tool. It's on you to create your frameworks or whatever to optimize your workflow. I've been using Databricks and it's such a great platform, at least compared to what I've used in the past (EMR/Sagemaker). Simply build a framework where you offload the notebooks and exploratory stuff to databricks and build out productionable pipelines locally using sample data from dbricks, then test on databricks. That's what we've been doing and it's working very nicely. Furthermore you can remotely kick off jobs on databricks, so once you have your pipeline worked out you really don't need to interact with notebooks if you don't want...just use it as a scalable backend, dashboard, model tracker, whatever fits in your workflow. I feel like instead of using the tool a lot of folks have specific expectations and if the tool doesn't fit exactly into how they work then it's just written off.

mlthoughts2018 6y ago
The trouble is that those expectations exist for a reason, and there’s a whole world of best practices and efficient patterns that have developed specifically to solve the challenges of orchestrating and monitoring ETLs. What feels convenient for researchers using a notebook is just fool’s gold (for known, legit reasons), and it damages credibility with other domains of engineering when their widely vetted best practices are ignored just for the sake of convenience tools like notebooks or manual job kickoff interfaces through a vendor tool like Databricks.
- bertomart 6y ago
  They (Databricks) are not advertising themselves as a best practices way of doing anything, they're just a platform for doing data science and analytics. It's up to the data scientist to use the tool properly and to have proper methodologies. A platform is simply a set of tools that helps a target audience, and that's what databricks is. Now the issue here is that a lot of datascience either cannot code/do engineering or just don't want to. But in my view people like that are just glorified analysts. Data Science is built on the foundation of software engineering (+ stats, viz, math, etc...); this is why it's so complicated. If you cannot code or don't understand the SDLC or best practices you're basically a carpenter who can't hammer or saw.

bertomart 6y ago

Also there's a level of automation for model tuning (experiments) especially if this is an update to an existing model (you still need to do model validataion). For the initial model there's gonna be automation but a lot of time will be spent validating the model with business KPI, not just some model metric like accuracy. Databricks makes this automation and tracking pretty simple. You don't have to do this in notebooks. Once you're done with exploration, build out the pipeline in your "framework" and kick off jobs to execute it on databricks. Yes you can use ETL tools here to, I dunno, train every x hours...whatever

mlthoughts2018 6y ago
Each round trip of model training (such as when optimizing hyperparameters) should be a distinct run through the ETL stages, with dedicated artifact tracking. Observability and monitoring for ML model training ETLs should always include whatever end criteria the business uses to judge the model, such as an ETL step for real acceptance testing, not just domain specific accuracy like F1 score or ROC curves.
- bertomart 6y ago
  I agree, but you're looking at everything through an automated ETL workflow which sorta invoke the idea of using a tool like airflow. But generally you don't need that. What you need is a specific set of functionality (training, tuning, testing) so you can either kick off manually when you wanna do things interactively or hook up into a pipeline when you wanna automate things. And yes you can track everything using MLFlow on databricks. Matter of fact Delta Lake is one of the most powerful features as you track not just migrations but data changes, so you can track end-to-end lineage and so far it's the ONLY platform (that I've tested) that allows this with ease.
  - mlthoughts2018 6y ago
    > “ But generally you don't need that.” I think this is wrong. This is what ML researchers tend to think when they are ignorant of why DevOps best practices exist and what other teams need in order to provide underlying infrastructure support. You’re only thinking of “what you need” from the point of view of the developer experience of the ML engineer, which frankly is usually the least important part by a wide margin.