4 ms·

It’s not that big of a deal. 1. Collect new data 2. Clean data 3. Annotate 4. Train models and store versions 5. Analyze errors/model metrics (and re-train

It’s not that big of a deal. 1. Collect new data 2. Clean data 3. Annotate 4. Train models and store versions 5. Analyze errors/model metrics (and re-train as need be) 6. Deploy model/s 7. Monitor 8. Repeat steps 1 - 7 Yes, there are many tools that can help with each the above. Use whatever suits to automate it and make your job easier.

modernpink 5y ago
The problem is simple, except that Data Scientists seem to resist good software engineering practices and storm ahead to the model development part which leaves a lot of mess for engineers to pick up. MLOps then looks like a 'difficult' problem because of this mess.
- linspace 5y ago
  That's quite a simplified and antagonistic comment. Data scientist have their own problems too, like crappy data (if any) and unclear business objectives. This usually means fast iteration at all costs. That's why Python or R are the languages of choice, which are not precisely the most amenable to deployment. Now, think about their KPIs vs yours and everything falls into place.
- pyronik19 5y ago
  I view the job of the MLOps framework as essentially a contract where "You place your messy code that does X here, and we'll take care of turning it into a retrainable, resilient, secure API endpoint without any extra fuss". This requires the software engineer/dev ops person to think through ahead of time what the data scientist will need to do as a part of their process and templatize it and form a good abstraction. The problem is that the abstractions are often leaky so there is some constraint that needs to be put on the data scientist. This also requires the MLOps engineer to work on new feature development to form a new abstraction to handle the new features. "Oh you want to do active learning? What does that look like?".
pizza 5y ago
Interesting, hah - I'm of the opinion that even just the first 3 steps you listed, before you actually get to the learning part itself, can each contain a world of hidden assumptions that can get you mired in a cycle of constantly repairing and tweaking pipelines
- discordance 5y ago
  You're totally right. All of the ML hype out there isn't helpful either. The hardest part for me is not the tooling or processes, but ironing out expectations, assumptions, what can and can't be done (or what's worthwhile) with ML. Getting to a common understanding about metrics and outcomes with an end user / customer can chunks of time out of a project.
mellavora 5y ago
Agree that those steps capture the process. But whose job is it to run those steps? And note that most of what I think of as DevOps might have an analogy, but no direct equivalent. Of course this is all solvable; the thing is that it is new. DevOps assumes that changes are to the code, and has processes and tools for managing code changes. In ML, it is the DATA which changes.
- 23pointsNorth 5y ago
  (I'm biased, as I'm the CTO/co-founder of Efemarai) Yes, there are usually several changes that need to be tracked - code/model changes (those usually happen early on and the stabalise), input/code changes (e.g. pre-processing the data with either new transformations or _other_ models), and data changes (both changes for training and testing). At Efemarai we are thinking about it as any changes to the above should automatically trigger a test suit of the model/process. And under test we're thinking not just the different forms of unit testing the input/output formats and sizes from the model, but also unit testing on the model performance of the data you've collected + stress testing the model with data the model is expected to see in production. So in reality, it's indeed nothing new, but the standard DevOps pipeline needs to be extended to work with the ML assumptions.

modernpink 5y ago

The problem is simple, except that Data Scientists seem to resist good software engineering practices and storm ahead to the model development part which leaves a lot of mess for engineers to pick up. MLOps then looks like a 'difficult' problem because of this mess.

linspace 5y ago
That's quite a simplified and antagonistic comment. Data scientist have their own problems too, like crappy data (if any) and unclear business objectives. This usually means fast iteration at all costs. That's why Python or R are the languages of choice, which are not precisely the most amenable to deployment. Now, think about their KPIs vs yours and everything falls into place.
pyronik19 5y ago
I view the job of the MLOps framework as essentially a contract where "You place your messy code that does X here, and we'll take care of turning it into a retrainable, resilient, secure API endpoint without any extra fuss". This requires the software engineer/dev ops person to think through ahead of time what the data scientist will need to do as a part of their process and templatize it and form a good abstraction. The problem is that the abstractions are often leaky so there is some constraint that needs to be put on the data scientist. This also requires the MLOps engineer to work on new feature development to form a new abstraction to handle the new features. "Oh you want to do active learning? What does that look like?".

pizza 5y ago

Interesting, hah - I'm of the opinion that even just the first 3 steps you listed, before you actually get to the learning part itself, can each contain a world of hidden assumptions that can get you mired in a cycle of constantly repairing and tweaking pipelines

discordance 5y ago
You're totally right. All of the ML hype out there isn't helpful either. The hardest part for me is not the tooling or processes, but ironing out expectations, assumptions, what can and can't be done (or what's worthwhile) with ML. Getting to a common understanding about metrics and outcomes with an end user / customer can chunks of time out of a project.

mellavora 5y ago

Agree that those steps capture the process. But whose job is it to run those steps? And note that most of what I think of as DevOps might have an analogy, but no direct equivalent. Of course this is all solvable; the thing is that it is new. DevOps assumes that changes are to the code, and has processes and tools for managing code changes. In ML, it is the DATA which changes.

23pointsNorth 5y ago
(I'm biased, as I'm the CTO/co-founder of Efemarai) Yes, there are usually several changes that need to be tracked - code/model changes (those usually happen early on and the stabalise), input/code changes (e.g. pre-processing the data with either new transformations or _other_ models), and data changes (both changes for training and testing). At Efemarai we are thinking about it as any changes to the above should automatically trigger a test suit of the model/process. And under test we're thinking not just the different forms of unit testing the input/output formats and sizes from the model, but also unit testing on the model performance of the data you've collected + stress testing the model with data the model is expected to see in production. So in reality, it's indeed nothing new, but the standard DevOps pipeline needs to be extended to work with the ML assumptions.