3 ms·

Nice article, however, I disagree on a couple of things : - "Plateform MLE" is juste regular Devops and Software Engineering. It's not because we're dealing wi

by Fiahil 4y ago

Nice article, however, I disagree on a couple of things : - "Plateform MLE" is juste regular Devops and Software Engineering. It's not because we're dealing with models and their accuracies that it is fundamentally different from what uses to be Devops before. We don't need to make a "special" title out of everything in Data Science. - I still like to introduce "MLOps" into the conversation, thus making it special, and infringing the rule above. Oups. - over fitting slightly on recent data, without accounting for gaps in modeling, is likely to lead to dire situations. See also : March 2020, when all forecasts went bananas. And , no, retraining at that time did not improved the situation. That's what the MLE was talking about here : > “I know it’s not really addressing the data drift problem,” ; they were right. - Everything about data and model drift is just the tip of the iceberg: What happen when your model ends up in production ? It will start affecting the behaviour of the very thing you're trying to predict. A prime example of this on the retail markdown case : did you sold more of that article because the product was rated better (as qualified by the marketer to compensate for lack of control over markdowns), and the in-store stock was higher, or did your markdown on that item had the effect you were looking for and the rating was actually okay ? Did the sales went down this markdown season because your model had a perfect but unattractive markdown strategy last season ? This is already very difficult to model properly, let alone trying to measure their contribution to the drift. Good luck with that.

usgroup 4y ago
> What happen when your model ends up in production ? It will start affecting the behaviour of the very thing you're trying to predict Indeed, again part of the reasons I think MLOps is a problem. One ideally needs a causal theory and rational expectations regarding how and why a model will work: you should be able to explain it. The reason is that there are many more ways for things to look like they work, then ways for them to actually work. Forcing -- what should be -- the scientific effort of modelling into a cut box, tune, re-train, monitor workflow is reductionist and will invariably make it near impossible to exercise proper due diligence IMO. Scientific discovery is generally not a sausage machine -- its messy and dependent on what happens in brains more than anything else. I think its a fools errand to expect differently from -- what should be -- the same abstract tool box applied to commerce.
- Fiahil 4y ago
  So. I understand what point your making, but I completely disagree with you. MLOps is as much as an issue to Data Science than DevOps is to Software Engineering. Automating things doesn't make them stupid or constrained ; I don't know why you would think this creates a "cut box" workflow and sabotage the beautiful Scientific process you imagine is happening. Maybe you have an issue with some Data Scientists you are working with ? To your other point below : > The DS cuts a black box which MLOps tunes, monitors and re-trains. MLOps is not about adapting or tuning a model. Not even close, it's just completely out of scope. MLOps is just a set of best practices to ensure you can compare and reproduce previous outputs (a model, plot, shap output, png, or json benchmark report, whatever you want) at any given point in time. What you do with that model, and how/when you want to train it, is your business alone. We need to call these best practices "MLOps" because it's a significant enough break from regular software engineering practices : suddenly pinning a git tag is not enough to ensure reproducibility, now, we also need to pin datasets to their specific state in time. And what about outputs that may improve over as a result ? Same. Calling them "ML" Ops is actually a bit disingenuous. Optimisation cases (with constraint or MIP solvers) that have nothing to do with ML and models also benefit from the same framework.
zach_garwood 4y ago
Completely irrelevant: you spelled "oops" as "oups" and now I'm imagining thats the French spelling.

by Fiahil 4y ago

usgroup 4y ago

> What happen when your model ends up in production ? It will start affecting the behaviour of the very thing you're trying to predict Indeed, again part of the reasons I think MLOps is a problem. One ideally needs a causal theory and rational expectations regarding how and why a model will work: you should be able to explain it. The reason is that there are many more ways for things to look like they work, then ways for them to actually work. Forcing -- what should be -- the scientific effort of modelling into a cut box, tune, re-train, monitor workflow is reductionist and will invariably make it near impossible to exercise proper due diligence IMO. Scientific discovery is generally not a sausage machine -- its messy and dependent on what happens in brains more than anything else. I think its a fools errand to expect differently from -- what should be -- the same abstract tool box applied to commerce.

Fiahil 4y ago
So. I understand what point your making, but I completely disagree with you. MLOps is as much as an issue to Data Science than DevOps is to Software Engineering. Automating things doesn't make them stupid or constrained ; I don't know why you would think this creates a "cut box" workflow and sabotage the beautiful Scientific process you imagine is happening. Maybe you have an issue with some Data Scientists you are working with ? To your other point below : > The DS cuts a black box which MLOps tunes, monitors and re-trains. MLOps is not about adapting or tuning a model. Not even close, it's just completely out of scope. MLOps is just a set of best practices to ensure you can compare and reproduce previous outputs (a model, plot, shap output, png, or json benchmark report, whatever you want) at any given point in time. What you do with that model, and how/when you want to train it, is your business alone. We need to call these best practices "MLOps" because it's a significant enough break from regular software engineering practices : suddenly pinning a git tag is not enough to ensure reproducibility, now, we also need to pin datasets to their specific state in time. And what about outputs that may improve over as a result ? Same. Calling them "ML" Ops is actually a bit disingenuous. Optimisation cases (with constraint or MIP solvers) that have nothing to do with ML and models also benefit from the same framework.

zach_garwood 4y ago

Completely irrelevant: you spelled "oops" as "oups" and now I'm imagining thats the French spelling.