3 ms·

> What happen when your model ends up in production ? It will start affecting the behaviour of the very thing you're trying to predict Indeed, again part of th

> What happen when your model ends up in production ? It will start affecting the behaviour of the very thing you're trying to predict Indeed, again part of the reasons I think MLOps is a problem. One ideally needs a causal theory and rational expectations regarding how and why a model will work: you should be able to explain it. The reason is that there are many more ways for things to look like they work, then ways for them to actually work. Forcing -- what should be -- the scientific effort of modelling into a cut box, tune, re-train, monitor workflow is reductionist and will invariably make it near impossible to exercise proper due diligence IMO. Scientific discovery is generally not a sausage machine -- its messy and dependent on what happens in brains more than anything else. I think its a fools errand to expect differently from -- what should be -- the same abstract tool box applied to commerce.

Fiahil 4y ago
So. I understand what point your making, but I completely disagree with you. MLOps is as much as an issue to Data Science than DevOps is to Software Engineering. Automating things doesn't make them stupid or constrained ; I don't know why you would think this creates a "cut box" workflow and sabotage the beautiful Scientific process you imagine is happening. Maybe you have an issue with some Data Scientists you are working with ? To your other point below : > The DS cuts a black box which MLOps tunes, monitors and re-trains. MLOps is not about adapting or tuning a model. Not even close, it's just completely out of scope. MLOps is just a set of best practices to ensure you can compare and reproduce previous outputs (a model, plot, shap output, png, or json benchmark report, whatever you want) at any given point in time. What you do with that model, and how/when you want to train it, is your business alone. We need to call these best practices "MLOps" because it's a significant enough break from regular software engineering practices : suddenly pinning a git tag is not enough to ensure reproducibility, now, we also need to pin datasets to their specific state in time. And what about outputs that may improve over as a result ? Same. Calling them "ML" Ops is actually a bit disingenuous. Optimisation cases (with constraint or MIP solvers) that have nothing to do with ML and models also benefit from the same framework.

Fiahil 4y ago

So. I understand what point your making, but I completely disagree with you. MLOps is as much as an issue to Data Science than DevOps is to Software Engineering. Automating things doesn't make them stupid or constrained ; I don't know why you would think this creates a "cut box" workflow and sabotage the beautiful Scientific process you imagine is happening. Maybe you have an issue with some Data Scientists you are working with ? To your other point below : > The DS cuts a black box which MLOps tunes, monitors and re-trains. MLOps is not about adapting or tuning a model. Not even close, it's just completely out of scope. MLOps is just a set of best practices to ensure you can compare and reproduce previous outputs (a model, plot, shap output, png, or json benchmark report, whatever you want) at any given point in time. What you do with that model, and how/when you want to train it, is your business alone. We need to call these best practices "MLOps" because it's a significant enough break from regular software engineering practices : suddenly pinning a git tag is not enough to ensure reproducibility, now, we also need to pin datasets to their specific state in time. And what about outputs that may improve over as a result ? Same. Calling them "ML" Ops is actually a bit disingenuous. Optimisation cases (with constraint or MIP solvers) that have nothing to do with ML and models also benefit from the same framework.