4 ms·

In part because ML fails silently by design. Even if the code runs flawlessly with no errors, the outputs could be completely bunk, useless, or even harmful, an

by uoaei 5y ago

In part because ML fails silently by design. Even if the code runs flawlessly with no errors, the outputs could be completely bunk, useless, or even harmful, and you won't have any idea if that is true just from watching The Number go down during training. It's not enough to know how to build it but also how it works. It's the difference between designing the JWST and assembling it.

minimaxir 5y ago
> In part because ML fails silently by design. That's why there's so much iteration and feedback gathering (e.g. A/B tests) as a part of DS/ML, which incidentally is rarely a part of the interview loop. Anyone who claims they can get a good model the first time they train it is dangerously optimistic. Even the "how it works" aspect has become more and more marginal due to black boxing.
borroka 5y ago
But the OP was asking something different, that is why someone should excessively focus on theory, when, by the way, DL theory is very far from being solid and trial and error in ML and AI is the common way of operating. The "model is in place, but I have no clue what's doing and so it can fail without me understanding when and how is straw-man". Especially for supervised learning, that is, we have a label for data, it is immediately clear whether the output of the model is "bunk, useless, or even harmful". There is no "fail silently by design". I have been working in the field for almost 20 years in academia and in industry and it is not that I starting every PCA thinking about eigenvectors and eigenvalues and if you ask me now without preparing what are those, I would be between approximately right and wrong. But I fit many, many very accurate models.
- uoaei 5y ago
  You are considering only the technical aspects of the model. While of course important to understand, those are less interesting when considering potential harms than the downstream effects of the inference pipeline, particularly when it comes to interpretations of outputs. What is absolutely the worst possible MO is to offload the interpretation portion of a pipeline to a machine using proxy metrics without an exceptional model which justifies the approach unequivocally. For instance, if we put an MSE loss function on a classification NN with sigmoid outputs, and used a classification dataset, we could generate an entire zoo of "many, many very accurate models" as measured by MSE. But once your model returns outputs, how do you interpret them to predict a label for some input data? You could hack some algorithm together (eg argmax of the highest value) which is indistinguishable from the "correct" procedure but the described probabilities are so incorrect that no ML professional would be comfortable trusting anything it says, not least because of the violation of the condition that the probabilities are non-negative and sum to one. But being able to explain why we use MSE or cross-entropy or any other loss function and which output activations (hint: and probability distributions) they are typically associated with actually has a very deep origin in the foundations of probability theory which blows open a whole new way of thinking about statistical modelling that is not made available in any of the programs whose materials I've been exposed to.
  - borroka 5y ago
    What you described seems to me pretty standard in ML and even more in statistical modeling. Maybe because I am coming from applied math and statistics.
  - spekcular 5y ago
    "But being able to explain why we use MSE or cross-entropy or any other loss function and which output activations (hint: and probability distributions) they are typically associated with actually has a very deep origin in the foundations of probability theory which blows open a whole new way of thinking about statistical modelling that is not made available in any of the programs whose materials I've been exposed to. " What is the "very deep origin"? What is this "new way of thinking"? And what's so wrong with using argmax to make a classifier, if I don't care about estimating probabilities and just want the answer?
    - uoaei 5y ago
      A lot of processes downstream to inference benefit from having a minimum of care put into the system design. We're talking 80/20 rule stuff here. It's a simple reorientation vs a janky argmax-classifier, but results in assumptions being obeyed broadly, in a max-entropy sense. The key insight is that all prediction models can equally be framed as energy-based models (y = f(x) -> E = g(x, y)) and the job of ML is to estimate the joint distribution of x and y with suitable max-entropy surrogate distributions, and performing MLE on this variational distribution vs some training data. All the math in the theory follows from this (perhaps excluding causal stuff but actually I am not familiar enough with those techniques to say for sure). Things get a little more complicated when you consider e.g. autoencoders but above still holds. Obviously with the choice of a poor surrogate distribution, your predictions will on average be worse. Yes, even if you don't care about probabilities and just want max-likelihood predictions -- your predictions will on average be worse. By construction, analysis proceeds by framing the problem as this and following through. A janky argmax-classifier is not exempted from this -- it, too, already implies a surrogate distribution, but you know, statistically speaking, it's probably a pretty bad one. So it makes sense to put a tiny bit more effort to get way closer to representing the space that your data lives in. Naturally, you could easily find a janky model that outperforms some relatively unoptimized principled model on a specific use case, and many do get lucky with this. But the principled model has a lot more headroom specifically in terms of the information it can hold, because if the design is more or less correct to the problem specification then the inductive bias built into the model matches closely with the structure of the data which is observed.
mattkrause 5y ago
I'm sure this happens, but do you think the problem is actually one of mathematical savvy? My guess would be that more machine learning projects go off the rails for want of understanding the data or the {business, research} problem.
- Mehdi2277 5y ago
  My experience is bulk of the problem is insufficient monitoring. ML systems need heavy monitoring and should be sending lots of metrics to stuff like prometheus/grafana. There should also be validation/consistency checks for all data pipeline/feature transformations. And you should strongly avoid duplicating logic for stuff like feature preprocessing. I've seen people implement "same" feature preprocessing pipeline twice (one python, one java) and it is so common to find edge case bugs for a long time especially when these bugs only slightly impact model behavior. Another issue is proliferation of data pipelines. The more distinct pipelines you have, the more painful they become to monitor. It is much better to minimize pipelines and do views on a small number. I think proliferations of models is a similar issue. It is often easier to build 4 models instead of 1 multi-task model, but monitoring/operational tasks grow more and more painful as you manage more models.
- uoaei 5y ago
  Not necessarily mathematical savvy though a lot of deeper understanding can follow from a strong grasp on the fundamentals. I think it has more to do with the alignment between intuitions and outcomes, and this is not taught well in most academic programs as far as I can tell.
dekhn 5y ago
ML doesn't just fail silently by design. because ML is based on error minimization, it fails in a way that is maximally hard to tell from random garbage. This is, remarkably, a subtlety that is lost on most people, which is a real surprise- my introduction to this was in structural biology, where you always do hold-outs and check the performance on the hold-out set before overfitting is such a problem.
- uoaei 5y ago
  Absolutely, the result you get is "the best you can do given the baked-in assumptions". But of course the assumptions can be wrong. And it takes time to learn how to evaluate and revise your assumptions in any analytical field, hard or soft.
light_hue_1 5y ago
And you learn literally zero about a candidate's ability to understand when and why things work by asking questions about eigenvectors. Someone can understand what an eigenvector is and still not have any clue about how you figure out a system is working, why it's working, what is likely to happen in production, how you test the limits of your method's ability to generalize, how you take an real problem and find something that you can productively use ML on, etc. People say things like "you need to know how it works" but "it" doesn't work using your knowledge of eigenvectors. If you want to test how "it" works, test that, literally. Put up a model on the board and a dataset. Ask people about what might happen when you apply one to the other. What changes they would make in response to changes in the data. What they would do in response to the following training curves, budget limitations, etc. These interviews are terrible and they select for people that regurgitate facts.
- uoaei 5y ago
  The "trivia" tests, when used (IMO) correctly, are not for testing whether or not the candidate recognizes the term and can regurgitate a definition. I prefer to listen to how they phrase their response to get a sense of the intuition behind the understanding of the concept as well as how it may fit in to a larger mathematical framework (i.e. their internal model for mathematical analysis). I am not looking for someone to answer the question correctly, but to answer the question in a way that demonstrates deeper insights, which helps immensely in research settings as re-using properties of mathematical constructs in novel ways is often how theory and practice both are advanced. I would be much less interested in someone giving a precise definition of eigenvalues than to describe them in such a way that they understand e.g. what can be deduced about an operator when one of its eigenvalues is zero.

by uoaei 5y ago

minimaxir 5y ago

> In part because ML fails silently by design. That's why there's so much iteration and feedback gathering (e.g. A/B tests) as a part of DS/ML, which incidentally is rarely a part of the interview loop. Anyone who claims they can get a good model the first time they train it is dangerously optimistic. Even the "how it works" aspect has become more and more marginal due to black boxing.

borroka 5y ago

But the OP was asking something different, that is why someone should excessively focus on theory, when, by the way, DL theory is very far from being solid and trial and error in ML and AI is the common way of operating. The "model is in place, but I have no clue what's doing and so it can fail without me understanding when and how is straw-man". Especially for supervised learning, that is, we have a label for data, it is immediately clear whether the output of the model is "bunk, useless, or even harmful". There is no "fail silently by design". I have been working in the field for almost 20 years in academia and in industry and it is not that I starting every PCA thinking about eigenvectors and eigenvalues and if you ask me now without preparing what are those, I would be between approximately right and wrong. But I fit many, many very accurate models.

uoaei 5y ago
You are considering only the technical aspects of the model. While of course important to understand, those are less interesting when considering potential harms than the downstream effects of the inference pipeline, particularly when it comes to interpretations of outputs. What is absolutely the worst possible MO is to offload the interpretation portion of a pipeline to a machine using proxy metrics without an exceptional model which justifies the approach unequivocally. For instance, if we put an MSE loss function on a classification NN with sigmoid outputs, and used a classification dataset, we could generate an entire zoo of "many, many very accurate models" as measured by MSE. But once your model returns outputs, how do you interpret them to predict a label for some input data? You could hack some algorithm together (eg argmax of the highest value) which is indistinguishable from the "correct" procedure but the described probabilities are so incorrect that no ML professional would be comfortable trusting anything it says, not least because of the violation of the condition that the probabilities are non-negative and sum to one. But being able to explain why we use MSE or cross-entropy or any other loss function and which output activations (hint: and probability distributions) they are typically associated with actually has a very deep origin in the foundations of probability theory which blows open a whole new way of thinking about statistical modelling that is not made available in any of the programs whose materials I've been exposed to.
- borroka 5y ago
  What you described seems to me pretty standard in ML and even more in statistical modeling. Maybe because I am coming from applied math and statistics.
- spekcular 5y ago
  "But being able to explain why we use MSE or cross-entropy or any other loss function and which output activations (hint: and probability distributions) they are typically associated with actually has a very deep origin in the foundations of probability theory which blows open a whole new way of thinking about statistical modelling that is not made available in any of the programs whose materials I've been exposed to. " What is the "very deep origin"? What is this "new way of thinking"? And what's so wrong with using argmax to make a classifier, if I don't care about estimating probabilities and just want the answer?
  - uoaei 5y ago
    A lot of processes downstream to inference benefit from having a minimum of care put into the system design. We're talking 80/20 rule stuff here. It's a simple reorientation vs a janky argmax-classifier, but results in assumptions being obeyed broadly, in a max-entropy sense. The key insight is that all prediction models can equally be framed as energy-based models (y = f(x) -> E = g(x, y)) and the job of ML is to estimate the joint distribution of x and y with suitable max-entropy surrogate distributions, and performing MLE on this variational distribution vs some training data. All the math in the theory follows from this (perhaps excluding causal stuff but actually I am not familiar enough with those techniques to say for sure). Things get a little more complicated when you consider e.g. autoencoders but above still holds. Obviously with the choice of a poor surrogate distribution, your predictions will on average be worse. Yes, even if you don't care about probabilities and just want max-likelihood predictions -- your predictions will on average be worse. By construction, analysis proceeds by framing the problem as this and following through. A janky argmax-classifier is not exempted from this -- it, too, already implies a surrogate distribution, but you know, statistically speaking, it's probably a pretty bad one. So it makes sense to put a tiny bit more effort to get way closer to representing the space that your data lives in. Naturally, you could easily find a janky model that outperforms some relatively unoptimized principled model on a specific use case, and many do get lucky with this. But the principled model has a lot more headroom specifically in terms of the information it can hold, because if the design is more or less correct to the problem specification then the inductive bias built into the model matches closely with the structure of the data which is observed.

mattkrause 5y ago

I'm sure this happens, but do you think the problem is actually one of mathematical savvy? My guess would be that more machine learning projects go off the rails for want of understanding the data or the {business, research} problem.

Mehdi2277 5y ago
My experience is bulk of the problem is insufficient monitoring. ML systems need heavy monitoring and should be sending lots of metrics to stuff like prometheus/grafana. There should also be validation/consistency checks for all data pipeline/feature transformations. And you should strongly avoid duplicating logic for stuff like feature preprocessing. I've seen people implement "same" feature preprocessing pipeline twice (one python, one java) and it is so common to find edge case bugs for a long time especially when these bugs only slightly impact model behavior. Another issue is proliferation of data pipelines. The more distinct pipelines you have, the more painful they become to monitor. It is much better to minimize pipelines and do views on a small number. I think proliferations of models is a similar issue. It is often easier to build 4 models instead of 1 multi-task model, but monitoring/operational tasks grow more and more painful as you manage more models.
uoaei 5y ago
Not necessarily mathematical savvy though a lot of deeper understanding can follow from a strong grasp on the fundamentals. I think it has more to do with the alignment between intuitions and outcomes, and this is not taught well in most academic programs as far as I can tell.

dekhn 5y ago

ML doesn't just fail silently by design. because ML is based on error minimization, it fails in a way that is maximally hard to tell from random garbage. This is, remarkably, a subtlety that is lost on most people, which is a real surprise- my introduction to this was in structural biology, where you always do hold-outs and check the performance on the hold-out set before overfitting is such a problem.

uoaei 5y ago
Absolutely, the result you get is "the best you can do given the baked-in assumptions". But of course the assumptions can be wrong. And it takes time to learn how to evaluate and revise your assumptions in any analytical field, hard or soft.

light_hue_1 5y ago

And you learn literally zero about a candidate's ability to understand when and why things work by asking questions about eigenvectors. Someone can understand what an eigenvector is and still not have any clue about how you figure out a system is working, why it's working, what is likely to happen in production, how you test the limits of your method's ability to generalize, how you take an real problem and find something that you can productively use ML on, etc. People say things like "you need to know how it works" but "it" doesn't work using your knowledge of eigenvectors. If you want to test how "it" works, test that, literally. Put up a model on the board and a dataset. Ask people about what might happen when you apply one to the other. What changes they would make in response to changes in the data. What they would do in response to the following training curves, budget limitations, etc. These interviews are terrible and they select for people that regurgitate facts.

uoaei 5y ago
The "trivia" tests, when used (IMO) correctly, are not for testing whether or not the candidate recognizes the term and can regurgitate a definition. I prefer to listen to how they phrase their response to get a sense of the intuition behind the understanding of the concept as well as how it may fit in to a larger mathematical framework (i.e. their internal model for mathematical analysis). I am not looking for someone to answer the question correctly, but to answer the question in a way that demonstrates deeper insights, which helps immensely in research settings as re-using properties of mathematical constructs in novel ways is often how theory and practice both are advanced. I would be much less interested in someone giving a precise definition of eigenvalues than to describe them in such a way that they understand e.g. what can be deduced about an operator when one of its eigenvalues is zero.