9 ms·

"Unknown unknowns" is an interesting term of art, but it might hide what the issue really is. A UU Example for an ML system is simply proof that it has failed

"Unknown unknowns" is an interesting term of art, but it might hide what the issue really is. A UU Example for an ML system is simply proof that it has failed to acquire the relevant classification (etc.) criteria -- and on most problems, ML systems cannot in principle do so (eg., consider that 3D data is often needed and only 2D given). I'd prefer to call these "breaking examples", or, "beyond-scope examples". What's happening is that the domain of inference (X) fails to contain data about the target (Y), though Y belongs to the "Problem Domain" of which X stands as an incomplete sample. eg., X = images of roads; Y = pedestrian in street; PD = X + 3D data + behavioural data + pedestraian_theoryofmind + weather_data + .. Actually deciding Y requires PD; we have X. So there are many "breaking examples" which fall within PD but outside of X. Note, in general, we cannot measure the full PD using contemporary machine systems. Humans are able to capture much more of the PD in complex ways which would take a while to spell out. Much of modern ML, as advertised, is a bit of delusion and a bit of charlatanism: we are offered a means of obtaining Y via X, but never consider PD. We just keep "evaluating" the systems on examples consistent with X. This is the point of this article. However, it is a much more radical one than it is making out. This practice is really a means of disguising that ML systems do not work.

deleted 6y ago
[deleted]
plutonorm 6y ago
What would you make of Tesla driving AI? It is exposed to the full problem domain. Does this make a different kind of thing?
- nerdponx 6y ago
  It is not. It is exposed to whatever training data it was trained on, and maybe whatever new data it encounters if it is being continuously trained during use.
  - plutonorm 6y ago
    That doesn't make any sense. It has eyes and ears and the training data is the environment.
0-_-0 6y ago
I've heard the difference being referred to as aleatoric uncertainty (noise inherent in the observations) vs. epistemic uncertainty (model uncertainty caused by lack of data)
hallqv 6y ago
What is the real issue in your view? That ml-systems don’t work? In that case I strong disagree, I use multitudes of them every day (and you most likely do as well) and they work very well. As someone working with building ML-systems that solve real-world problems I can say that distinction between known unkowns and unknowns unkowns is a very useful one.
- fock 6y ago
  what kind of ML-systems are you using daily? (non-embedded ones) Probably stuff around search qualifies today, but this does actually not work for me more often than it does. Sometimes I'd just like the index...
  - nerdponx 6y ago
    What constitutes "this" in your case? Machine learning "works" all over the world, all the time, in a wide variety of industries and problem domains. But in many cases it takes months of development and hard work by people with advanced degrees to get to that point. Despite what various AI startups want to sell you, usually you can't just upload a CSV of all your business data to some API and get something useful out of it.
    - fock 6y ago
      ok, so you could you give an example on a day, where I'm only browsing wikipedia with my Linux-computer? I would tend to agree with you and while there certainly are applications, the relevancy is really blown out of proportion. Technically my naive-bayes spam filter is ML. Practically it's not as flashy as people think ML is - just imagine 15 years back and you'll see that your life could work quite well without using ML in the googly-sense on a daily basis.
dumb1224 6y ago
I think the issue this article talks about has been widely discussed. E.g https://www-users.cs.umn.edu/~qzhao/adversarial.html https://www-users.cs.umn.edu/~qzhao/adversarial.html It is a very specific problem in the realm of ML in my opinion. The networks that were mentioned are large and often with millions of parameters. Such models will be more difficult to evaluate why it fails comparing with a simpler model. Besides, the trick to produce such adversarial examples is an optimisation problem in the same sense as a hacking attack that you perturb the input signals in a way that when using this trained gigantic function it maps to a specific classification. The solution is a certain noise, then you add 'noise-like' real life images to make sure the network would ignore (but human would pick out for the obvious).
nerdponx 6y ago
Much of modern ML, as advertised, is a bit of delusion and a bit of charlatanism: we are offered a means of obtaining Y via X, but never consider PD. We just keep "evaluating" the systems on examples consistent with X. The field of statistics has been fully aware of the problems of "in sample" and "out of sample" data for well over 100 years. And it's still one of the first things people learn, and statistics is still one of the foundations of data science and machine learning. Don't project your own lack of knowledge onto people who certainly have the knowledge you claim that they lack.
- mjburgess 6y ago
  Statistics is a "foundation" for ML. But I am talking about the practice of ML. It is precisely via my knowledge of statistics that I make this point: and even google agrees! Hence this very article. However, note, that this isn't about in-out samples or about the "preservation" of the data distribution. It is about the relevant variables for deciding Y not being present in X. Statisticians, in practice, are able to better overcome these issues by simulating relevant variables and, eg., performing a bayesian analysis. Machine Learning systems have no theoretical basis to automatically build the environmental models to "complete X". It is much more radical problem. Statistical model building cannot be automated. What we have, in ML, is the automation of aproximations on top of approximations of model building.
  - dumb1224 6y ago
    'It is about the relevant variables for deciding Y not being present in X.' Could you explain a bit more about this? Can I interpret that as 'X contains some signals that indicate a target Y'? If that's the case then it still isn't that different from classical statistical inference, am I wrong? There are many problems that we can hardly be sure there are relevant signals in X when we are studying the target Y (e.g Biology/Physics), correct?
    - mjburgess 6y ago
      I think the key point here is that ML is *automated* inference. In the sciences "inference to a model" is a process, not a dataset, called experimentation. The scientific method is the sequencing of relevant datasets, each of the latter "ruling out or in" the former. There are many issues with ML systems, fundamentally, that the "automation" involved is not an automation of science (/"learning proper") -- but the automation of function aproximation. A fatal issue is then that the datasets we are asking to be automatically aproximated are radically incomplete, and there is no "scientific method" within the ML system to appeal to. Eg., we are trying to get self-driving cars with devices (cars) which cannot estimate the intentions (mental states) of pedestrians (ie., a missing variable within the PD but not in X). And very profoundly, have even no means to acquire this data, or to temper their "decisions" via being aware of its absense.
sgt101 6y ago
I think it would be better to say that ML works within the PD as defined by its training. Sometimes this is inadequate for application because the PD is not well defined, and sometimes it's inadequate for application because the PD that is defined by training is not the PD in the application. But - when the PD is adequate (not perfect, but adequate) then it works. For example the PD for reading post codes off envelopes is adequately defined. The PD for matching mugshots to crowd video is not adequately defined. Claiming ML systems don't work is a denial of objective reality - your world is made of ML systems that are working every day. You may as well claim that the sun doesn't work because it's dark at night.
- zwaps 6y ago
  There are however many ML systems that do not work, and many more that do not work as intended. Have you seen how many people have been kicked out of insert big corp by some AI decision? That's a problem that comes back to what is well known in statistics: Your model of the PD may be wrong, and the crucial question is - how can you find out? The ML answer is quite simple: I know my model is right, because it works. But this is very limited. Theoretically, you could enumerate all sorts of issues here. At the fundamental level, you will find that no model, no approach and no identification strategy is assumption free. "My model work" is a very simple identification strategy. It assumes, among other things, that the output of both PD in training and the estimated model is congruent with the latent distribution I'll find in reality. Such attempts at induction are, naturally, not as easy as the formulation suggests. You speak of reality, and I find that we are faced with model failures almost every day on HN. And, with all due respect to all ML researchers, there are a lot of teams with smaller experience and resources than Google, Apple and Amazon. I think there's resistance in the ML community to the recognition that somehow, at some point, one will have to deal with icky issues such as causal inference. On the other hand, I think this is precisely where the field moves forward. Until then, there's at least two costs we can readily identify: "Outliers" - people being somehow mishandled by the model - a collection of which is highlighted every day on HN, and second, changes (external or endogenous) to the PD, temporarily breaking models more or less. Again, it's subtle, but it all comes down to identification. I always tell the story of an engineer designing a robot to layer sausage slices. Well-defined PD - as you would say - worked great. Up until someone left the outside door open. Suddenly, the algorithm could not handle the problem anymore. What's the PD, and what changed? Good to think about such things every once in a while.

deleted 6y ago

[deleted]

plutonorm 6y ago

What would you make of Tesla driving AI? It is exposed to the full problem domain. Does this make a different kind of thing?

nerdponx 6y ago
It is not. It is exposed to whatever training data it was trained on, and maybe whatever new data it encounters if it is being continuously trained during use.
- plutonorm 6y ago
  That doesn't make any sense. It has eyes and ears and the training data is the environment.

0-_-0 6y ago

I've heard the difference being referred to as aleatoric uncertainty (noise inherent in the observations) vs. epistemic uncertainty (model uncertainty caused by lack of data)

hallqv 6y ago

What is the real issue in your view? That ml-systems don’t work? In that case I strong disagree, I use multitudes of them every day (and you most likely do as well) and they work very well. As someone working with building ML-systems that solve real-world problems I can say that distinction between known unkowns and unknowns unkowns is a very useful one.

fock 6y ago
what kind of ML-systems are you using daily? (non-embedded ones) Probably stuff around search qualifies today, but this does actually not work for me more often than it does. Sometimes I'd just like the index...
- nerdponx 6y ago
  What constitutes "this" in your case? Machine learning "works" all over the world, all the time, in a wide variety of industries and problem domains. But in many cases it takes months of development and hard work by people with advanced degrees to get to that point. Despite what various AI startups want to sell you, usually you can't just upload a CSV of all your business data to some API and get something useful out of it.
  - fock 6y ago
    ok, so you could you give an example on a day, where I'm only browsing wikipedia with my Linux-computer? I would tend to agree with you and while there certainly are applications, the relevancy is really blown out of proportion. Technically my naive-bayes spam filter is ML. Practically it's not as flashy as people think ML is - just imagine 15 years back and you'll see that your life could work quite well without using ML in the googly-sense on a daily basis.

dumb1224 6y ago

I think the issue this article talks about has been widely discussed. E.g https://www-users.cs.umn.edu/~qzhao/adversarial.html https://www-users.cs.umn.edu/~qzhao/adversarial.html It is a very specific problem in the realm of ML in my opinion. The networks that were mentioned are large and often with millions of parameters. Such models will be more difficult to evaluate why it fails comparing with a simpler model. Besides, the trick to produce such adversarial examples is an optimisation problem in the same sense as a hacking attack that you perturb the input signals in a way that when using this trained gigantic function it maps to a specific classification. The solution is a certain noise, then you add 'noise-like' real life images to make sure the network would ignore (but human would pick out for the obvious).

nerdponx 6y ago

Much of modern ML, as advertised, is a bit of delusion and a bit of charlatanism: we are offered a means of obtaining Y via X, but never consider PD. We just keep "evaluating" the systems on examples consistent with X. The field of statistics has been fully aware of the problems of "in sample" and "out of sample" data for well over 100 years. And it's still one of the first things people learn, and statistics is still one of the foundations of data science and machine learning. Don't project your own lack of knowledge onto people who certainly have the knowledge you claim that they lack.

mjburgess 6y ago
Statistics is a "foundation" for ML. But I am talking about the practice of ML. It is precisely via my knowledge of statistics that I make this point: and even google agrees! Hence this very article. However, note, that this isn't about in-out samples or about the "preservation" of the data distribution. It is about the relevant variables for deciding Y not being present in X. Statisticians, in practice, are able to better overcome these issues by simulating relevant variables and, eg., performing a bayesian analysis. Machine Learning systems have no theoretical basis to automatically build the environmental models to "complete X". It is much more radical problem. Statistical model building cannot be automated. What we have, in ML, is the automation of aproximations on top of approximations of model building.
- dumb1224 6y ago
  'It is about the relevant variables for deciding Y not being present in X.' Could you explain a bit more about this? Can I interpret that as 'X contains some signals that indicate a target Y'? If that's the case then it still isn't that different from classical statistical inference, am I wrong? There are many problems that we can hardly be sure there are relevant signals in X when we are studying the target Y (e.g Biology/Physics), correct?
  - mjburgess 6y ago
    I think the key point here is that ML is *automated* inference. In the sciences "inference to a model" is a process, not a dataset, called experimentation. The scientific method is the sequencing of relevant datasets, each of the latter "ruling out or in" the former. There are many issues with ML systems, fundamentally, that the "automation" involved is not an automation of science (/"learning proper") -- but the automation of function aproximation. A fatal issue is then that the datasets we are asking to be automatically aproximated are radically incomplete, and there is no "scientific method" within the ML system to appeal to. Eg., we are trying to get self-driving cars with devices (cars) which cannot estimate the intentions (mental states) of pedestrians (ie., a missing variable within the PD but not in X). And very profoundly, have even no means to acquire this data, or to temper their "decisions" via being aware of its absense.

sgt101 6y ago

I think it would be better to say that ML works within the PD as defined by its training. Sometimes this is inadequate for application because the PD is not well defined, and sometimes it's inadequate for application because the PD that is defined by training is not the PD in the application. But - when the PD is adequate (not perfect, but adequate) then it works. For example the PD for reading post codes off envelopes is adequately defined. The PD for matching mugshots to crowd video is not adequately defined. Claiming ML systems don't work is a denial of objective reality - your world is made of ML systems that are working every day. You may as well claim that the sun doesn't work because it's dark at night.

zwaps 6y ago
There are however many ML systems that do not work, and many more that do not work as intended. Have you seen how many people have been kicked out of insert big corp by some AI decision? That's a problem that comes back to what is well known in statistics: Your model of the PD may be wrong, and the crucial question is - how can you find out? The ML answer is quite simple: I know my model is right, because it works. But this is very limited. Theoretically, you could enumerate all sorts of issues here. At the fundamental level, you will find that no model, no approach and no identification strategy is assumption free. "My model work" is a very simple identification strategy. It assumes, among other things, that the output of both PD in training and the estimated model is congruent with the latent distribution I'll find in reality. Such attempts at induction are, naturally, not as easy as the formulation suggests. You speak of reality, and I find that we are faced with model failures almost every day on HN. And, with all due respect to all ML researchers, there are a lot of teams with smaller experience and resources than Google, Apple and Amazon. I think there's resistance in the ML community to the recognition that somehow, at some point, one will have to deal with icky issues such as causal inference. On the other hand, I think this is precisely where the field moves forward. Until then, there's at least two costs we can readily identify: "Outliers" - people being somehow mishandled by the model - a collection of which is highlighted every day on HN, and second, changes (external or endogenous) to the PD, temporarily breaking models more or less. Again, it's subtle, but it all comes down to identification. I always tell the story of an engineer designing a robot to layer sausage slices. Well-defined PD - as you would say - worked great. Up until someone left the outside door open. Suddenly, the algorithm could not handle the problem anymore. What's the PD, and what changed? Good to think about such things every once in a while.