5 ms·

>The trouble with pure neural net controllers is that they sometimes do something totally bogus for some data point within the normal input space. That's not OK

>The trouble with pure neural net controllers is that they sometimes do something totally bogus for some data point within the normal input space. That's not OK in control systems. Is that actually true? Everything we do at work with neural nets seems to indicate continuity. We do stuff with mathematical simulations and imagelike processing. Latent space for auto encoders tend to be smoothly varying throughout the range of outputs. I thought the problem is just that it isn't proven that neural nets are sufficiently smooth.

mjburgess 7y ago
It's not a continuity problem. It's that "bogus" is defined with respect to what it should do, and that criteria requires a working understanding of the environment which isn't present in the model. If you had all possible images of all possible roads in all possible (lighting, weather,..) conditions, etc. then the ideal behaviour is just a function from all inputs to all desired actions. In the absence of this infinity of data you need a model. Any model which has no explicit understanding of the causal behaviour of an environment is going to misbehave for one of those unseen data points (eg. an unseen road in a weather/lighting/etc. condition). "Correct"(/Safe) behaviour is not a function of pixels. No statistical model which associates pixel patterns with action can be correct.
- allovernow 7y ago
  >Any model which has no explicit understanding of the causal behaviour of an environment is going to misbehave for one of those unseen data points (eg. an unseen road in a weather/lighting/etc. condition I disagree. That sounds like an older interpretation of behavior of simple deep nets like the inception family. My experience with auto encoders and GANs is such that the nets can and unquestionably do interpolate between training data points. What's more, the latent spaces display order reflecting logic - much in the way that you can perform arithmetic on BERT encodings in what amounts to a form of logic. This is the where SOTA is now and I think it's a strong step towards AI, though we're still far from it. The trick is in understanding the boundaries in high dimensional space that your training data represents. So there is some degree of covering all your bases, so to speak, and that requires a new kind of intuition. These are very exciting times in tech/ML.
  - mjburgess 7y ago
    Again you're treating this as-if its a problem of finding a sufficiently smooth function over data points. The problem is that the right behaviour is not a model fitted to that data. It isn't "fitted" at all. When I turn the thermostat the temperature in the room increases. If the thermostat is broken, the temperature does not. The predicting the effect of the thermostat on the room requires intervening in the room to find out if it is broken; it requires having a model of the room, of the thermostat, etc. No system that is not in direct causal contact with its environment can adapt to it. The system -- as it is in contact --- needs to be explicitly modelling the causally relevant features of that environment. This isn't a statistical problem. You cannot learn a function over images to actions because the environment is absent from those images. An infinite number of 2D images contain no 3D information. An infinite number of 3D images contain no skeletons (ie., inner-structure of objects). An infinite number of images of object pieces contain no information on object behaviour. An infinite number of videos of an object behaving contains no information of its behaviour when broken. An infinite number of videos of all possible breakages in all existing environments contains no information about behaviour in new environments. Animals solve this problem by playing with objects, building models of those objects (their causal properties, ie., how they interact with other objects). That requires being-in an environment and explicitly modelling it. There cannot be a "non-bogus" system arrived at via ML. Statistics itself is deficient in providing tools to design systems that "do what they should". Any paradigm which makes the cartesian assumption that "behaviour is a function of data" is necessarily incapable of intelligent adaptation. Environments "as data" are infinities. ie., For statistics to work you need all the relevant variables of an environment to make the "correct decision", and all the data needed to train against those. That's infinite. The "child on road" column isn't going to be fed into the machine. The machine does not, and cannot, even model what a "child" is. Patterns among fractions of infinites is not a basis for saftey (, nor for intelligence).
    - jdmichal 7y ago
      I think most of the systems do try to model the objects around them. Of course, that is also subject to its own statistical anomalies. See report of the Uber/pedestrian collision for what happens when that goes wrong. I don't mean to dismiss your points, though. I wholely agree that if we really want to be moving towards an AI that is anything like animal intelligence, things like play are crucial. I'm a big fan of the attention schema theory. And if that is right, then building systems that can generate such models for their own mental state and others is the way to "consciousness". https://en.wikipedia.org/wiki/Attention_schema_theory https://en.wikipedia.org/wiki/Attention_schema_theory It's all well and good to build systems that can look at faces and statistically say, "oh they're mad". And I think people have this idea that maybe if we build enough of these systems and jam them together, we'll make an SI. But until we can build systems that can model why that person is mad, based on the system's own experience and models, we're just making parlor tricks.
- amelius 7y ago
  Yes but the control-theoretical approach can't interpret pixels at all. But translated to the problem at hand, your argument is correct.
LoSboccacc 7y ago
the Uber incident investigation showed how a continuous input generated wildly varying output in practice. in theory the existence of adversarial images and adversarial training has been studied for a while, but the state of the art can still produce singularities around continuous inputs. start here: https://arxiv.org/pdf/1312.6199.pdf https://arxiv.org/pdf/1312.6199.pdf
deleted 7y ago
[deleted]
marcosdumay 7y ago
> I thought the problem is just that it isn't proven that neural nets are sufficiently smooth. I think this is exactly what the GP was trying to say (at least, this is what I understand from it). The continuity bit is about the (non ML) controller, not about the neural network. Nearly all neural networks generate continuous output. They do interpolate between their learned points. The problem is that it is very hard to verify (and impossible to predict) the interpolation function, so it may cross outside of the safety area on any point.

mjburgess 7y ago

It's not a continuity problem. It's that "bogus" is defined with respect to what it should do, and that criteria requires a working understanding of the environment which isn't present in the model. If you had all possible images of all possible roads in all possible (lighting, weather,..) conditions, etc. then the ideal behaviour is just a function from all inputs to all desired actions. In the absence of this infinity of data you need a model. Any model which has no explicit understanding of the causal behaviour of an environment is going to misbehave for one of those unseen data points (eg. an unseen road in a weather/lighting/etc. condition). "Correct"(/Safe) behaviour is not a function of pixels. No statistical model which associates pixel patterns with action can be correct.

allovernow 7y ago
>Any model which has no explicit understanding of the causal behaviour of an environment is going to misbehave for one of those unseen data points (eg. an unseen road in a weather/lighting/etc. condition I disagree. That sounds like an older interpretation of behavior of simple deep nets like the inception family. My experience with auto encoders and GANs is such that the nets can and unquestionably do interpolate between training data points. What's more, the latent spaces display order reflecting logic - much in the way that you can perform arithmetic on BERT encodings in what amounts to a form of logic. This is the where SOTA is now and I think it's a strong step towards AI, though we're still far from it. The trick is in understanding the boundaries in high dimensional space that your training data represents. So there is some degree of covering all your bases, so to speak, and that requires a new kind of intuition. These are very exciting times in tech/ML.
- mjburgess 7y ago
  Again you're treating this as-if its a problem of finding a sufficiently smooth function over data points. The problem is that the right behaviour is not a model fitted to that data. It isn't "fitted" at all. When I turn the thermostat the temperature in the room increases. If the thermostat is broken, the temperature does not. The predicting the effect of the thermostat on the room requires intervening in the room to find out if it is broken; it requires having a model of the room, of the thermostat, etc. No system that is not in direct causal contact with its environment can adapt to it. The system -- as it is in contact --- needs to be explicitly modelling the causally relevant features of that environment. This isn't a statistical problem. You cannot learn a function over images to actions because the environment is absent from those images. An infinite number of 2D images contain no 3D information. An infinite number of 3D images contain no skeletons (ie., inner-structure of objects). An infinite number of images of object pieces contain no information on object behaviour. An infinite number of videos of an object behaving contains no information of its behaviour when broken. An infinite number of videos of all possible breakages in all existing environments contains no information about behaviour in new environments. Animals solve this problem by playing with objects, building models of those objects (their causal properties, ie., how they interact with other objects). That requires being-in an environment and explicitly modelling it. There cannot be a "non-bogus" system arrived at via ML. Statistics itself is deficient in providing tools to design systems that "do what they should". Any paradigm which makes the cartesian assumption that "behaviour is a function of data" is necessarily incapable of intelligent adaptation. Environments "as data" are infinities. ie., For statistics to work you need all the relevant variables of an environment to make the "correct decision", and all the data needed to train against those. That's infinite. The "child on road" column isn't going to be fed into the machine. The machine does not, and cannot, even model what a "child" is. Patterns among fractions of infinites is not a basis for saftey (, nor for intelligence).
  - jdmichal 7y ago
    I think most of the systems do try to model the objects around them. Of course, that is also subject to its own statistical anomalies. See report of the Uber/pedestrian collision for what happens when that goes wrong. I don't mean to dismiss your points, though. I wholely agree that if we really want to be moving towards an AI that is anything like animal intelligence, things like play are crucial. I'm a big fan of the attention schema theory. And if that is right, then building systems that can generate such models for their own mental state and others is the way to "consciousness". https://en.wikipedia.org/wiki/Attention_schema_theory https://en.wikipedia.org/wiki/Attention_schema_theory It's all well and good to build systems that can look at faces and statistically say, "oh they're mad". And I think people have this idea that maybe if we build enough of these systems and jam them together, we'll make an SI. But until we can build systems that can model why that person is mad, based on the system's own experience and models, we're just making parlor tricks.
amelius 7y ago
Yes but the control-theoretical approach can't interpret pixels at all. But translated to the problem at hand, your argument is correct.

LoSboccacc 7y ago

the Uber incident investigation showed how a continuous input generated wildly varying output in practice. in theory the existence of adversarial images and adversarial training has been studied for a while, but the state of the art can still produce singularities around continuous inputs. start here: https://arxiv.org/pdf/1312.6199.pdf https://arxiv.org/pdf/1312.6199.pdf

deleted 7y ago

[deleted]

marcosdumay 7y ago

> I thought the problem is just that it isn't proven that neural nets are sufficiently smooth. I think this is exactly what the GP was trying to say (at least, this is what I understand from it). The continuity bit is about the (non ML) controller, not about the neural network. Nearly all neural networks generate continuous output. They do interpolate between their learned points. The problem is that it is very hard to verify (and impossible to predict) the interpolation function, so it may cross outside of the safety area on any point.