3 ms·

> But SARA doesn’t rely as much on “high-order” functions like AI since those are harder to certify. (“To be FAA certifiable, you need a certain level of determ

> But SARA doesn’t rely as much on “high-order” functions like AI since those are harder to certify. (“To be FAA certifiable, you need a certain level of determinism in the outcomes,” Van Buiten says.) I wonder whether a hybrid approach is the future, since in some tasks neural networks are just far better than anything we have. If the only tasks of the neural network is to estimate/classify some sensor-input and is trained in a purely supervised setting, the "right thing" for the neural network is still pretty well defined and rigorous testing should be possible (simple tasks can be very complex to implement). Then, interpretable, high-level reasoning could be solved by old-school coding (and maybe verifying). This is not possible with end-to-end training. But I am not sure what they mean, normally neural network (and their training) is purely deterministic. It's not that they are just very good at rolling a dice. I am not into this stuff (autonomous, "intelligent" systems, more the data-analysis guy), but I would use neural networks for simple to define, hard problems that involve a lot of noisy data (where some kind of accuracy on some test-set is a well-defined metric) and then build a higher-level reasing system by hand.

sgillen 8y ago
I think they are using the wrong word here, I think by "determinism" they mean interpretable. You train your big neural network to classify images, it works fine on the test set but then one day in the wild you discover your network classified a turtle as a rifle [0]. Why did it do that? what can we change to fix that, how could we have seen this coming? Those answers won't come easily compared to a more old school system. [0] https://arxiv.org/abs/1707.07397 https://arxiv.org/abs/1707.07397
- LeanderK 8y ago
  I am reasonably well versed in the whole adversarial-input research. I am not sure whether this is really the problem. It's a tradeoff. This essentially introduces a random chance that you classification will fail (since adversarial input is not constrained to hard-to-classify situations), but while less accurate methods may be more robust to random errors (which is hard to verify for many approaches) but they have a higher likelihood to fail with hard situations. Doesn't every component in an aircraft have a random chance of failure? There's even a name for one critical component, the Jesus nut. I don't really get the interpretable argument. I think what you want is to verify it to a reasonable degree.
  - areoform 8y ago
    > Doesn't every component in an aircraft have a random chance of failure? There's even a name for one critical component, the Jesus nut. Two things; 1) This might be theoretically true, but Boeing and other aircraft manufacturers monitor data on every component at a very fine-grained level and they use inspections to predict what will go bad and when. They're even adding real-time capabilities to it so that parts are replaced immediately after a flight if imminent failure is predicted; https://www.boeing.com/commercial/aeromagazine/articles/qtr_3_07/AERO_Q307_article4.pdf https://www.boeing.com/commercial/aeromagazine/articles/qtr_... 2) The core issue over here isn't that a component can fail. The issue is that when it fails can we trace the error and forestall it in the future? If the error is something that's unreplicable and unpredictable then that's quite literally impossible and it makes the machine by its very definition untenably unsafe as you don't know when or why it won't work. Someday neural networks will become a part of safety-critical systems, but this generation of neural networks probably won't be it.
    - LeanderK 8y ago
      I convinced that neural network will never be interpretable like code. They will just get even more complex, so to employ them in critical applications (my motivation is more healthcare, where similiar constraints are present. People can die if there's a wrong prediction) we have to build a notion why we allow something and what we don't allow without some human in the loop getting convined that the reasoning itself is sound. I think this boils down to assumptions about robustness (like the l2-norm adversarial examples in image classification) and statistics. to 1). Neural networks are code, they are always 100% the same if you ship it. So you can test them way more thoughly than some equipment, which (i hope) would allow you to minimze the chance of failure to a reasonable degree. to 2) "you don't know when or why it won't work." I think a fundamental disagreement lies here. I am convinced that this is not possible with these complex function approximators, where we just optimize them until we are happy. If we adopt a thinking like above, we can retrain the networks with these additional inputs, adjust our notion of robustness and think whether there's a flaw in our statistical assumptions. But these are just random thoughts. I am in touch with people working in healthcare with this stuff, so there's some exposure to these kinds of problems. But I have never read any work discussing these issues or really reflection whether my reasoning is actually sound. Maybe I'm wrong and we can somehow make them interpretable in a sense that is actually relevant to verifying them.

sgillen 8y ago

I think they are using the wrong word here, I think by "determinism" they mean interpretable. You train your big neural network to classify images, it works fine on the test set but then one day in the wild you discover your network classified a turtle as a rifle [0]. Why did it do that? what can we change to fix that, how could we have seen this coming? Those answers won't come easily compared to a more old school system. [0] https://arxiv.org/abs/1707.07397 https://arxiv.org/abs/1707.07397

LeanderK 8y ago
I am reasonably well versed in the whole adversarial-input research. I am not sure whether this is really the problem. It's a tradeoff. This essentially introduces a random chance that you classification will fail (since adversarial input is not constrained to hard-to-classify situations), but while less accurate methods may be more robust to random errors (which is hard to verify for many approaches) but they have a higher likelihood to fail with hard situations. Doesn't every component in an aircraft have a random chance of failure? There's even a name for one critical component, the Jesus nut. I don't really get the interpretable argument. I think what you want is to verify it to a reasonable degree.
- areoform 8y ago
  > Doesn't every component in an aircraft have a random chance of failure? There's even a name for one critical component, the Jesus nut. Two things; 1) This might be theoretically true, but Boeing and other aircraft manufacturers monitor data on every component at a very fine-grained level and they use inspections to predict what will go bad and when. They're even adding real-time capabilities to it so that parts are replaced immediately after a flight if imminent failure is predicted; https://www.boeing.com/commercial/aeromagazine/articles/qtr_3_07/AERO_Q307_article4.pdf https://www.boeing.com/commercial/aeromagazine/articles/qtr_... 2) The core issue over here isn't that a component can fail. The issue is that when it fails can we trace the error and forestall it in the future? If the error is something that's unreplicable and unpredictable then that's quite literally impossible and it makes the machine by its very definition untenably unsafe as you don't know when or why it won't work. Someday neural networks will become a part of safety-critical systems, but this generation of neural networks probably won't be it.
  - LeanderK 8y ago
    I convinced that neural network will never be interpretable like code. They will just get even more complex, so to employ them in critical applications (my motivation is more healthcare, where similiar constraints are present. People can die if there's a wrong prediction) we have to build a notion why we allow something and what we don't allow without some human in the loop getting convined that the reasoning itself is sound. I think this boils down to assumptions about robustness (like the l2-norm adversarial examples in image classification) and statistics. to 1). Neural networks are code, they are always 100% the same if you ship it. So you can test them way more thoughly than some equipment, which (i hope) would allow you to minimze the chance of failure to a reasonable degree. to 2) "you don't know when or why it won't work." I think a fundamental disagreement lies here. I am convinced that this is not possible with these complex function approximators, where we just optimize them until we are happy. If we adopt a thinking like above, we can retrain the networks with these additional inputs, adjust our notion of robustness and think whether there's a flaw in our statistical assumptions. But these are just random thoughts. I am in touch with people working in healthcare with this stuff, so there's some exposure to these kinds of problems. But I have never read any work discussing these issues or really reflection whether my reasoning is actually sound. Maybe I'm wrong and we can somehow make them interpretable in a sense that is actually relevant to verifying them.