4 ms·

> Traceability A question for the AI Researchers and Engineers On the topic of _Traceability_ and as someone outside the field this is something that has puzzl

> Traceability A question for the AI Researchers and Engineers On the topic of _Traceability_ and as someone outside the field this is something that has puzzled me: (Broad assumptions) Assuming a 'neural network' is a black box probabilistic model. How do we _prove_ in the context of _safety-critical_ systems that they will behave in a known determined and traceable way for all known cases? I would be curious to understand the process/methods that people use in ADAS system or Medical Imaging (and now Defense applications as well) in order to provide 'verification' evidence for such a system? (Focusing more on verification/provability from a regulatory standpoint)

kuu 7y ago
There is work in progress for understanding the content of a neural network and avoid this "black box" effect. This is still far from reached but there are some advances. See [0] for example. Also, remember that AI includes more than NN. You can use some other models, as a Linear Regression or a Random Forest which are perfectly explainable. [0] https://christophm.github.io/interpretable-ml-book/neural-networks.html https://christophm.github.io/interpretable-ml-book/neural-ne...
- DoingIsLearning 7y ago
  Indeed ML is vast and not limited to NN. My question was more towards people who are actively using NN as a solution and _have_already_ shipped products in a 'regulated' industry. The products are obviously out there, so to me it is confusing what evidence can a team provide at this point in time (with the current NN implementations) in order to get a product using NN (with a non-deterministic uncertainty) approved by a regulatory body? (automotive, medical, etc.) Examples that jump to mind are Tesla's stereovision sub-systems (I am assuming NN are used), or NN based medical imaging classifier software (NN instead of Clustering or SVM techniques)
  - yoav_hollander 7y ago
    Here is a (longish) post I wrote about this issue (mainly from the perspective of verifying autonomous vehicles): https://blog.foretellix.com/2017/07/06/where-machine-learning-meets-rule-based-verification/ https://blog.foretellix.com/2017/07/06/where-machine-learnin...
    - DoingIsLearning 7y ago
      Thank you very much on point. Thanks for sharing!
ganzuul 7y ago
Various model compression schemes get rid of high-frequency noise in the model, combating over-fitting. This results in robustness against adversarial inputs. Some of these schemes are motivated by rather robust theory. Gaussian Processes are a good starting point to learn more.
- DoingIsLearning 7y ago
  This is certainly out of my depth but how would this model 'reduction' improve provability? It would (I assume) reduce instances of non-desirable behaviour. But how would it improve the evidence I am able to provide to a regulating authority?
  - ganzuul 7y ago
    To my layman's understanding; by reducing instances of undefined behavior as those instances should come from the noise of the data. Noise looks like structure in very-high dimensional views of data, so you need bounds on that stuff. For autonomous vehicles the 'proof' to the regulator can be the signature of the engineer. - Fortunately some engineers have higher standards. - I think in the future the trainig data will be from simulations that create curriculum learning datasets so that the noise characteristics are perfectly known. The ML algorithms can be written with dependent types, so that you can prove your code does what you think it does. Another challenge is inductive bias, which is a lot like confirmation bias. This bias comes from choosing an ML algo which is sensitive to certain information and blind to others. You need to navigate the set of all possible functions, AKA Hilbert space, to overcome it. Fortunately only a small corner of this space is relevant to our universe and Tensor Networks seem to address this problem. It looks like a lot of work to put these pieces together but at least it looks like the problem is tractable.
bumby 7y ago
DARPA has acknowledged this problem and is working on it.[1] See slide 11 for some of the technical approaches to making AI more interpretable. [1]https://www.darpa.mil/attachments/XAIProgramUpdate.pdf https://www.darpa.mil/attachments/XAIProgramUpdate.pdf

kuu 7y ago

There is work in progress for understanding the content of a neural network and avoid this "black box" effect. This is still far from reached but there are some advances. See [0] for example. Also, remember that AI includes more than NN. You can use some other models, as a Linear Regression or a Random Forest which are perfectly explainable. [0] https://christophm.github.io/interpretable-ml-book/neural-networks.html https://christophm.github.io/interpretable-ml-book/neural-ne...

DoingIsLearning 7y ago
Indeed ML is vast and not limited to NN. My question was more towards people who are actively using NN as a solution and _have_already_ shipped products in a 'regulated' industry. The products are obviously out there, so to me it is confusing what evidence can a team provide at this point in time (with the current NN implementations) in order to get a product using NN (with a non-deterministic uncertainty) approved by a regulatory body? (automotive, medical, etc.) Examples that jump to mind are Tesla's stereovision sub-systems (I am assuming NN are used), or NN based medical imaging classifier software (NN instead of Clustering or SVM techniques)
- yoav_hollander 7y ago
  Here is a (longish) post I wrote about this issue (mainly from the perspective of verifying autonomous vehicles): https://blog.foretellix.com/2017/07/06/where-machine-learning-meets-rule-based-verification/ https://blog.foretellix.com/2017/07/06/where-machine-learnin...
  - DoingIsLearning 7y ago
    Thank you very much on point. Thanks for sharing!

ganzuul 7y ago

Various model compression schemes get rid of high-frequency noise in the model, combating over-fitting. This results in robustness against adversarial inputs. Some of these schemes are motivated by rather robust theory. Gaussian Processes are a good starting point to learn more.

DoingIsLearning 7y ago
This is certainly out of my depth but how would this model 'reduction' improve provability? It would (I assume) reduce instances of non-desirable behaviour. But how would it improve the evidence I am able to provide to a regulating authority?
- ganzuul 7y ago
  To my layman's understanding; by reducing instances of undefined behavior as those instances should come from the noise of the data. Noise looks like structure in very-high dimensional views of data, so you need bounds on that stuff. For autonomous vehicles the 'proof' to the regulator can be the signature of the engineer. - Fortunately some engineers have higher standards. - I think in the future the trainig data will be from simulations that create curriculum learning datasets so that the noise characteristics are perfectly known. The ML algorithms can be written with dependent types, so that you can prove your code does what you think it does. Another challenge is inductive bias, which is a lot like confirmation bias. This bias comes from choosing an ML algo which is sensitive to certain information and blind to others. You need to navigate the set of all possible functions, AKA Hilbert space, to overcome it. Fortunately only a small corner of this space is relevant to our universe and Tensor Networks seem to address this problem. It looks like a lot of work to put these pieces together but at least it looks like the problem is tractable.

bumby 7y ago

DARPA has acknowledged this problem and is working on it.[1] See slide 11 for some of the technical approaches to making AI more interpretable. [1]https://www.darpa.mil/attachments/XAIProgramUpdate.pdf https://www.darpa.mil/attachments/XAIProgramUpdate.pdf