8 ms·

With open source software, you have the possibility of knowing how safe it is. With closed source, you don't (ok, it's a lot harder to read through binaries, bu

With open source software, you have the possibility of knowing how safe it is. With closed source, you don't (ok, it's a lot harder to read through binaries, but still vaguely possible).

tigershark 10y ago
No way that you can understand how safe is the neural network that is driving your car. And this applies equally to open source autopilot systems and closed ones. The article is just a rant that misses this fundamental point.
- chmars 10y ago
  I cannot, others can - I can even pay them to do if necessary.
  - tigershark 10y ago
    No one can. If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. Good luck for doing it to automatically drive a car. Strangely every company in the world is focusing on NN for autopilot, not on hand written algorithms.
    - ktRolster 10y ago
      If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. You can, actually, the result would be extremely complex, though.
  - EGreg 10y ago
    Can you pay someone to solve the halting problem?
    - halomru 10y ago
      No, but neither can anyone else. I can't proof that the neutral network always behaves correctly, but with testing and statistics you can verify the network up to a chosen certainty (below 100% obviously). Or you can actively search for problems with fuzzing techniques. All without solving the halting problem.
      - EGreg 10y ago
        Right, but with the code in the open, it's easier to find ONE easy hole than to patch ALL the easy holes.
        dogma1138 10y ago
        No that's not really how ANN's work.
        EGreg 10y ago
        ANNs?
        dogma1138 10y ago
        artificial neural networks, you won't really be able to opening the code won't tell you much if anything, it's simply beyond the human capability to review. So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end. You can test ANN's by simply fuzzing them and seeing the output, getting much insight to why an outcome was chosen for a given data set is near impossible.
        nickpsecurity 10y ago
        Lots of ways to review them: https://people.cs.umass.edu/~btaylor/publications/PSI000008.pdf https://people.cs.umass.edu/~btaylor/publications/PSI000008.... These are also just the first steps. Not bad given what hardware and software verification used to look like a few decades ago. I think the verifiable ANN's will be more structured or constrained in their development or even use cases.
- deleted 10y ago
  [deleted]
- chris_wot 10y ago
  Says who? You may not be able to (I sure can't) but that doesn't mean there aren't those who can and do understand this sort of thing. And there are those who will literally sit down for months and years till they do understand it. Or they might rip it out and replace it.
  - tigershark 10y ago
    Guys, are you really saying that you think that you can certify every response of a NN for every input? Especially for NN as complicated as autopilot systems? Are you aware that if that was true then you would be the richest men in the world given that everyone would come to you to write a perfect algorithm to drive a car?
    - ptero 10y ago
      Certifiable reliability is not the only goal. But even in safety critical tasks I have seen NNs and MDPs used to derive a good enough solution, then training turned off and the system certified (e.g. by running a large gold standard collection of test cases)
  - jobigoud 10y ago
    They will understand how it works in principle, not why it took this or that specific decision and they might not be able to exactly predict what it will do in an unforeseen situation. The decision itself might be taken using an amount of data completely overwhelming to human being analysis, processed at inhuman speed.
    - tigershark 10y ago
      Exactly, so having an open source autopilot systems would not have changed absolutely nothing, and the guy on the tesla watching Harry Potter would have been dead the same.
- nickpsecurity 10y ago
  Nice strawman given most systems use deterministic methods for that reason. Plus, simplest is often best in safety-critical. That said, you could've gotten more answers on Google with terms formal verification neural nets. NASA paper with good methods was first result for me.
  - tigershark 10y ago
    Good methods for what? Not certainly to certify that each combination of inputs gives a certain output, as you can read from that paper. And that is the only way that would have ensured that the guy on the tesla didn't die. That methodology furthermore has nothing to do with open source.
    - nickpsecurity 10y ago
      "Good methods for what? Not certainly to certify that each combination of inputs gives a certain output, as you can read from that paper." You're ignoring the biggest issue in your whole tangent: real-world engineers with sense won't use NN's in systems unless the problem itself is non-deterministic & other methods don't work. Personally, I push control theory, state machines, ladder logic, whatever for applications like this. You can deterministically analyze them. Whereas, problems with too much uncertainty or requiring approximate solutions will always have an error margin. You personally have been pushing this thread to focus as much as possible on the NN case while ignoring 99% of real-world, use cases representing how the OP topic will usually play out. You also act like the inherent error in the tooling is bad when the problem domain itself has inherent error. So, I decided to address your tangent. The first solution, safety monitors, goes way back in safety-critical use. That's to have a simple solution checking the ranges or other behavior of the NN that's able to override faulty NN with simpler logic. This can be done for GPS-guided drones that usually move in straight lines or smooth curves. The NN part is reset, diagnosed, or deactivated while the fail-safe system takes plane on simpler path or out of the area. Second part checks the nets themselves. Again, given a non-deterministic problem domain, the nets will be non-deterministic with the best one can do getting the error margin way down or within fail-safe bounds. This other paper I found summarizes a lot of V&V methods for NN's: https://people.cs.umass.edu/~btaylor/publications/PSI000008.pdf https://people.cs.umass.edu/~btaylor/publications/PSI000008.... Safety monitors. Training data or updates with constraints built in. Automated, test-case generation tweaking expected data. Rule extraction. System visualization. Simulation. All should contribute to safer NN's. They can't be perfectly correct, though, because vision, complex flying, thinking... none of this is 100% accurate no matter what method you use.
- amalcon 10y ago
  Sure you can. It's just not by verifying the algorithm, it's by analyzing real-world performance. This is the same way that we understand how safe the human driving the car is, and the insurance industry is very good at this.
  - eridius 10y ago
    Analyzing real-world performance doesn't care if the source is open or proprietary, so you're kind of missing the point of this whole thread.
nickpsecurity 10y ago
This isn't true at all. What's necessary is code written in a verifiable way, qualified reviewer assessing it, trust in that reviewer/assessment, and way to ensure you're using what was reviewed in evaluated configuration. It's how main, security certifications work if you do it at high-assurance level. Lower levels are paper BS or simply inadequate given short-term, code review only catches so much. I broke it down here with a quick revision just now: http://pastebin.com/pjc0cueA http://pastebin.com/pjc0cueA

With open source software, you have the possibility of knowing how safe it is. With closed source, you don't (ok, it's a lot harder to read through binaries, but still vaguely possible).

tigershark 10y ago

No way that you can understand how safe is the neural network that is driving your car. And this applies equally to open source autopilot systems and closed ones. The article is just a rant that misses this fundamental point.

chmars 10y ago
I cannot, others can - I can even pay them to do if necessary.
- tigershark 10y ago
  No one can. If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. Good luck for doing it to automatically drive a car. Strangely every company in the world is focusing on NN for autopilot, not on hand written algorithms.
  - ktRolster 10y ago
    If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. You can, actually, the result would be extremely complex, though.
- EGreg 10y ago
  Can you pay someone to solve the halting problem?
  - halomru 10y ago
    No, but neither can anyone else. I can't proof that the neutral network always behaves correctly, but with testing and statistics you can verify the network up to a chosen certainty (below 100% obviously). Or you can actively search for problems with fuzzing techniques. All without solving the halting problem.
    - EGreg 10y ago
      Right, but with the code in the open, it's easier to find ONE easy hole than to patch ALL the easy holes.
      - dogma1138 10y ago
        No that's not really how ANN's work.
        EGreg 10y ago
        ANNs?
        dogma1138 10y ago
        artificial neural networks, you won't really be able to opening the code won't tell you much if anything, it's simply beyond the human capability to review. So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end. You can test ANN's by simply fuzzing them and seeing the output, getting much insight to why an outcome was chosen for a given data set is near impossible.
        nickpsecurity 10y ago
        Lots of ways to review them: https://people.cs.umass.edu/~btaylor/publications/PSI000008.pdf https://people.cs.umass.edu/~btaylor/publications/PSI000008.... These are also just the first steps. Not bad given what hardware and software verification used to look like a few decades ago. I think the verifiable ANN's will be more structured or constrained in their development or even use cases.
deleted 10y ago
[deleted]
chris_wot 10y ago
Says who? You may not be able to (I sure can't) but that doesn't mean there aren't those who can and do understand this sort of thing. And there are those who will literally sit down for months and years till they do understand it. Or they might rip it out and replace it.
- tigershark 10y ago
  Guys, are you really saying that you think that you can certify every response of a NN for every input? Especially for NN as complicated as autopilot systems? Are you aware that if that was true then you would be the richest men in the world given that everyone would come to you to write a perfect algorithm to drive a car?
  - ptero 10y ago
    Certifiable reliability is not the only goal. But even in safety critical tasks I have seen NNs and MDPs used to derive a good enough solution, then training turned off and the system certified (e.g. by running a large gold standard collection of test cases)
- jobigoud 10y ago
  They will understand how it works in principle, not why it took this or that specific decision and they might not be able to exactly predict what it will do in an unforeseen situation. The decision itself might be taken using an amount of data completely overwhelming to human being analysis, processed at inhuman speed.
  - tigershark 10y ago
    Exactly, so having an open source autopilot systems would not have changed absolutely nothing, and the guy on the tesla watching Harry Potter would have been dead the same.
nickpsecurity 10y ago
Nice strawman given most systems use deterministic methods for that reason. Plus, simplest is often best in safety-critical. That said, you could've gotten more answers on Google with terms formal verification neural nets. NASA paper with good methods was first result for me.
- tigershark 10y ago
  Good methods for what? Not certainly to certify that each combination of inputs gives a certain output, as you can read from that paper. And that is the only way that would have ensured that the guy on the tesla didn't die. That methodology furthermore has nothing to do with open source.
  - nickpsecurity 10y ago
    "Good methods for what? Not certainly to certify that each combination of inputs gives a certain output, as you can read from that paper." You're ignoring the biggest issue in your whole tangent: real-world engineers with sense won't use NN's in systems unless the problem itself is non-deterministic & other methods don't work. Personally, I push control theory, state machines, ladder logic, whatever for applications like this. You can deterministically analyze them. Whereas, problems with too much uncertainty or requiring approximate solutions will always have an error margin. You personally have been pushing this thread to focus as much as possible on the NN case while ignoring 99% of real-world, use cases representing how the OP topic will usually play out. You also act like the inherent error in the tooling is bad when the problem domain itself has inherent error. So, I decided to address your tangent. The first solution, safety monitors, goes way back in safety-critical use. That's to have a simple solution checking the ranges or other behavior of the NN that's able to override faulty NN with simpler logic. This can be done for GPS-guided drones that usually move in straight lines or smooth curves. The NN part is reset, diagnosed, or deactivated while the fail-safe system takes plane on simpler path or out of the area. Second part checks the nets themselves. Again, given a non-deterministic problem domain, the nets will be non-deterministic with the best one can do getting the error margin way down or within fail-safe bounds. This other paper I found summarizes a lot of V&V methods for NN's: https://people.cs.umass.edu/~btaylor/publications/PSI000008.pdf https://people.cs.umass.edu/~btaylor/publications/PSI000008.... Safety monitors. Training data or updates with constraints built in. Automated, test-case generation tweaking expected data. Rule extraction. System visualization. Simulation. All should contribute to safer NN's. They can't be perfectly correct, though, because vision, complex flying, thinking... none of this is 100% accurate no matter what method you use.
amalcon 10y ago
Sure you can. It's just not by verifying the algorithm, it's by analyzing real-world performance. This is the same way that we understand how safe the human driving the car is, and the insurance industry is very good at this.
- eridius 10y ago
  Analyzing real-world performance doesn't care if the source is open or proprietary, so you're kind of missing the point of this whole thread.

nickpsecurity 10y ago

This isn't true at all. What's necessary is code written in a verifiable way, qualified reviewer assessing it, trust in that reviewer/assessment, and way to ensure you're using what was reviewed in evaluated configuration. It's how main, security certifications work if you do it at high-assurance level. Lower levels are paper BS or simply inadequate given short-term, code review only catches so much. I broke it down here with a quick revision just now: http://pastebin.com/pjc0cueA http://pastebin.com/pjc0cueA