10 ms·

I cannot, others can - I can even pay them to do if necessary.

by chmars 10y ago

tigershark 10y ago
No one can. If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. Good luck for doing it to automatically drive a car. Strangely every company in the world is focusing on NN for autopilot, not on hand written algorithms.
- ktRolster 10y ago
  If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. You can, actually, the result would be extremely complex, though.
EGreg 10y ago
Can you pay someone to solve the halting problem?
- halomru 10y ago
  No, but neither can anyone else. I can't proof that the neutral network always behaves correctly, but with testing and statistics you can verify the network up to a chosen certainty (below 100% obviously). Or you can actively search for problems with fuzzing techniques. All without solving the halting problem.
  - EGreg 10y ago
    Right, but with the code in the open, it's easier to find ONE easy hole than to patch ALL the easy holes.
    - dogma1138 10y ago
      No that's not really how ANN's work.
      - EGreg 10y ago
        ANNs?
        dogma1138 10y ago
        artificial neural networks, you won't really be able to opening the code won't tell you much if anything, it's simply beyond the human capability to review. So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end. You can test ANN's by simply fuzzing them and seeing the output, getting much insight to why an outcome was chosen for a given data set is near impossible.
        nickpsecurity 10y ago
        Lots of ways to review them: https://people.cs.umass.edu/~btaylor/publications/PSI000008.pdf https://people.cs.umass.edu/~btaylor/publications/PSI000008.... These are also just the first steps. Not bad given what hardware and software verification used to look like a few decades ago. I think the verifiable ANN's will be more structured or constrained in their development or even use cases.
        dogma1138 10y ago
        Have you actually read what the paper you've linked? I haven't said that it's impossible to review ANN's, but you do not need to review their code to do it, infact reviewing their code is the wrong approach in this case. You do not want to be conducting code inspection on nondeterministic systems as it would yield very little. This means that having the ANN being open sourced does not add additional security by having the code inspectable by everyone like with traditional software. This is what the whole argument is about, not about can ANN's be reviewed.
        10y ago
- chmars 10y ago
  I think we all agree that open source is not the holy grail but it shines with comparative advantages. It is like democracy: Far from perfect but still the best political system: 'Many forms of Government have been tried, and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time.' https://richardlangworth.com/worst-form-of-government https://richardlangworth.com/worst-form-of-government (OK, that was slightly OT!)
  - EGreg 10y ago
    Well if you must :) http://magarshak.com/blog/?p=212 http://magarshak.com/blog/?p=212
    - nickpsecurity 10y ago
      https://en.wikipedia.org/wiki/Tyranny_of_the_majority https://en.wikipedia.org/wiki/Tyranny_of_the_majority
      - EGreg 10y ago
        So the plutocrats will protect us from the majority? :) Obviously we need constitutional rights and bureaucrats to implement the will of the people as expressed in the polls. But why do we need representatives? Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?
        nickpsecurity 10y ago
        "So the plutocrats will protect us from the majority? :)" No. The lesson, as usual, is that extremes rarely work out for us. The plutocrats will screw us if totally in control. So will politicians if they can get money from them or special interest groups. So will democracy if something horrible is trending in popularity. As usual, the solution will be a compromise between various points in the design space with more effective checks on various interactions and risks each pose to each other. I can't tell you what that design will be but looking for it is best investment with these things. "Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?" The First Amendment is subject to interpretation by representatives and/or courts. Fourth, Fifth, and Fourteenth have seriously been watered down. They're among the best examples of the risk. A politician can outlast the media fervor of the moment, get advise from well-informed people, and have teams check the side effects of the law. The current situation exists because apathetic democracy allows them to get away with not doing that. Also, passing laws in exchange for bribes. That could all be stopped with enough voter action followed by legislation or alternative models. The benefits of representational democracy remain. I'm not saying I believe it's the best system. I'm just saying plutocrats and mobs cause lots of problems it can prevent with less effort than constantly outsmarting plutocrats or fighting mobs. So, it's a consideration.
- dllthomas 10y ago
  No, but for any given program I may be able to hire someone to tell me whether it halts. Safety critical automotive software is going to tend to be an easier case, since it's quite likely to be extremely time bounded.
- nickpsecurity 10y ago
  I made money doing it one time. They told me I couldn't prove an arbitrary program would halt. I wrote a program that loaded an arbitrary algorithm into a process, counted down to zero from starting constant, attempted to kill the process when at zero, and otherwise shut down the whole system. "sklogic" similarly showed how pointless the question was when he said he could just put a while loop around any algorithm doing basically the same thing within the program. So, not only can you prove a program halts using a verified halter, the techniques for doing so have prevented and detected many real-world problems. One of those situations where abstract math ignores simple, concrete solutions. :)

I cannot, others can - I can even pay them to do if necessary.

by chmars 10y ago

I cannot, others can - I can even pay them to do if necessary.

No one can. If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. Good luck for doing it to automatically drive a car. Strangely every company in the world is focusing on NN for autopilot, not on hand written algorithms.

ktRolster 10y ago
If anyone could certify the neural network entire response mapping then you could simply write an exact algorithm with the same responses. You can, actually, the result would be extremely complex, though.

EGreg 10y ago

Can you pay someone to solve the halting problem?

halomru 10y ago
No, but neither can anyone else. I can't proof that the neutral network always behaves correctly, but with testing and statistics you can verify the network up to a chosen certainty (below 100% obviously). Or you can actively search for problems with fuzzing techniques. All without solving the halting problem.
- EGreg 10y ago
  Right, but with the code in the open, it's easier to find ONE easy hole than to patch ALL the easy holes.
  - dogma1138 10y ago
    No that's not really how ANN's work.
    - EGreg 10y ago
      ANNs?
      - dogma1138 10y ago
        artificial neural networks, you won't really be able to opening the code won't tell you much if anything, it's simply beyond the human capability to review. So you'll end up with ANN's reviewing other ANN's which are reviewed by other ANN's at the end. You can test ANN's by simply fuzzing them and seeing the output, getting much insight to why an outcome was chosen for a given data set is near impossible.
        nickpsecurity 10y ago
        Lots of ways to review them: https://people.cs.umass.edu/~btaylor/publications/PSI000008.pdf https://people.cs.umass.edu/~btaylor/publications/PSI000008.... These are also just the first steps. Not bad given what hardware and software verification used to look like a few decades ago. I think the verifiable ANN's will be more structured or constrained in their development or even use cases.
        dogma1138 10y ago
        Have you actually read what the paper you've linked? I haven't said that it's impossible to review ANN's, but you do not need to review their code to do it, infact reviewing their code is the wrong approach in this case. You do not want to be conducting code inspection on nondeterministic systems as it would yield very little. This means that having the ANN being open sourced does not add additional security by having the code inspectable by everyone like with traditional software. This is what the whole argument is about, not about can ANN's be reviewed.
        10y ago
chmars 10y ago
I think we all agree that open source is not the holy grail but it shines with comparative advantages. It is like democracy: Far from perfect but still the best political system: 'Many forms of Government have been tried, and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time.' https://richardlangworth.com/worst-form-of-government https://richardlangworth.com/worst-form-of-government (OK, that was slightly OT!)
- EGreg 10y ago
  Well if you must :) http://magarshak.com/blog/?p=212 http://magarshak.com/blog/?p=212
  - nickpsecurity 10y ago
    https://en.wikipedia.org/wiki/Tyranny_of_the_majority https://en.wikipedia.org/wiki/Tyranny_of_the_majority
    - EGreg 10y ago
      So the plutocrats will protect us from the majority? :) Obviously we need constitutional rights and bureaucrats to implement the will of the people as expressed in the polls. But why do we need representatives? Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?
      - nickpsecurity 10y ago
        "So the plutocrats will protect us from the majority? :)" No. The lesson, as usual, is that extremes rarely work out for us. The plutocrats will screw us if totally in control. So will politicians if they can get money from them or special interest groups. So will democracy if something horrible is trending in popularity. As usual, the solution will be a compromise between various points in the design space with more effective checks on various interactions and risks each pose to each other. I can't tell you what that design will be but looking for it is best investment with these things. "Do you really think eg a Tea Party representative will protect Muslims from the majority better than the first amendment and courts?" The First Amendment is subject to interpretation by representatives and/or courts. Fourth, Fifth, and Fourteenth have seriously been watered down. They're among the best examples of the risk. A politician can outlast the media fervor of the moment, get advise from well-informed people, and have teams check the side effects of the law. The current situation exists because apathetic democracy allows them to get away with not doing that. Also, passing laws in exchange for bribes. That could all be stopped with enough voter action followed by legislation or alternative models. The benefits of representational democracy remain. I'm not saying I believe it's the best system. I'm just saying plutocrats and mobs cause lots of problems it can prevent with less effort than constantly outsmarting plutocrats or fighting mobs. So, it's a consideration.
dllthomas 10y ago
No, but for any given program I may be able to hire someone to tell me whether it halts. Safety critical automotive software is going to tend to be an easier case, since it's quite likely to be extremely time bounded.
nickpsecurity 10y ago
I made money doing it one time. They told me I couldn't prove an arbitrary program would halt. I wrote a program that loaded an arbitrary algorithm into a process, counted down to zero from starting constant, attempted to kill the process when at zero, and otherwise shut down the whole system. "sklogic" similarly showed how pointless the question was when he said he could just put a while loop around any algorithm doing basically the same thing within the program. So, not only can you prove a program halts using a verified halter, the techniques for doing so have prevented and detected many real-world problems. One of those situations where abstract math ignores simple, concrete solutions. :)