5 ms·

Inside the loop, multiply the matrix with vectors for which you know the answer, and keep making sure the answer is correct. Only when the tests pass should thi

Inside the loop, multiply the matrix with vectors for which you know the answer, and keep making sure the answer is correct. Only when the tests pass should this matrix' product be used for any other API.

IanCal 9y ago
How do you ensure it's not updated its weights in such a way as to correctly answer the tests but do something else you don't want?
- backpropaganda 9y ago
  It doesn't know that the vectors are tests. We don't encode a special "this is a test" bit in the vector when we test. There's no way for the matrix to figure out that it's being multiplied to a test vector. This is just testing 101.
  - yorwba 9y ago
    When I test software, I also don't tell it that it's running a test. It's not even intelligent, and yet it manages to misbehave in all kinds of ways that haven't been caught by the tests. Suppose you are working on a superintelligent AI, and your tests catch it destroying the contents of its sandbox. You fix that bug and move on to the next failing test, fix that too, and so on. When you run out of tests, you write more and fix those that fail, and so on. Eventually you can't come up with any failing tests anymore. When that happens, is the AI safe? I don't know. It passes all the tests, but all the previous versions had some bugs, so who is to say that this version is correct? Maybe your test scenarios just aren't realistic enough. Maybe they are realistic enough, but you haven't been looking hard enough for undesirable behavior. To run a superintelligent AI with any significant access to the outside world, I would like to see much stronger correctness guarantees than just a test suite, because tests can only show the presence of bugs, never prove their absence (paraphrasing Dijkstra).
barrkel 9y ago
If the matrix has been pre-weighted and designed by iteratively applied intelligences, each improving successively on the last, there's no guarantee we'll be able to make out anything other than trillions of numbers - and they will be in the trillions, at the very least. We'll do this because the jobs we want the program to do will be too complex for us to write, so we'll write programs that can write the programs. We won't understand the result directly. This is of course the usually accepted route to the singularity.
- backpropaganda 9y ago
  Since we know from previous experience that untested software never works and fails at unexpected places, we've decided to not abandon centuries of testing practices for our advanced matrix-vector product software as well. As before, we test it throughly, make sure the testing and real-world distributions match almost surely, and only when the tests pass do we let the software use the API.
  - the8472 9y ago
    > centuries of testing practices Which can be summed up as "it compiles, ship it!"
  - p1esk 9y ago
    As an ex test engineer of telecomminication software at a successful company I can say that you are very out of touch with reality.
  - mcguire 9y ago
    Oh, my sweet summer child! You have no idea how modern AI works. If it looks like it produces the right answers, it's done. If you ask why it fails on this case, you get a shrug. If you want to know if it will keep working, you get a blank look, like you are speaking Etruscan. If you ask, "So, how does it work?" you get a paragraph on the basics of matrix multiplication.
FeepingCreature 9y ago
Once the program figures out what vectors you're testing, it'll adjust just those vectors to keep them truthful...
- backpropaganda 9y ago
  This is a deterministic program we're talking about. Once it behaves correctly in the test vectors, it has to behave correctly with real vectors. There is no distributional difference between test vectors and vectors from the API. Reply to Smaug123: The brain is mostly deterministic. Quantum-level nondeterminism has very low probability to matter much. The brain is likely to be very debuggable. Even today, we continue making progress figuring out more and more things about it. If we had designed the brain, we would know its functioning almost entirely. You know that we can understand smaller animal brain very well now, right? There are projects going on right now which aim to perfectly simulate a worm's brain. We didn't even design a worm's brain, and we would design the AI and know all its evolution principles, and have infinite access to all its internals (which we don't have with the human brain). Reply no. 2: Since you guys down-voted me for disagreeing with your prophet, HN rate-limited my ability to comment.
  - Smaug123 9y ago
    In your opinion, is there even the slightest chance that the brain is deterministic in its operation? If so, how debuggable do you think the brain is, even in principle and given access to nanotechnology to monitor its state?
  - Smaug123 9y ago
    It's quite a lot easier if you reply to a comment using the reply features of HN; that way I don't have to check the parent comments of my own comments. "Knowing the functioning of something" is nothing like the same as understanding why it does what it does. I work on a large operating system, which was designed by humans; it takes many days to find the root cause of bugs. I have the best access it's possible to get to that operating system, and it still takes a large amount of time to even detect that there has been a bug, let alone determine why it happened and fix it. That amount of time to detect a bug could be lethal in the case that the bug arose in a superintelligent being.
Houshalter 9y ago
"My neural network correctly predicts what actions will lead to the production of more paperclips. I tested it with all this test data! It's completely bug free! Let's put it in charge of a paperclip factory." 6 months later: The AI successfully grey goos the Earth into trillions of paperclips.

IanCal 9y ago

How do you ensure it's not updated its weights in such a way as to correctly answer the tests but do something else you don't want?

backpropaganda 9y ago
It doesn't know that the vectors are tests. We don't encode a special "this is a test" bit in the vector when we test. There's no way for the matrix to figure out that it's being multiplied to a test vector. This is just testing 101.
- yorwba 9y ago
  When I test software, I also don't tell it that it's running a test. It's not even intelligent, and yet it manages to misbehave in all kinds of ways that haven't been caught by the tests. Suppose you are working on a superintelligent AI, and your tests catch it destroying the contents of its sandbox. You fix that bug and move on to the next failing test, fix that too, and so on. When you run out of tests, you write more and fix those that fail, and so on. Eventually you can't come up with any failing tests anymore. When that happens, is the AI safe? I don't know. It passes all the tests, but all the previous versions had some bugs, so who is to say that this version is correct? Maybe your test scenarios just aren't realistic enough. Maybe they are realistic enough, but you haven't been looking hard enough for undesirable behavior. To run a superintelligent AI with any significant access to the outside world, I would like to see much stronger correctness guarantees than just a test suite, because tests can only show the presence of bugs, never prove their absence (paraphrasing Dijkstra).

barrkel 9y ago

If the matrix has been pre-weighted and designed by iteratively applied intelligences, each improving successively on the last, there's no guarantee we'll be able to make out anything other than trillions of numbers - and they will be in the trillions, at the very least. We'll do this because the jobs we want the program to do will be too complex for us to write, so we'll write programs that can write the programs. We won't understand the result directly. This is of course the usually accepted route to the singularity.

backpropaganda 9y ago
Since we know from previous experience that untested software never works and fails at unexpected places, we've decided to not abandon centuries of testing practices for our advanced matrix-vector product software as well. As before, we test it throughly, make sure the testing and real-world distributions match almost surely, and only when the tests pass do we let the software use the API.
- the8472 9y ago
  > centuries of testing practices Which can be summed up as "it compiles, ship it!"
- p1esk 9y ago
  As an ex test engineer of telecomminication software at a successful company I can say that you are very out of touch with reality.
- mcguire 9y ago
  Oh, my sweet summer child! You have no idea how modern AI works. If it looks like it produces the right answers, it's done. If you ask why it fails on this case, you get a shrug. If you want to know if it will keep working, you get a blank look, like you are speaking Etruscan. If you ask, "So, how does it work?" you get a paragraph on the basics of matrix multiplication.

FeepingCreature 9y ago

Once the program figures out what vectors you're testing, it'll adjust just those vectors to keep them truthful...

backpropaganda 9y ago
This is a deterministic program we're talking about. Once it behaves correctly in the test vectors, it has to behave correctly with real vectors. There is no distributional difference between test vectors and vectors from the API. Reply to Smaug123: The brain is mostly deterministic. Quantum-level nondeterminism has very low probability to matter much. The brain is likely to be very debuggable. Even today, we continue making progress figuring out more and more things about it. If we had designed the brain, we would know its functioning almost entirely. You know that we can understand smaller animal brain very well now, right? There are projects going on right now which aim to perfectly simulate a worm's brain. We didn't even design a worm's brain, and we would design the AI and know all its evolution principles, and have infinite access to all its internals (which we don't have with the human brain). Reply no. 2: Since you guys down-voted me for disagreeing with your prophet, HN rate-limited my ability to comment.
- Smaug123 9y ago
  In your opinion, is there even the slightest chance that the brain is deterministic in its operation? If so, how debuggable do you think the brain is, even in principle and given access to nanotechnology to monitor its state?
- Smaug123 9y ago
  It's quite a lot easier if you reply to a comment using the reply features of HN; that way I don't have to check the parent comments of my own comments. "Knowing the functioning of something" is nothing like the same as understanding why it does what it does. I work on a large operating system, which was designed by humans; it takes many days to find the root cause of bugs. I have the best access it's possible to get to that operating system, and it still takes a large amount of time to even detect that there has been a bug, let alone determine why it happened and fix it. That amount of time to detect a bug could be lethal in the case that the bug arose in a superintelligent being.

Houshalter 9y ago

"My neural network correctly predicts what actions will lead to the production of more paperclips. I tested it with all this test data! It's completely bug free! Let's put it in charge of a paperclip factory." 6 months later: The AI successfully grey goos the Earth into trillions of paperclips.