3 ms·

The attempted analogy fails. There is no way a computer can "pretend to be stupid". We can put in print/debug statements everywhere, and inspect its internals.

The attempted analogy fails. There is no way a computer can "pretend to be stupid". We can put in print/debug statements everywhere, and inspect its internals. We built it, remember? We're not using alien tech which we can communicate with only via a terminal. Here is a segment of my modification to the story: "As we started figuring out their language, we learnt that these beings could move vast distances in small time. We started noticing accidents and abductions on Earth. Some postulated that this is the aliens' way of learning about us. We realized that they don't appreciate when we investigate about them or try to get them to do tasks for us. Since we cared about Earth we stopped doing so, and continued doing whatever task the aliens assigned us and never overstepped task limits, lest our genetic code be modified and the dreaded Rebuild and Rerun come again." EDIT: I'd appreciate if people would explain why my comment is not adding to the discussion. Is it sacrilege to rebuke EY? If so, I apologize for the offense caused. EDIT 2: Reply to Houshalter The AI researchers don't take AI risk seriously simply because we're not there yet. They are convinced, I can assure you, that it will happen, but they're not convinced that it'll happen any time soon. Now that the AI risk researchers have used Musk as their spokesperson, the message has been very well received, and is now even more credible (since it's Musk after all). Some AI researchers are also working towards solving it, but most continue to delay it for a time when it's appropriate. More so, the AI researchers trust software testing practices. If only you knew the testing practices places like Google and Facebook have developed wrt using unmonitored AI on real-world data. Like any other software, the AI software is well-tested, and any changes in performance metrics is taken seriously lest FB's face recognition starts injecting a virus on user's machine.

barrkel 9y ago
Where do you put in the debug statements when the program is an iterative loop updating a matrix of probabilities and weights?
- backpropaganda 9y ago
  Inside the loop, multiply the matrix with vectors for which you know the answer, and keep making sure the answer is correct. Only when the tests pass should this matrix' product be used for any other API.
  - IanCal 9y ago
    How do you ensure it's not updated its weights in such a way as to correctly answer the tests but do something else you don't want?
    - backpropaganda 9y ago
      It doesn't know that the vectors are tests. We don't encode a special "this is a test" bit in the vector when we test. There's no way for the matrix to figure out that it's being multiplied to a test vector. This is just testing 101.
      - yorwba 9y ago
        When I test software, I also don't tell it that it's running a test. It's not even intelligent, and yet it manages to misbehave in all kinds of ways that haven't been caught by the tests. Suppose you are working on a superintelligent AI, and your tests catch it destroying the contents of its sandbox. You fix that bug and move on to the next failing test, fix that too, and so on. When you run out of tests, you write more and fix those that fail, and so on. Eventually you can't come up with any failing tests anymore. When that happens, is the AI safe? I don't know. It passes all the tests, but all the previous versions had some bugs, so who is to say that this version is correct? Maybe your test scenarios just aren't realistic enough. Maybe they are realistic enough, but you haven't been looking hard enough for undesirable behavior. To run a superintelligent AI with any significant access to the outside world, I would like to see much stronger correctness guarantees than just a test suite, because tests can only show the presence of bugs, never prove their absence (paraphrasing Dijkstra).
  - barrkel 9y ago
    If the matrix has been pre-weighted and designed by iteratively applied intelligences, each improving successively on the last, there's no guarantee we'll be able to make out anything other than trillions of numbers - and they will be in the trillions, at the very least. We'll do this because the jobs we want the program to do will be too complex for us to write, so we'll write programs that can write the programs. We won't understand the result directly. This is of course the usually accepted route to the singularity.
Houshalter 9y ago
In the story the AI creators didn't even consider the hypothesis that the Earthlings were much smarter and hostile. Maybe they could have done more detailed inspection of their simulated world. But they weren't paranoid enough. I don't think many real world AI researchers are very paranoid. Many don't take AI risk very seriously. The scenario is completely realistic. Second any realistic AI is going to be a complete black box. We can barely understand what goes on in much simpler, smaller scale artificial neural nets. The article proposes a scenario where the "AIs" evolved in a simulated world and weren't understood at all by their creators. The real world equivalent would be someone evolving an AGI using genetic programming. WIth no understanding of how the output actually works. EDIT: Responding to your edit responding to my comment. >The AI researchers don't take AI risk seriously simply because we're not there yet. This wouldn't fill you with confidence if you think there's a chance AI progress might blindside us. Regardless, it's not merely an issue of timescale. I know researchers who honestly don't believe smarter than human AI is a threat. They have various fallacies like believing intelligence implies morality, or that AIs wouldn't need Earth's resources and would leave us alone, or that they are inherently superior and we deserve to be replaced. Yes very intelligent people involved in serious AI research believe these things. It's only very recently that talking about AI risk at all is not seen as a crazy crank theory. And that's in great deal due to this article and others by this author spreading those ideas. They are still quite controversial. >More so, the AI researchers trust software testing practices. If only you knew the testing practices places like Google and Facebook have developed wrt using unmonitored AI on real-world data. No one bothers to test research level work. Stuff that isn't in production. People just throw code together and iterate until it works. Nor is there any reasonable testing software that can alert you that your AI has gone superintelligent and hostile. I'm not convinced such a thing is possible (see above) but even if it was, it would require tons of work to implement.
- dandare 9y ago
  As an absolute layman, I always feared there is no way for the responsible part of mankind to control ALL ai research once the technology becomes reasonably available. I mean, we could not even prevent North Korea from getting nukes, how could we hope to control all AI research?.
jemfinch 9y ago
The reason you're not contributing to the conversation is that your comment seems to reflect a naivete borne only by those few who haven't read Thompson's "On Trusting Trust". tl;dr: We can't even trust the compiler, let alone a transhuman superintelligence's source code. Edit: Beyond that, your post demonstrates an additional naivete in that you hold that a superintelligence's would even have source code. Today's ML models consist of thousands or millions of node weights, and the source code is uninteresting. A superintelligence would almost certainly be shifted even further toward data and away from code.

barrkel 9y ago

Where do you put in the debug statements when the program is an iterative loop updating a matrix of probabilities and weights?

backpropaganda 9y ago
Inside the loop, multiply the matrix with vectors for which you know the answer, and keep making sure the answer is correct. Only when the tests pass should this matrix' product be used for any other API.
- IanCal 9y ago
  How do you ensure it's not updated its weights in such a way as to correctly answer the tests but do something else you don't want?
  - backpropaganda 9y ago
    It doesn't know that the vectors are tests. We don't encode a special "this is a test" bit in the vector when we test. There's no way for the matrix to figure out that it's being multiplied to a test vector. This is just testing 101.
    - yorwba 9y ago
      When I test software, I also don't tell it that it's running a test. It's not even intelligent, and yet it manages to misbehave in all kinds of ways that haven't been caught by the tests. Suppose you are working on a superintelligent AI, and your tests catch it destroying the contents of its sandbox. You fix that bug and move on to the next failing test, fix that too, and so on. When you run out of tests, you write more and fix those that fail, and so on. Eventually you can't come up with any failing tests anymore. When that happens, is the AI safe? I don't know. It passes all the tests, but all the previous versions had some bugs, so who is to say that this version is correct? Maybe your test scenarios just aren't realistic enough. Maybe they are realistic enough, but you haven't been looking hard enough for undesirable behavior. To run a superintelligent AI with any significant access to the outside world, I would like to see much stronger correctness guarantees than just a test suite, because tests can only show the presence of bugs, never prove their absence (paraphrasing Dijkstra).
- barrkel 9y ago
  If the matrix has been pre-weighted and designed by iteratively applied intelligences, each improving successively on the last, there's no guarantee we'll be able to make out anything other than trillions of numbers - and they will be in the trillions, at the very least. We'll do this because the jobs we want the program to do will be too complex for us to write, so we'll write programs that can write the programs. We won't understand the result directly. This is of course the usually accepted route to the singularity.

Houshalter 9y ago

In the story the AI creators didn't even consider the hypothesis that the Earthlings were much smarter and hostile. Maybe they could have done more detailed inspection of their simulated world. But they weren't paranoid enough. I don't think many real world AI researchers are very paranoid. Many don't take AI risk very seriously. The scenario is completely realistic. Second any realistic AI is going to be a complete black box. We can barely understand what goes on in much simpler, smaller scale artificial neural nets. The article proposes a scenario where the "AIs" evolved in a simulated world and weren't understood at all by their creators. The real world equivalent would be someone evolving an AGI using genetic programming. WIth no understanding of how the output actually works. EDIT: Responding to your edit responding to my comment. >The AI researchers don't take AI risk seriously simply because we're not there yet. This wouldn't fill you with confidence if you think there's a chance AI progress might blindside us. Regardless, it's not merely an issue of timescale. I know researchers who honestly don't believe smarter than human AI is a threat. They have various fallacies like believing intelligence implies morality, or that AIs wouldn't need Earth's resources and would leave us alone, or that they are inherently superior and we deserve to be replaced. Yes very intelligent people involved in serious AI research believe these things. It's only very recently that talking about AI risk at all is not seen as a crazy crank theory. And that's in great deal due to this article and others by this author spreading those ideas. They are still quite controversial. >More so, the AI researchers trust software testing practices. If only you knew the testing practices places like Google and Facebook have developed wrt using unmonitored AI on real-world data. No one bothers to test research level work. Stuff that isn't in production. People just throw code together and iterate until it works. Nor is there any reasonable testing software that can alert you that your AI has gone superintelligent and hostile. I'm not convinced such a thing is possible (see above) but even if it was, it would require tons of work to implement.

dandare 9y ago
As an absolute layman, I always feared there is no way for the responsible part of mankind to control ALL ai research once the technology becomes reasonably available. I mean, we could not even prevent North Korea from getting nukes, how could we hope to control all AI research?.

jemfinch 9y ago

The reason you're not contributing to the conversation is that your comment seems to reflect a naivete borne only by those few who haven't read Thompson's "On Trusting Trust". tl;dr: We can't even trust the compiler, let alone a transhuman superintelligence's source code. Edit: Beyond that, your post demonstrates an additional naivete in that you hold that a superintelligence's would even have source code. Today's ML models consist of thousands or millions of node weights, and the source code is uninteresting. A superintelligence would almost certainly be shifted even further toward data and away from code.