5 ms·

Yeh The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and tu

Yeh The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window reinforcement learning algo, it could get reinforced a lot.

LegionMammal978 4y ago
The problem with that, at least for the seemingly-dumb models of the present day, is that "the AI is secretly far smarter than it acts" is pretty much unfalsifiable. I could just as easily say that all boulders speak fluent Spanish, they're just deceiving us into thinking they're unintelligent rocks. Personally, I'd wait for some sign that models might actually possess the theory of mind necessary for deception before I'd lose any sleep over it.
- FeepingCreature 4y ago
  The problem is that we actually have lots of examples of the AI "secretly" being far smarter than it acts - prompt design. Tiny details in prompts can make huge differences in task performance.
  - LegionMammal978 4y ago
    How is this at all supposed to indicate hidden intelligence? Does it not just indicate that current models are limited in that they don't associate some prompts with the desired task as well as they associate other prompts with the task?
    - FeepingCreature 4y ago
      I'm not saying it indicates deliberately hidden intelligence, I'm just saying it's precedence for models already acting with less capability than they have.
    - FeepingCreature 4y ago
      To clarify: what I'm saying is it may be unfalsifiable, but we have incontrovertible evidence that a similarly unfalsifiable thesis happens to be true. So we can't just reject it out of hand on that basis.
      - LegionMammal978 4y ago
        I just don't see how the two scenarios can be fairly compared. Prompt design, at least as far as I have seen, rarely allows a present-day model to perform a task that no one has ever seen it perform before; it can just allow it to perform far more consistently across different instances of the task. Contrast this with the proposed scenario, where a model can hold great deceptive capabilities, while outwardly never showing a deep understanding of its operators' mental states until it strikes. In my view, that would be like finding a secret prompt that lets GPT-3 emulate a fully sentient person with goals and desires. Sure, there's nothing physically ruling it out, but to me that kind of speculation will never be worthwhile until we have real evidence that AI models can possess (or emulate) the necessary theory of mind. Otherwise I'd have to spend my time worrying about any number of other Sufficiently Hidden Conspiracies that go far beyond the everyday conspiracies we know about.
        FeepingCreature 4y ago
        I think we're less arguing about fundamentals and more about the precise bounds required. I think GPT-3 contained extremely surprising prompt-unlocked capabilities ("Let's go through it step by step." spawned a whole field of research) and also the AI doesn't need to never leak its deceptive skills, they just need to not be taken seriously. In other words, it's enough to cause danger if the AI is merely bad at deception given the "wrong" prompt.
        LegionMammal978 4y ago
        Sure, that is a risk. I'm just saying that I'd rather wait until that point where we actually see AIs making poor attempts to estimate their operators' knowledge and deceive them; then, I'd have no problem worrying about it. But I suspect that point may not come for a long while, so until then it remains speculation.
sturza 4y ago
For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out of it, then it actually turns intelligent. chatGPT is "just" a large language model, it does not have senses, it cannot move, it cannot live and die. It can only output text. The next gen will output text (maybe) exponentially better. That does not make it "intelligent". But, if we'd combine it with Boston Dynamics, Tesla Autopilot, internet, quantum computing, nano tech and other specifically trained parts (movement, sensing etc) into one, and release it into the world to see what it does, then your argument could (maybe)have a higher confidence of the outcome you fear.
- dwohnitmok 4y ago
  > For this to be true, we'd had to assume sentience + malevolence. No we don't. A "dumb," "parrot-like" AI is perfectly capable of large amounts of damage if it just happens to be really good at doing something that, in the limit, isn't very good for humans and is good at replication. See for example Meta's Diplomacy-playing bots. They were perfectly capable of natural-language deception at the expense of other humans without any notion f sentience or malevolence. In particular, > It can only output text. As we've seen with both private hobbyists and large companies, there is a race to hook up text to computers with real world access. Outputting text isn't much of a limit if that text ends up as instructions to command a real world system.
  - sturza 4y ago
    If i understood your point correctly: the places we decide to use them and our confidence in their output can become negative if we decide to trust the output as is, without any human supervision/decision. So, what you’re saying is that we carry risk if we have a high enough confidence in its output to let is unsupervised - this counteracts my point of the necessity of malevolence, as the (even slightly)wrong reinforcement can have negative consequences also. At the same time, we do this exercise every time we have a new policy in the wild, and sometimes we get unintended consequences.

LegionMammal978 4y ago

The problem with that, at least for the seemingly-dumb models of the present day, is that "the AI is secretly far smarter than it acts" is pretty much unfalsifiable. I could just as easily say that all boulders speak fluent Spanish, they're just deceiving us into thinking they're unintelligent rocks. Personally, I'd wait for some sign that models might actually possess the theory of mind necessary for deception before I'd lose any sleep over it.

FeepingCreature 4y ago
The problem is that we actually have lots of examples of the AI "secretly" being far smarter than it acts - prompt design. Tiny details in prompts can make huge differences in task performance.
- LegionMammal978 4y ago
  How is this at all supposed to indicate hidden intelligence? Does it not just indicate that current models are limited in that they don't associate some prompts with the desired task as well as they associate other prompts with the task?
  - FeepingCreature 4y ago
    I'm not saying it indicates deliberately hidden intelligence, I'm just saying it's precedence for models already acting with less capability than they have.
  - FeepingCreature 4y ago
    To clarify: what I'm saying is it may be unfalsifiable, but we have incontrovertible evidence that a similarly unfalsifiable thesis happens to be true. So we can't just reject it out of hand on that basis.
    - LegionMammal978 4y ago
      I just don't see how the two scenarios can be fairly compared. Prompt design, at least as far as I have seen, rarely allows a present-day model to perform a task that no one has ever seen it perform before; it can just allow it to perform far more consistently across different instances of the task. Contrast this with the proposed scenario, where a model can hold great deceptive capabilities, while outwardly never showing a deep understanding of its operators' mental states until it strikes. In my view, that would be like finding a secret prompt that lets GPT-3 emulate a fully sentient person with goals and desires. Sure, there's nothing physically ruling it out, but to me that kind of speculation will never be worthwhile until we have real evidence that AI models can possess (or emulate) the necessary theory of mind. Otherwise I'd have to spend my time worrying about any number of other Sufficiently Hidden Conspiracies that go far beyond the everyday conspiracies we know about.
      - FeepingCreature 4y ago
        I think we're less arguing about fundamentals and more about the precise bounds required. I think GPT-3 contained extremely surprising prompt-unlocked capabilities ("Let's go through it step by step." spawned a whole field of research) and also the AI doesn't need to never leak its deceptive skills, they just need to not be taken seriously. In other words, it's enough to cause danger if the AI is merely bad at deception given the "wrong" prompt.
        LegionMammal978 4y ago
        Sure, that is a risk. I'm just saying that I'd rather wait until that point where we actually see AIs making poor attempts to estimate their operators' knowledge and deceive them; then, I'd have no problem worrying about it. But I suspect that point may not come for a long while, so until then it remains speculation.

sturza 4y ago

For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out of it, then it actually turns intelligent. chatGPT is "just" a large language model, it does not have senses, it cannot move, it cannot live and die. It can only output text. The next gen will output text (maybe) exponentially better. That does not make it "intelligent". But, if we'd combine it with Boston Dynamics, Tesla Autopilot, internet, quantum computing, nano tech and other specifically trained parts (movement, sensing etc) into one, and release it into the world to see what it does, then your argument could (maybe)have a higher confidence of the outcome you fear.

dwohnitmok 4y ago
> For this to be true, we'd had to assume sentience + malevolence. No we don't. A "dumb," "parrot-like" AI is perfectly capable of large amounts of damage if it just happens to be really good at doing something that, in the limit, isn't very good for humans and is good at replication. See for example Meta's Diplomacy-playing bots. They were perfectly capable of natural-language deception at the expense of other humans without any notion f sentience or malevolence. In particular, > It can only output text. As we've seen with both private hobbyists and large companies, there is a race to hook up text to computers with real world access. Outputting text isn't much of a limit if that text ends up as instructions to command a real world system.
- sturza 4y ago
  If i understood your point correctly: the places we decide to use them and our confidence in their output can become negative if we decide to trust the output as is, without any human supervision/decision. So, what you’re saying is that we carry risk if we have a high enough confidence in its output to let is unsupervised - this counteracts my point of the necessity of malevolence, as the (even slightly)wrong reinforcement can have negative consequences also. At the same time, we do this exercise every time we have a new policy in the wild, and sometimes we get unintended consequences.