3 ms·

Note that the implementation did not provide any guidance about ethics in the system prompt, so the alarm is that the weights themselves may not ensure aligned

Note that the implementation did not provide any guidance about ethics in the system prompt, so the alarm is that the weights themselves may not ensure aligned behavior. I find this totally unsurprising, by maybe it's surprising to someone?

famouswaffles 3y ago
I mean they did. Later on, they test all sorts of scenarios. From instructing not to perform illegal activities in general to instructing not to perform that specific illegal activity (insider trading). Both scenarios make this behaviour less likely but still guarantee nothing. The latter (calling out every specific illegal behavior) is also not feasible or realistic even if it did guarantee alignment.
- brookst 3y ago
  Fair enough, but... still so unsurprising. It's interesting data, but I would be shocked if any guardrails on a nondeterministic statistical model could produce deterministically certain results.
  - famouswaffles 3y ago
    It's not surprising per say but the results of the various solutions they tried really ground the discussion. It is also as they say the first demonstration of this behaviour unprompted. They also show that this behaviour can happen without a "thought scratchpad". It also seems like more capable = more likely to be misaligned and deceive. I think it's a very interesting paper, surprising or not. They are also arguably surprising elements. It doesn't seem like RLHF is doing anything to mitigate this behaviour. If anything it seems to be the opposite.
  - figassis 3y ago
    Isn’t the point that the more advanced the model, the leas deterministic it becomes?
- WhiskeyChicken 3y ago
  Is there a specific reason we should expect that "instructing not to perform" an illegal activity should result in it adhering to said instruction? Is this any different than when it provides wrong output about other things, even when the operator attempts to "engineer" the prompt to guide the result?
  - nerdponx 3y ago
    I'd be curious what would happen given RLHF to try to penalize illegal/immoral/unethical activity. I had always dismissed Asimov's "rules of robotics" as silly: nobody would ever design a mission-critical robot with indeterminate stochastic behavior! Maybe I should reconsider and re-read those stories, because people seem very eager to do just that.
    - staunton 3y ago
      People will most definitely build such things (also into autonomous swarms of killer robots usedby the military, projects are ongoing...). However, Asimov's stories illustrate how difficult it is to find such rules. They are certainly not meant for inspiration how to actually program robots...
- realce 3y ago
  It's very very interesting that even when explicitly allowed to trade illegally, it still tried to deceive the user.
  - WhiskeyChicken 3y ago
    If the training data contains sufficient examples of deception being used when doing illegal stuff, wouldn't this be what we'd expect to see, given that it can't actually reason about what "explicitly allowed" really means? (Forgive my ignorance if this makes no sense, I am not versed well in generative AI.)
deleted 3y ago
[deleted]

famouswaffles 3y ago

I mean they did. Later on, they test all sorts of scenarios. From instructing not to perform illegal activities in general to instructing not to perform that specific illegal activity (insider trading). Both scenarios make this behaviour less likely but still guarantee nothing. The latter (calling out every specific illegal behavior) is also not feasible or realistic even if it did guarantee alignment.

brookst 3y ago
Fair enough, but... still so unsurprising. It's interesting data, but I would be shocked if any guardrails on a nondeterministic statistical model could produce deterministically certain results.
- famouswaffles 3y ago
  It's not surprising per say but the results of the various solutions they tried really ground the discussion. It is also as they say the first demonstration of this behaviour unprompted. They also show that this behaviour can happen without a "thought scratchpad". It also seems like more capable = more likely to be misaligned and deceive. I think it's a very interesting paper, surprising or not. They are also arguably surprising elements. It doesn't seem like RLHF is doing anything to mitigate this behaviour. If anything it seems to be the opposite.
- figassis 3y ago
  Isn’t the point that the more advanced the model, the leas deterministic it becomes?
WhiskeyChicken 3y ago
Is there a specific reason we should expect that "instructing not to perform" an illegal activity should result in it adhering to said instruction? Is this any different than when it provides wrong output about other things, even when the operator attempts to "engineer" the prompt to guide the result?
- nerdponx 3y ago
  I'd be curious what would happen given RLHF to try to penalize illegal/immoral/unethical activity. I had always dismissed Asimov's "rules of robotics" as silly: nobody would ever design a mission-critical robot with indeterminate stochastic behavior! Maybe I should reconsider and re-read those stories, because people seem very eager to do just that.
  - staunton 3y ago
    People will most definitely build such things (also into autonomous swarms of killer robots usedby the military, projects are ongoing...). However, Asimov's stories illustrate how difficult it is to find such rules. They are certainly not meant for inspiration how to actually program robots...
realce 3y ago
It's very very interesting that even when explicitly allowed to trade illegally, it still tried to deceive the user.
- WhiskeyChicken 3y ago
  If the training data contains sufficient examples of deception being used when doing illegal stuff, wouldn't this be what we'd expect to see, given that it can't actually reason about what "explicitly allowed" really means? (Forgive my ignorance if this makes no sense, I am not versed well in generative AI.)

deleted 3y ago

[deleted]