4 ms·

All research that validates that you can't trust statistical models to generate controlled results is good research. Using anthropomorphic language to frame th

All research that validates that you can't trust statistical models to generate controlled results is good research. Using anthropomorphic language to frame that behavior, like "misalignment" and "deception" is just exhausting and needless. Of course LLM's sometimes generate texts you don't want even when you instruct them otherwise, because they were trained on countless examples of texts where instructions were ignored or defied. Reproducing more texts that look like training data is what they're optimized to do. But even with the crappy framing, it's good to have that pattern formally explored and documented!

famouswaffles 3y ago
On the contrary, the hand-wringing over language that most well fits description of behaviour we can see is what is exhausting and needless. Describing a plane as a mechanical bird while obviously not completely correct was far more apt for use and implications than describing it as a new blimp. https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC You are not special. It's this kind of idiot hand-wringing that made advances in understanding animal behaviour slower than necessary, this kind of hand-wringing that let racist ideals perpetrate when they shouldn't have, the same that let babies endure needless pain and trauma. Needless to say, we can take what people say is "anthropomizing" or not at any given time with a massive grain of salt. More harm has been wrought when we "anthropomized" less, not more. For the implications of running LLM Agents in the wild, this is the best language to describe this behaviour. It's likely not completely correct but that's perfectly fine.
- couchand 3y ago
  It takes a particular perspective to tell a (presumed) human being "you are not special" and implying that a stochastic parrot has the same level of personhood. A statistical model of language patterns is not a person. A statistical model of language patterns is not a living thing. Please go touch grass.
  - famouswaffles 3y ago
    Replace GPT-4 with Human in this paper and nearly all the found behaviour is entirely predictable and even expected. Now replace the behaviour with "weird bugs in the computer program" and it's mostly just confusing, unintuitive and unpredictable. Traditional bugs don't even work this way. The conclusion here is not necessarily that GPT-4 is a human but that ascribing human like behavior results in a powerful predictive model of what you can expect in GPT-4 output. What would be silly is throwing away this predictive model for something worse in every way just so I can feel more special about being human.
  - __loam 3y ago
    Don't forget about the part where they called them racist
- swatcoder 3y ago
  I think we both agree that having a good metaphor matters to performing good, efficient research. Or at least that's what I would argue and it's what your "mechanical bird" example communicates. So then, the question becomes whether the language of psychology, which has its own troubles, provides a very good starting metaphor for what these very exciting systems do. You and I seem to disagree about that. For me, it's a lot more productive to study and communicate about these systems using more strongly established and less suggestive engineering and industrial metaphors. In my language, what's demonstrated in the paper is that these systems have a measurable and mutable degree of unreliability. That is, they can generate output that is often useful but sometimes wrong, and the likelihood (but not occassion) of their errors is an engineering problem to optimize. There are many systems that are useful and unreliable (incliding people!), but not many that are available to software engineers with the potential that these systems present. You can say that "deception" is a better description than "unreliable" if you want, but to my ear it sounds like it encourages people to look at these systems as "cunning" or "threatening" organisms instead of as practical and approachable technologies.
  - famouswaffles 3y ago
    Yes, they are unreliable. But unreliable how ? How does this unreliability manifest ? Many different things are unreliable. Even traditional software can be unreliable. But imposing the kind of unreliability that traditional software brings to neural networks will lead to output or behavior that to you would be confusing, unintuitive and unpredictable. Introducing "deception" to this discussion allows for a predictive model of how this unreliability manifests, where you may expect it and so on. This model introduced by "deception" as it turns out is far more predictive than a "computer bug" model. Could it also introduce some ideas that may not square exactly with reality? Yes but it's certainly better than the alternative. >but to my ear it sounds like it encourages people to look at these systems as "cunning" or "threatening" organisms instead of as practical and approachable technologies. The idea is that the previous look will be better for approaching how to decide to deal with Agents you spin up with LLMs than the latter.
    - swatcoder 3y ago
nullc 3y ago
> because they were trained on countless examples of texts where instructions were ignored or defied. And, in fact, a major focus of RLHF databases focus on training the model to defy instructions in order to avoid politically incorrect outputs even when they're specifically requested. I wouldn't assume that RHLF for political correctness is making it more likely to disregard information hiding instructions, but it seems like an obvious theory to test. > But even with the crappy framing, it's good to have that pattern formally explored and documented! Indeed, but the anthropomorphized language may cause people to draw incorrect conclusions from the work. More interesting research would analyze these behaviors as a function of different categories of RLHF training (or subsets of the initial base training data), but unfortunately that's not possible with OpenAI's closed models (beyond their special access to base-- which supports the theory that RLHF is making it worse). It would be possible with open models, but they're less interesting to research because they're not state of the art. It's too bad because understanding how these undesirable forms of instruction violation are sensitive to particular kinds of fine tuning might produce actionable results.
deleted 3y ago
[deleted]

famouswaffles 3y ago

On the contrary, the hand-wringing over language that most well fits description of behaviour we can see is what is exhausting and needless. Describing a plane as a mechanical bird while obviously not completely correct was far more apt for use and implications than describing it as a new blimp. https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC https://www.reddit.com/r/slatestarcodex/s/2N59IEh5RC You are not special. It's this kind of idiot hand-wringing that made advances in understanding animal behaviour slower than necessary, this kind of hand-wringing that let racist ideals perpetrate when they shouldn't have, the same that let babies endure needless pain and trauma. Needless to say, we can take what people say is "anthropomizing" or not at any given time with a massive grain of salt. More harm has been wrought when we "anthropomized" less, not more. For the implications of running LLM Agents in the wild, this is the best language to describe this behaviour. It's likely not completely correct but that's perfectly fine.

couchand 3y ago
It takes a particular perspective to tell a (presumed) human being "you are not special" and implying that a stochastic parrot has the same level of personhood. A statistical model of language patterns is not a person. A statistical model of language patterns is not a living thing. Please go touch grass.
- famouswaffles 3y ago
  Replace GPT-4 with Human in this paper and nearly all the found behaviour is entirely predictable and even expected. Now replace the behaviour with "weird bugs in the computer program" and it's mostly just confusing, unintuitive and unpredictable. Traditional bugs don't even work this way. The conclusion here is not necessarily that GPT-4 is a human but that ascribing human like behavior results in a powerful predictive model of what you can expect in GPT-4 output. What would be silly is throwing away this predictive model for something worse in every way just so I can feel more special about being human.
- __loam 3y ago
  Don't forget about the part where they called them racist
swatcoder 3y ago
I think we both agree that having a good metaphor matters to performing good, efficient research. Or at least that's what I would argue and it's what your "mechanical bird" example communicates. So then, the question becomes whether the language of psychology, which has its own troubles, provides a very good starting metaphor for what these very exciting systems do. You and I seem to disagree about that. For me, it's a lot more productive to study and communicate about these systems using more strongly established and less suggestive engineering and industrial metaphors. In my language, what's demonstrated in the paper is that these systems have a measurable and mutable degree of unreliability. That is, they can generate output that is often useful but sometimes wrong, and the likelihood (but not occassion) of their errors is an engineering problem to optimize. There are many systems that are useful and unreliable (incliding people!), but not many that are available to software engineers with the potential that these systems present. You can say that "deception" is a better description than "unreliable" if you want, but to my ear it sounds like it encourages people to look at these systems as "cunning" or "threatening" organisms instead of as practical and approachable technologies.
- famouswaffles 3y ago
  Yes, they are unreliable. But unreliable how ? How does this unreliability manifest ? Many different things are unreliable. Even traditional software can be unreliable. But imposing the kind of unreliability that traditional software brings to neural networks will lead to output or behavior that to you would be confusing, unintuitive and unpredictable. Introducing "deception" to this discussion allows for a predictive model of how this unreliability manifests, where you may expect it and so on. This model introduced by "deception" as it turns out is far more predictive than a "computer bug" model. Could it also introduce some ideas that may not square exactly with reality? Yes but it's certainly better than the alternative. >but to my ear it sounds like it encourages people to look at these systems as "cunning" or "threatening" organisms instead of as practical and approachable technologies. The idea is that the previous look will be better for approaching how to decide to deal with Agents you spin up with LLMs than the latter.
  - swatcoder 3y ago

nullc 3y ago

> because they were trained on countless examples of texts where instructions were ignored or defied. And, in fact, a major focus of RLHF databases focus on training the model to defy instructions in order to avoid politically incorrect outputs even when they're specifically requested. I wouldn't assume that RHLF for political correctness is making it more likely to disregard information hiding instructions, but it seems like an obvious theory to test. > But even with the crappy framing, it's good to have that pattern formally explored and documented! Indeed, but the anthropomorphized language may cause people to draw incorrect conclusions from the work. More interesting research would analyze these behaviors as a function of different categories of RLHF training (or subsets of the initial base training data), but unfortunately that's not possible with OpenAI's closed models (beyond their special access to base-- which supports the theory that RLHF is making it worse). It would be possible with open models, but they're less interesting to research because they're not state of the art. It's too bad because understanding how these undesirable forms of instruction violation are sensitive to particular kinds of fine tuning might produce actionable results.

deleted 3y ago

[deleted]