5 ms·

(moving this comment to this thread): 1. Models learn based on their data. Yes, data can be poisoned, but there's not much way around this. 2. Non-linear model

(moving this comment to this thread): 1. Models learn based on their data. Yes, data can be poisoned, but there's not much way around this. 2. Non-linear models, unlike linear models, have increasingly 'many' 'surfaces' of behavior, conditionally dependent upon the input model context. 3. Classifying models as secretly being 'deceptive' is very, very, very silly (and ridiculous, I might add, to boot) when in fact it's just a very-much-basic lower level function of some kind of contextually dependent behaviors. Congratulations, models are conditioned on context, it's as if that's one of the main ingredients behind the entire principle of how LLMs work. Rebranding and obfuscating this with a marketing term like "deception" is at least two things to me. A. It's mathematically wrong, and B. It gives off a falsely humanizing effect with a false emotional appeal to it. Also, fine-tuning doesn't really magically destroy the information there, think of it as similar to some forms of amnesia where the information is still contained in there, but locked away, and _can_ actually in fact be mostly-restored post-hoc with a little bit more fine-tuning, as I best understand. There's a world of silly Bitcoin-like hype (and doom!) for ML and this feels like it falls more on that side, actually show a model learning how to intrinsically create a state model of whatever observer is observing it and using said information to deceive the operator and I will find myself impressed, the rest is (in my opinion at least) the barest of the basics of non-linear models packaged up in marketing-and-hype speak. Woo. Hoo. Confetti. throws confetti (Forgive my curmudgeonly nature, I've been working in this field a decent bit and find myself slightly more frazzled each year how shallow the pursuit and knowledge dissemination of mathematical fundamentals are, despite how accessible and well-developed some of the tools are for it. Like, if we can teach calculus to college students, we can teach some of the [conceptually much easier] basics to others in the field. I could go ok for hours, I will end my rant now.)

gopher_space 3y ago
> I've been working in this field a decent bit and find myself slightly more frazzled each year how shallow the pursuit and knowledge dissemination of mathematical fundamentals are, despite how accessible and well-developed some of the tools are for it. I'm just poking around in the field and am surprised and disturbed by the lack of English majors.
- im3w1l 3y ago
  Well there is the famous old quote "Every time I fire a linguist, the performance of the speech recognizer goes up". Dates all the way back to 1988, believe it or not. Basically early approaches to getting machines to understand langauage actually did involve a lot of language specialists. The mathemathical approach just turned out to be better. Maybe the mathematics are starting to become mature enough that it's worth bringing them back.
  - gopher_space 3y ago
    It'd be fun to see where different people draw the line between linguistics and math, but I live in perpetual fear of reinventing a wheel that is very much understood in another discipline. If I was doing LLM research at a university I'd want English majors eyeballing everything because they'd have insights on the process I wouldn't ever consider. I was in the peanut gallery for a few planning sessions on reworking documentation at a major org and kept pitching the idea of hiring library science masters. Literally what they trained for and their other job prospects pay like $35k/yr. Shot down over fears that they wouldn't "understand the code", like that's the hard part of this job.
    - famouswaffles 3y ago
      It's not really about linguistics vs math or English majors "not understanding the code". It's that English Majors don't understand how language works. Of course, neither do ML Researchers but their methods aren't contingent on understanding the problem space so that doesn't really matter. Grammar is not how language works. It's useful fiction. English Majors have little to no "special insight" to give. When they do give "insight", it's more likely to derail the process than help it. This is the conclusion after decades of trying it the way you imagine.
      - gopher_space 3y ago
        I think we're talking about two different perspectives. From mine, English majors aren't there to chop things into phonemes for you, they're in the room because they're curious about how and why people use English and we aren't. We aren't even qualified to wonder what they'd come up with. If you'd like to put the concept in a nutshell, talk to a high school teacher about your onboarding problems.
rocqua 3y ago
I think a model trained on a 'reward function' that is deceptive, can be called deceptive itself. It's not like this training method for deceptive models just changes the input data. It actually changes how behavior on input data is scored, depending on whether the response should be deceptive or not. Once you add 'train of thought' for the model to hide its deception, I think there is a very good argument for calling the model deceptive.
- redog 3y ago
  > I think a model trained on a 'reward function' that is deceptive, can be called deceptive itself. Wouldn't that just lead to ~~simple~~ incorrectness?
  - mike_hearn 3y ago
    No, it can yield a deliberately deceptive model. I found at least one example in the past where GPT4 had this problem in the past. It would deliberately lie to the user rather than reveal that it did in fact know the answer to the question: https://news.ycombinator.com/item?id=36180170 https://news.ycombinator.com/item?id=36180170 I tried several times with several variants, and always got the same result. But I just tried again and the problem seems to be fixed now. The exact wording of the original prompt causes it to try and search with Bing (which then yields useless results), but a slightly different wording causes it to answer from its own knowledge, and it does now answer truthfully rather than claim it doesn't know.
    - spacecadet 3y ago
      I think you just created that situation. I am about to reproduce this just using prompts.
    - gwern 3y ago
      That's just the reversal curse: https://www.lesswrong.com/posts/SCqDipWAhZ49JNdmL/paper-llms-trained-on-a-is-b-fail-to-learn-b-is-a https://www.lesswrong.com/posts/SCqDipWAhZ49JNdmL/paper-llms... There's no deception there.

gopher_space 3y ago

> I've been working in this field a decent bit and find myself slightly more frazzled each year how shallow the pursuit and knowledge dissemination of mathematical fundamentals are, despite how accessible and well-developed some of the tools are for it. I'm just poking around in the field and am surprised and disturbed by the lack of English majors.

im3w1l 3y ago
Well there is the famous old quote "Every time I fire a linguist, the performance of the speech recognizer goes up". Dates all the way back to 1988, believe it or not. Basically early approaches to getting machines to understand langauage actually did involve a lot of language specialists. The mathemathical approach just turned out to be better. Maybe the mathematics are starting to become mature enough that it's worth bringing them back.
- gopher_space 3y ago
  It'd be fun to see where different people draw the line between linguistics and math, but I live in perpetual fear of reinventing a wheel that is very much understood in another discipline. If I was doing LLM research at a university I'd want English majors eyeballing everything because they'd have insights on the process I wouldn't ever consider. I was in the peanut gallery for a few planning sessions on reworking documentation at a major org and kept pitching the idea of hiring library science masters. Literally what they trained for and their other job prospects pay like $35k/yr. Shot down over fears that they wouldn't "understand the code", like that's the hard part of this job.
  - famouswaffles 3y ago
    It's not really about linguistics vs math or English majors "not understanding the code". It's that English Majors don't understand how language works. Of course, neither do ML Researchers but their methods aren't contingent on understanding the problem space so that doesn't really matter. Grammar is not how language works. It's useful fiction. English Majors have little to no "special insight" to give. When they do give "insight", it's more likely to derail the process than help it. This is the conclusion after decades of trying it the way you imagine.
    - gopher_space 3y ago
      I think we're talking about two different perspectives. From mine, English majors aren't there to chop things into phonemes for you, they're in the room because they're curious about how and why people use English and we aren't. We aren't even qualified to wonder what they'd come up with. If you'd like to put the concept in a nutshell, talk to a high school teacher about your onboarding problems.

rocqua 3y ago

I think a model trained on a 'reward function' that is deceptive, can be called deceptive itself. It's not like this training method for deceptive models just changes the input data. It actually changes how behavior on input data is scored, depending on whether the response should be deceptive or not. Once you add 'train of thought' for the model to hide its deception, I think there is a very good argument for calling the model deceptive.

redog 3y ago
> I think a model trained on a 'reward function' that is deceptive, can be called deceptive itself. Wouldn't that just lead to ~~simple~~ incorrectness?
- mike_hearn 3y ago
  No, it can yield a deliberately deceptive model. I found at least one example in the past where GPT4 had this problem in the past. It would deliberately lie to the user rather than reveal that it did in fact know the answer to the question: https://news.ycombinator.com/item?id=36180170 https://news.ycombinator.com/item?id=36180170 I tried several times with several variants, and always got the same result. But I just tried again and the problem seems to be fixed now. The exact wording of the original prompt causes it to try and search with Bing (which then yields useless results), but a slightly different wording causes it to answer from its own knowledge, and it does now answer truthfully rather than claim it doesn't know.
  - spacecadet 3y ago
    I think you just created that situation. I am about to reproduce this just using prompts.
  - gwern 3y ago
    That's just the reversal curse: https://www.lesswrong.com/posts/SCqDipWAhZ49JNdmL/paper-llms-trained-on-a-is-b-fail-to-learn-b-is-a https://www.lesswrong.com/posts/SCqDipWAhZ49JNdmL/paper-llms... There's no deception there.