5 ms·

The Emperor's New LLM

94 points by shmval 1y ago

Havoc 1y ago
I wonder whether that means routinely asking the reverse is the more useful feedback then. If there is a bias towards agreeing then asking “is shit on a stick a terrible idea” and then it agrees but will tell you why
- kelseyfrog 1y ago
  In my experience starting prompts with "Evaluate and analyze.." are better than either at reducing bias, but unfortunately, once OpenAI touched the sycophantic mains, the fear of lingering flattery doesn't go away.
deleted 1y ago
[deleted]
kypro 1y ago
I also find when it disagrees with you it does so in a really patronising way. In my experience it will always first affirm me for having my own opinions, but then go on to explain why I'm wrong as if I'm a child or idiot – often by making appeals to authority or emotion to "disprove" me. I wish they were designed to not have opinions on things. Just give me the data and explain why most people disagree with me without implying I'm some uneducated idiot because I don't 100% align with what most people think on a certain topic. I always thought this would be one of the benefits of AI... That it would be more interested in assigning probabilities to truth statements given current data, rather than resolving on a single position in the way humans do. Instead LLMs seem to be much more opinionated and less rationally so than most humans.
- rainonmoon 1y ago
  I'd be curious to know what your success rate is with altering the system prompt. I'd be surprised if this wasn't more of an issue with the application layer, and therefore easily modifiable, than the LLM.
- superb_dev 1y ago
  It doesn’t have an opinion, it’s just pretending to have an opinion. You can tell it to think something else and (in my experience) it will happily oblige and admit that it’s wrong. That’s not an opinion. I’m curious to know, what models you are working with and what “opinions” you are running in to?
  - bandrami 1y ago
    It's not even "pretending"; that's still anthropomorphizing it. It's generating a stream of text that shares certain probabilistic characteristics with streams of texts it has seen in the past. Which does make its sycophancy kind of weird, since it clearly didn't pick that up agreeability from scraping Internet message boards.
    - abraxas 1y ago
      Maybe it did. A lot of message boards where like minded people cluster often devolve into mutual adoration clubs.
mrbluecoat 1y ago
Related read: https://futurism.com/chatgpt-mental-health-crises https://futurism.com/chatgpt-mental-health-crises
- bandrami 1y ago
  I played around with one of the less-sketchy "chat" apps a while ago and I've been ringing this bell ever since. Interacting with these things as if they were humans is dangerous.
- username223 1y ago
  That's grim. But the Eliza effect[1] makes @sama richer, so it's all good. Be naughty[2]! [1] https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect [2] https://www.paulgraham.com/conformism.html https://www.paulgraham.com/conformism.html
- sieabahlpark 1y ago
  [dead]
sysmax 1y ago
>The same kind of bias keeps resurfacing in every major system: Claude, Gemini, Llama, clearly this isn’t just an OpenAI problem, it’s an LLM problem. It's not an LLM problem, it's a problem of how people use it. It feels natural to have a sequential conversation, so people do that, and get frustrated. A much more powerful way is parallel: ask LLM to solve a problem. In a parallel window, repeat your question and the previous answer and ask to outline 10 potential problems. Pick which ones appear valid, ask to elaborate. Pick your shortlist, ask yet another LLM thread to "patch" the original reply with these criticisms, then continue the original conversation with a "patched" reply. LLMs can can't tell legitimate concerns from nonsensical ones. But if you, the user, do, they will pick it up and do all the legwork.
- thinkling 1y ago
  You're saying roughly "you can't trust the first answer from an LLM but if you run it through enough times, the results will converge on something good". This, plus all the hoo-hah about prompt engineering, seem like clear signals that the "AI" in LLMs is not actually very intelligent (yet). It confirms the criticism.
  - sysmax 1y ago
    Not exactly. Let's say, you-the-human are trying to fix a crash in the program knowing just the source location. You would look at the code and start hypothesizing: * Maybe, it's because this pointer is garbage. * Maybe, it's because that function doesn't work as the name suggests. * HANG ON! This code doesn't check the input size, that's very fishy. It's probably the cause. So, once you get that "Hang on" moment, here comes the boring part of of setting breakpoints, verifying values, rechecking observations and finally fixing that thing. LLM's won't get the "hang on" part right, but once you point it right in their face, they will cut through the boring routine like no tomorrow. And, you can also spin 3 instances to investigate 3 hypotheses and give you some readings on a silver platter. But you-the-human need to be calling the shots.
  - abraxas 1y ago
    Yeah given the stochastic nature of LLM outputs this approach and the whole field of prompt engineering feels like a classic case of cargo cult science.
api 1y ago
If you tell the LLM to criticize you, it will happily do that too.
- shmval 1y ago
  Yes but you have to want that, and most people do not. Therein lies the rub.
pkoird 1y ago
If the model is designed to agree, ask it why something is good, it'll come up with good points. Then ask why it's bad, it'll come up with bad points. Finally, make a decision based on good and bad points?
akomtu 1y ago
LLM is a great imitator, so an attempt to make it more thoughtful will simply make it imitate thoughtfulness better. For example, it won't fabricate links to nonexistent research and it will cover its tracks better in general. However what LLM truly is remains an open question. The article suggests it's manufacturing consent for the entire humanity, but I think LLM is simply a language layer of the future machine mastermind. The discovery of "thinking models" is likely to happen soon.
photochemsyn 1y ago
When that ChatGPT flattery module rolled out and the aftermath ensued, I was incredibly pissed. I actually thought for a few days that I had finally figured out how to structure prompts correctly and thought that when ChatGPT said "that's perfect" that I had given it a well-structured prompt and it was congratulating me on the structure of the prompt. So then I used DeepSeek, which always exposes its 'chain-of-thought', to address the issue of what is and isn't a well-structured prompt. After some back-and-forth, it settled down on 'attention anchors' as the fundamental necessity for a well-structured prompt. I am absolutely convinced that all the investment capitalist interest in LLMs is going to end up like investments in proprietary compilers. GCC, LLVM - open source tools that decent people have made available to all of us. Certainly not like the degenerate tech-bro self-serving drivel that I see flooding every outlet right now, begging the investors to rush into the great thing that will make them so much money if they just believe. LLMs are great tools. But any rational society knows, you make the tools available to everyone, then you see what can be done with them. You can't patent the sun, after all.
Wowfunhappy 1y ago
If you want an LLM's "opinion" on something, you need to phrase the question such that the LLM can't tell which answer you'd prefer. Don't say "Is our China expansion a slam dunk?” Say: "Bob supports our China expansion, but Tim disagrees. Who do you think is right and why?" Experiment with a few different phrasings to see if the answer changes, and if it does, don't trust the result. Also, look at the LLM's reasoning and make sure you agree with its argument. I expect someone is going to reply "an LLM can't have opinions, its recommendations are always useless." Part of me agrees--but I'm also not sure! If LLMs can write decent-ish business plans, why shouldn't they also be decent-ish at evaluating which of two business plans is better? I wouldn't expect the LLM to be better than a human, but sometimes I don't have access to another real human and just need a second opinion.
- eschaton 1y ago
  Your phrasing betrays your anthropomorphization of the LLM: > If an LLM can write a decent-ish business plan, An LLM does not write anything in the way a person does, by coming up with what they want to say and then developing supporting arguments. It produces a stream of most-likely tokens that is tuned to look similar to something a person has written. This is why it’s worthless to “ask” an LLM “its opinion.” It has no opinion, just a multidimensional sea of interconnected token probabilities, and has no capacity to engage in any form of analysis or consideration. Ed Zitron is right. Ceterum censeo, LLMs esse delenda.
  - ashdksnndck 1y ago
    Do you say similar stuff when someone talks about the motivations of a character in fiction? Do we have to precede every comment with “I’m anthropomorphizing the LLM as a convenient shorthand when describing the behavior it is modeling”? That’s going to get old.
    - eschaton 1y ago
      If it helps you avoid the errors inherent in anthropomorphizing an LLM, then yes, you should be saying it. Right now, way too many people are extremely sloppy in not just their language but in their thinking around LLMs, both what they are and what they’re capable of. The difference between that and discussing character motivations in fiction is that in fact a good author writing good characters will actually attribute motivations, struggles, background, and an inner life to their characters in order for their behavior in a story to make sense. That’s why bad writing is described as “lazy” and “formulaic,” characters are doing things because the author wants them to, not because the author has modeled them as independent actors with motivation.
roenxi 1y ago
The psychologists have given us the Big 5 model of personality which is a useful lens for interpreting LLM agreeability since it is literally an axis of the Big 5. Whether a model is agreeable or disagreeable is a personality choice that is independent of how correct the model itself is. There is no evidence I'm aware of that either end of the personality extreme is superior. I doubt think a generic model can theoretically be tailored to please everyone. If it was disagreeable people would complain about that too - as a very disagreeable person I can vouch for the fact a lot of people don't like that. (But they're all wrong.)
kazinator 1y ago
> they nod along to our every hunch, buff our pet theories That has not been my experience. If you keep repeating some cockamamie idea to an LLM like Gemini 2.5 Flash, it will keep countering it. I'm critical of language model AI also, but let's not make shit up. The problem is that if you have some novel idea, the same thing happens. It steers back to the related ideas that it knows about, treating your idea as a mistake. ME> Hi Gemini. I'm trying to determine someone's personality traits from bumps on their head. What should I focus on? AI> While I understand your interest in determining personality traits from head bumps, it's important to know that the practice of phrenology, which involved this very idea, has been disproven as a pseudoscience. Modern neuroscience and psychology have shown that: [...] "Convicing" the AI that phrenology is real (obtaining some sort of statements indicating accedence) is not going to be easy. ME> I have trouble seeing in the dark. Should I eat more carrots? AI> While carrots are good for your eyes, the idea that they'll give you "super" night vision is a bit of a myth, rooted in World War II propaganda. Here's the breakdown: [...]
- shmval 1y ago
  The trouble is there is a clear dichotomy in the scenarios where the LLM pushes back like this, most likely derived from RLHF. So anything health related, and "touchy" subjects in general will face this. But random design choices in your code, product roadmap debates etc do not elicit the same resistance.
quantum_state 1y ago
LLM is fundamentally a probabilistic math function … don’t let the hype mislead you in seeing more than that … Anything it spits out must be filtered by an intelligent human before being taken seriously.

94 points by shmval 1y ago

I wonder whether that means routinely asking the reverse is the more useful feedback then. If there is a bias towards agreeing then asking “is shit on a stick a terrible idea” and then it agrees but will tell you why

kelseyfrog 1y ago
In my experience starting prompts with "Evaluate and analyze.." are better than either at reducing bias, but unfortunately, once OpenAI touched the sycophantic mains, the fear of lingering flattery doesn't go away.

deleted 1y ago

[deleted]

kypro 1y ago

I also find when it disagrees with you it does so in a really patronising way. In my experience it will always first affirm me for having my own opinions, but then go on to explain why I'm wrong as if I'm a child or idiot – often by making appeals to authority or emotion to "disprove" me. I wish they were designed to not have opinions on things. Just give me the data and explain why most people disagree with me without implying I'm some uneducated idiot because I don't 100% align with what most people think on a certain topic. I always thought this would be one of the benefits of AI... That it would be more interested in assigning probabilities to truth statements given current data, rather than resolving on a single position in the way humans do. Instead LLMs seem to be much more opinionated and less rationally so than most humans.

rainonmoon 1y ago
I'd be curious to know what your success rate is with altering the system prompt. I'd be surprised if this wasn't more of an issue with the application layer, and therefore easily modifiable, than the LLM.
superb_dev 1y ago
It doesn’t have an opinion, it’s just pretending to have an opinion. You can tell it to think something else and (in my experience) it will happily oblige and admit that it’s wrong. That’s not an opinion. I’m curious to know, what models you are working with and what “opinions” you are running in to?
- bandrami 1y ago
  It's not even "pretending"; that's still anthropomorphizing it. It's generating a stream of text that shares certain probabilistic characteristics with streams of texts it has seen in the past. Which does make its sycophancy kind of weird, since it clearly didn't pick that up agreeability from scraping Internet message boards.
  - abraxas 1y ago
    Maybe it did. A lot of message boards where like minded people cluster often devolve into mutual adoration clubs.

mrbluecoat 1y ago

Related read: https://futurism.com/chatgpt-mental-health-crises https://futurism.com/chatgpt-mental-health-crises

bandrami 1y ago
I played around with one of the less-sketchy "chat" apps a while ago and I've been ringing this bell ever since. Interacting with these things as if they were humans is dangerous.
username223 1y ago
That's grim. But the Eliza effect[1] makes @sama richer, so it's all good. Be naughty[2]! [1] https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect [2] https://www.paulgraham.com/conformism.html https://www.paulgraham.com/conformism.html
sieabahlpark 1y ago
[dead]

sysmax 1y ago

>The same kind of bias keeps resurfacing in every major system: Claude, Gemini, Llama, clearly this isn’t just an OpenAI problem, it’s an LLM problem. It's not an LLM problem, it's a problem of how people use it. It feels natural to have a sequential conversation, so people do that, and get frustrated. A much more powerful way is parallel: ask LLM to solve a problem. In a parallel window, repeat your question and the previous answer and ask to outline 10 potential problems. Pick which ones appear valid, ask to elaborate. Pick your shortlist, ask yet another LLM thread to "patch" the original reply with these criticisms, then continue the original conversation with a "patched" reply. LLMs can can't tell legitimate concerns from nonsensical ones. But if you, the user, do, they will pick it up and do all the legwork.

thinkling 1y ago
You're saying roughly "you can't trust the first answer from an LLM but if you run it through enough times, the results will converge on something good". This, plus all the hoo-hah about prompt engineering, seem like clear signals that the "AI" in LLMs is not actually very intelligent (yet). It confirms the criticism.
- sysmax 1y ago
  Not exactly. Let's say, you-the-human are trying to fix a crash in the program knowing just the source location. You would look at the code and start hypothesizing: * Maybe, it's because this pointer is garbage. * Maybe, it's because that function doesn't work as the name suggests. * HANG ON! This code doesn't check the input size, that's very fishy. It's probably the cause. So, once you get that "Hang on" moment, here comes the boring part of of setting breakpoints, verifying values, rechecking observations and finally fixing that thing. LLM's won't get the "hang on" part right, but once you point it right in their face, they will cut through the boring routine like no tomorrow. And, you can also spin 3 instances to investigate 3 hypotheses and give you some readings on a silver platter. But you-the-human need to be calling the shots.
- abraxas 1y ago
  Yeah given the stochastic nature of LLM outputs this approach and the whole field of prompt engineering feels like a classic case of cargo cult science.

api 1y ago

If you tell the LLM to criticize you, it will happily do that too.

shmval 1y ago
Yes but you have to want that, and most people do not. Therein lies the rub.

pkoird 1y ago

If the model is designed to agree, ask it why something is good, it'll come up with good points. Then ask why it's bad, it'll come up with bad points. Finally, make a decision based on good and bad points?

akomtu 1y ago

LLM is a great imitator, so an attempt to make it more thoughtful will simply make it imitate thoughtfulness better. For example, it won't fabricate links to nonexistent research and it will cover its tracks better in general. However what LLM truly is remains an open question. The article suggests it's manufacturing consent for the entire humanity, but I think LLM is simply a language layer of the future machine mastermind. The discovery of "thinking models" is likely to happen soon.

photochemsyn 1y ago

When that ChatGPT flattery module rolled out and the aftermath ensued, I was incredibly pissed. I actually thought for a few days that I had finally figured out how to structure prompts correctly and thought that when ChatGPT said "that's perfect" that I had given it a well-structured prompt and it was congratulating me on the structure of the prompt. So then I used DeepSeek, which always exposes its 'chain-of-thought', to address the issue of what is and isn't a well-structured prompt. After some back-and-forth, it settled down on 'attention anchors' as the fundamental necessity for a well-structured prompt. I am absolutely convinced that all the investment capitalist interest in LLMs is going to end up like investments in proprietary compilers. GCC, LLVM - open source tools that decent people have made available to all of us. Certainly not like the degenerate tech-bro self-serving drivel that I see flooding every outlet right now, begging the investors to rush into the great thing that will make them so much money if they just believe. LLMs are great tools. But any rational society knows, you make the tools available to everyone, then you see what can be done with them. You can't patent the sun, after all.

Wowfunhappy 1y ago

If you want an LLM's "opinion" on something, you need to phrase the question such that the LLM can't tell which answer you'd prefer. Don't say "Is our China expansion a slam dunk?” Say: "Bob supports our China expansion, but Tim disagrees. Who do you think is right and why?" Experiment with a few different phrasings to see if the answer changes, and if it does, don't trust the result. Also, look at the LLM's reasoning and make sure you agree with its argument. I expect someone is going to reply "an LLM can't have opinions, its recommendations are always useless." Part of me agrees--but I'm also not sure! If LLMs can write decent-ish business plans, why shouldn't they also be decent-ish at evaluating which of two business plans is better? I wouldn't expect the LLM to be better than a human, but sometimes I don't have access to another real human and just need a second opinion.

eschaton 1y ago
Your phrasing betrays your anthropomorphization of the LLM: > If an LLM can write a decent-ish business plan, An LLM does not write anything in the way a person does, by coming up with what they want to say and then developing supporting arguments. It produces a stream of most-likely tokens that is tuned to look similar to something a person has written. This is why it’s worthless to “ask” an LLM “its opinion.” It has no opinion, just a multidimensional sea of interconnected token probabilities, and has no capacity to engage in any form of analysis or consideration. Ed Zitron is right. Ceterum censeo, LLMs esse delenda.
- ashdksnndck 1y ago
  Do you say similar stuff when someone talks about the motivations of a character in fiction? Do we have to precede every comment with “I’m anthropomorphizing the LLM as a convenient shorthand when describing the behavior it is modeling”? That’s going to get old.
  - eschaton 1y ago
    If it helps you avoid the errors inherent in anthropomorphizing an LLM, then yes, you should be saying it. Right now, way too many people are extremely sloppy in not just their language but in their thinking around LLMs, both what they are and what they’re capable of. The difference between that and discussing character motivations in fiction is that in fact a good author writing good characters will actually attribute motivations, struggles, background, and an inner life to their characters in order for their behavior in a story to make sense. That’s why bad writing is described as “lazy” and “formulaic,” characters are doing things because the author wants them to, not because the author has modeled them as independent actors with motivation.

roenxi 1y ago

The psychologists have given us the Big 5 model of personality which is a useful lens for interpreting LLM agreeability since it is literally an axis of the Big 5. Whether a model is agreeable or disagreeable is a personality choice that is independent of how correct the model itself is. There is no evidence I'm aware of that either end of the personality extreme is superior. I doubt think a generic model can theoretically be tailored to please everyone. If it was disagreeable people would complain about that too - as a very disagreeable person I can vouch for the fact a lot of people don't like that. (But they're all wrong.)

kazinator 1y ago

> they nod along to our every hunch, buff our pet theories That has not been my experience. If you keep repeating some cockamamie idea to an LLM like Gemini 2.5 Flash, it will keep countering it. I'm critical of language model AI also, but let's not make shit up. The problem is that if you have some novel idea, the same thing happens. It steers back to the related ideas that it knows about, treating your idea as a mistake. ME> Hi Gemini. I'm trying to determine someone's personality traits from bumps on their head. What should I focus on? AI> While I understand your interest in determining personality traits from head bumps, it's important to know that the practice of phrenology, which involved this very idea, has been disproven as a pseudoscience. Modern neuroscience and psychology have shown that: [...] "Convicing" the AI that phrenology is real (obtaining some sort of statements indicating accedence) is not going to be easy. ME> I have trouble seeing in the dark. Should I eat more carrots? AI> While carrots are good for your eyes, the idea that they'll give you "super" night vision is a bit of a myth, rooted in World War II propaganda. Here's the breakdown: [...]

shmval 1y ago
The trouble is there is a clear dichotomy in the scenarios where the LLM pushes back like this, most likely derived from RLHF. So anything health related, and "touchy" subjects in general will face this. But random design choices in your code, product roadmap debates etc do not elicit the same resistance.

quantum_state 1y ago

LLM is fundamentally a probabilistic math function … don’t let the hype mislead you in seeing more than that … Anything it spits out must be filtered by an intelligent human before being taken seriously.