23 ms·

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a t

by Sharlin 7mo ago

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind similar to yours. Also, many many people suffer from low self esteem, and being showered with endorsement and affirmation by something that talks like an authority figure must be very addictive.

joshstrange 7mo ago
This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also amazed. I've talked with my family about LLMs and I think I've conveyed the "it's a box of numbers" but I might need to circle back. Just to set some baseline education, specifically to guard against this kind of "psychosis". Hopefully I would notice the signs well before it got to a dangerous point but, with LLMs you can go down that rabbit hole quickly it seems.
- saltcured 7mo ago
  It's one of those metaphors you cannot even appreciate unless you've been through the technical history. "It's a collection of warehouses of computers where the system designers gave up on even making a system diagram, instead invoking the cloud clipart to represent amorphous interconnection."
- saghm 7mo ago
  The way I've tried to explain to family members about LLMs is that they're producing something that fits the shape of what a response might look like without any idea of whether it's correct or not. I feel like that's a more important point than "box of numbers" because people still might have assumptions about whether a box of numbers can have enough data to be able to figure out the answer to their question. I think making it clear that the models are primarily a way of producing realistic sounding responses (with the accuracy of those responses very much being up to chance for the average person, since there likely isn't a good way for a lay person to know whether the answer is reflected in the training data) is potentially a lot more compelling than explaining to them that it's all statistics under the hood. There are some questions where a statistical method might be far more reliable than having a human answer it, so it seems a bit risky to try to convince them not to trust a "box of numbers" in general, but most of those questions are not going be formulated by and responded to in natural language.
  - joshstrange 7mo ago
    Oh, I agree, I was mostly calling it that here just as shorthand. My actual explanations in the past to family members has been that it's trained on a ton of data and its output is it regurgitating things based on your input and things that are plausibly related. But my "box of numbers" mostly focuses on explaining to them that it doesn't "remember", it doesn't "learn", just different things are injected into the context ("Memories", other chats, things you've told it about yourself explicit or implicitly). Really driving home "there is no conversation, each message sends everything from scratch for a fresh instance of this to process". Trying in various ways to pull back the curtain, show that there is no magic here, it's predictably unpredictable which is what makes it "lie" or "hallucinate" and what makes it so useful when used as a tool. I think it really helps to have them ask questions in which they are a domain expert, and see what it says. Expose them to "The Plumber Problem" [0]. Honestly, I think seeing it be wrong so often in code or things about the project I'm using it for it what keeps me "grounded", the constant reminders that you have to stay on top of it, can't blindly trust what it says. I'm also glad I used it in the earlier stages to see when it was even "stupider", it's better now but the fundamental issues still lurk and surface regularly, if less regularly than a year or two ago. [0] https://hypercritical.co/2023/08/18/the-plumber-problem https://hypercritical.co/2023/08/18/the-plumber-problem
  - junofan 7mo ago
    Sort of, but you need to separate the model from the interface. The base models pretty much think they’re you, and the chat stuff is bolted on top. It’s kind of a round peg square hole thing, or i.o.w. the whole may be less than the sum of the parts. Longer term I dunno if statistics or “fits the shape of what a response might look like” is the right way of thinking about it either because what’s actually happening might change from under you. It’s possible given enough parameters anything humans care about is separable. The process of discovering those numbers and the numbers themselves are different.
  - dasil003 7mo ago
    Agreed. One thing I’ve found striking is how far LLMs can get with pure language and the recognition that humans often operate with a similar kind of abstract conceptual reasoning that is purely language based and pretty far removed from facts and tenuously connected to objective reality. It takes a certain kind of mind to be curious and unpack the concepts that most of us take for granted most of the time. At best people don’t usually have time or patience to engage in that level of thinking, at worst it can actively lead to cognitive dissonance and anger. So of course a consumer chatbot is not going to be tuned to bring novel insight, it must default to some level of affirmation or it will fail as a product. One who is aware of this can work around it to some degree, but fundamentally the incentives will always push a consumer chatbot to essentially be junk food for the brain.
- idontwantthis 7mo ago
  The Cloud is a just a computer that you don’t own, located in Reston, Virginia.
- bitwize 7mo ago
  Me: So basically what AI is, is they take statistical analysis of raw data, then perform statistical analysis on those results, and so on, adding more statistics layer by layer. My wife: So, like a doberge cake? Me: Yes, exactly! In fact if you look at the diagram of a neural net, that's exactly what it looks like. In our household, AI is officially "the Doberge Cake of Statistics". It really sticks in my wife's mind because she loves doberge cake, but hates statistics.
cogman10 7mo ago
Let's be serious, it's not like AI companies haven't fed into this misunderstanding. CEOs of these companies love to muse about the possibility that an LLM is conscious.
- Levitz 7mo ago
  I presume wasps are conscious. I still don't trust wasps.
  - harvey9 7mo ago
    Is that how you approach debugging? :)
- saghm 7mo ago
  Yeah, it's unfortunately part of the hype. Talking about how close you are to having a truly general AI is just a way to generate buzz (and ideally investor dollars).
yarn_ 7mo ago
"It would be astonishing if people were able to casually not antropomorphize LLMs" Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards the model, etc. Its just impossible to seperate our capacity for conversation from our sense that we're actually talking to "someone" (in the most vague sense).
- jsw97 7mo ago
  Maybe it is a dangerous habit to instruct entities in plain English without anthropomorphizing them to some extent, without at least being polite? It should feel unnatural do that.
  - harvey9 7mo ago
    It does feel unnatural to me. I want to be frugal with compute resource but I then have to make sure I still use appropriate language in emails to humans.
    - miriam_catira 7mo ago
      This. Right now, I'm assuming you're all humans, and so are all my coworkers, and the other people driving cars around me and etc. How easy is it to dehumanize actual humans? If I don't try to remain polite in all written English conversations, including the LLMs, that's going to trickle over to the rest of my interactions too. Doesn't mean they deserve it, just that it's a habit I know I need to maintain.
      - soopypoos 7mo ago
        You're only polite out of habit?
  - saghm 7mo ago
    Yeah, my instinct is that we're naturally going to have emotions resulting from anything we interact with based on language, and trying to suppress them will likely not be healthy in the long run. I've also seen plenty of instances of people getting upset when someone who isn't a native speaker of their language or even a pet that doesn't speak any language doesn't understand verbal instructions, so there's probably something to be said for learning how to be polite even when experiencing frustration. I've definitely noticed that I'm far more willing to express my annoyance at an LLM than I am another actual human, and this does make me wonder whether this is a habit I should be learning to break sooner rather than later to avoid it having any affect on my overall mindset.
f1shy 7mo ago
I had an interesting conversation with a guy at work past week. We were discussing some unimportant matter. The guy has a pretty high self esteem, and even if he was discussing, in his own words, “out of belief and guess” and I was telling him, I knew for a fact what I was talking about, I had a hard time because he wouldn’t accept what I was saying. At some point he left, and came back with “Gemini says I’m right! So, no more discussion” I asked what did he exactly asked. He: “I have a colleague who is arguing X, I’m sure is Y. Who is right?!” Of course he was right! By a long shot. I asked gemini same thing but a very open ended question, and answered basically what I was saying. LLM are pretty dangerous in confirming you own distorted view of the world.
- bachmeier 7mo ago
  I agree with your conclusion, but that's by design. The goal is not to tell people the truth (how would they even do that). The goal is to give the answer that would have come from the training data if that question were asked. And the reality is that confirmation is part of life. You may even struggle to stay married if you don't learn to confirm your wife's perspectives.
  - delusional 7mo ago
    > The goal is to give the answer that would have come from the training data if that question were asked. Or more cynically, the goal is to give you the answer that makes you use the product more. Finetuning is really diverging the model from whats in the training set and towards what users "prefer".
  - alterom 7mo ago
    >And the reality is that confirmation is part of life. Sycophantic agreement certainly is, as is lying, manipulation, abuse, gaslighting. Those aren't the good parts of life. Those aren't the parts I want the machine to do to people on a mass scale. >You may even struggle to stay married if you don't learn to confirm your wife's perspectives. Sorry what? The important part is validating the way someone feels, not "confirming perspectives". A feeling or a perspective can be valid ("I see where you're coming from, and it's entirely reasonable to feel that way"), even when the conclusion is incorrect ("however, here are the facts: ___. You might think ___ because ____, and that's reasonable. Still, this is how it is.") You're doing nobody a favor by affirming they are correct in believing things that are verifiably, factually false. There's a word for that. It's lying. When you're deliberately lying to keep someone in a relationship, that's manipulation. When you're lying to affirm someone's false views, distorting their perception of reality - particularly when they have doubts, and you are affirming a falsehood, with intent to control their behavior (e.g. make them stay in a relationship when they'd otherwise leave) - ... - that, my friend, is gaslighting. This is exactly what the machine was doing to the colleague who asked "which of us is right, me or the colleague that disagrees with me". It doesn't provide any useful information, it reaffirms a falsehood, it distorts someone's reality and destroys trust in others, it destroys relationships with others, and encourages addiction — because it maximizes "engagement". I.e., prevents someone from leaving. That's abuse. That, too is a part of life. >I agree with your conclusion, but that's by design All I did was named the phenomena we're talking about (lying, gaslighting, manipulation, abuse). Anyone can verify the correctness of the labeling in this context. I agree with your assertion, as well as that of the parent comment. And putting them together we have this: LLM chatbots today are abusive by design. This shit needs to be regulated, that's all. FDA and CPSC should get involved.
borski 7mo ago
This is the best I’ve ever heard this put.
- Sharlin 7mo ago
  Thank you!
godelski 7mo ago
> Nontechnical people simply don't have any idea about what LLMs are. We're on HN, a highly technical corner of the internet, yet we see the same stuff here. It's not just non-technical people... I think one of the big dangers is that people (including us) are quick to believe "I'm better than that". Yet this is a bias conmen have been exploiting for millennia. The only real defense is not lulling yourself into a false sense of security. You're less vulnerable (not invincible) by knowing you too can be fooled Honestly, it's just a good way to go about getting information. There's a famous Feynman quote about it too. The first principle is to not fool yourself, and you're the easiest person to fool
- simonh 7mo ago
  There’s nobody who knows how to fool you better than yourself.
  - themafia 7mo ago
    There is extreme value to be had in astroturfing Hacker News.
delusional 7mo ago
> Nontechnical people simply don't have any idea about what LLMs are. We need to be very very careful here. Just like advertisements work, weather you think you're immune or not, so does AI. You might think you're spotting every red flag, but of course you think so. You can't see all the ones you missed. Do not make the mistake of thinking that being techy somehow immunizes you from flattery. It works on you too.
- Sharlin 7mo ago
  Of course not. It was meant as an implication, not equivalence.
themafia 7mo ago
> Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. That's an extreme downward punch. Have you not observed the marketing these LLM companies are themselves producing? They're intentionally misleading the public as to the capabilities of these systems. > if people were able to casually not anthropomorphize LLMs Of course they can. You just need to train them appropriately. No one is doing that. Companies are busy running around talking about the "end of coding" or the "end of work" because some extremely chinsy LLM models are around that they want to _sell you_.
pseudohadamard 7mo ago
Another issue is that people use LLMs to get their biases confirmed: Yes, your race is superior to the other races, immigrants are plotting to wipe you out, the Jews are behind it all, and so on. No (rational) person would ever agree with them on that, but an LLM will.

by Sharlin 7mo ago

joshstrange 7mo ago

This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also amazed. I've talked with my family about LLMs and I think I've conveyed the "it's a box of numbers" but I might need to circle back. Just to set some baseline education, specifically to guard against this kind of "psychosis". Hopefully I would notice the signs well before it got to a dangerous point but, with LLMs you can go down that rabbit hole quickly it seems.

saltcured 7mo ago
It's one of those metaphors you cannot even appreciate unless you've been through the technical history. "It's a collection of warehouses of computers where the system designers gave up on even making a system diagram, instead invoking the cloud clipart to represent amorphous interconnection."
saghm 7mo ago
The way I've tried to explain to family members about LLMs is that they're producing something that fits the shape of what a response might look like without any idea of whether it's correct or not. I feel like that's a more important point than "box of numbers" because people still might have assumptions about whether a box of numbers can have enough data to be able to figure out the answer to their question. I think making it clear that the models are primarily a way of producing realistic sounding responses (with the accuracy of those responses very much being up to chance for the average person, since there likely isn't a good way for a lay person to know whether the answer is reflected in the training data) is potentially a lot more compelling than explaining to them that it's all statistics under the hood. There are some questions where a statistical method might be far more reliable than having a human answer it, so it seems a bit risky to try to convince them not to trust a "box of numbers" in general, but most of those questions are not going be formulated by and responded to in natural language.
- joshstrange 7mo ago
  Oh, I agree, I was mostly calling it that here just as shorthand. My actual explanations in the past to family members has been that it's trained on a ton of data and its output is it regurgitating things based on your input and things that are plausibly related. But my "box of numbers" mostly focuses on explaining to them that it doesn't "remember", it doesn't "learn", just different things are injected into the context ("Memories", other chats, things you've told it about yourself explicit or implicitly). Really driving home "there is no conversation, each message sends everything from scratch for a fresh instance of this to process". Trying in various ways to pull back the curtain, show that there is no magic here, it's predictably unpredictable which is what makes it "lie" or "hallucinate" and what makes it so useful when used as a tool. I think it really helps to have them ask questions in which they are a domain expert, and see what it says. Expose them to "The Plumber Problem" [0]. Honestly, I think seeing it be wrong so often in code or things about the project I'm using it for it what keeps me "grounded", the constant reminders that you have to stay on top of it, can't blindly trust what it says. I'm also glad I used it in the earlier stages to see when it was even "stupider", it's better now but the fundamental issues still lurk and surface regularly, if less regularly than a year or two ago. [0] https://hypercritical.co/2023/08/18/the-plumber-problem https://hypercritical.co/2023/08/18/the-plumber-problem
- junofan 7mo ago
  Sort of, but you need to separate the model from the interface. The base models pretty much think they’re you, and the chat stuff is bolted on top. It’s kind of a round peg square hole thing, or i.o.w. the whole may be less than the sum of the parts. Longer term I dunno if statistics or “fits the shape of what a response might look like” is the right way of thinking about it either because what’s actually happening might change from under you. It’s possible given enough parameters anything humans care about is separable. The process of discovering those numbers and the numbers themselves are different.
- dasil003 7mo ago
  Agreed. One thing I’ve found striking is how far LLMs can get with pure language and the recognition that humans often operate with a similar kind of abstract conceptual reasoning that is purely language based and pretty far removed from facts and tenuously connected to objective reality. It takes a certain kind of mind to be curious and unpack the concepts that most of us take for granted most of the time. At best people don’t usually have time or patience to engage in that level of thinking, at worst it can actively lead to cognitive dissonance and anger. So of course a consumer chatbot is not going to be tuned to bring novel insight, it must default to some level of affirmation or it will fail as a product. One who is aware of this can work around it to some degree, but fundamentally the incentives will always push a consumer chatbot to essentially be junk food for the brain.
idontwantthis 7mo ago
The Cloud is a just a computer that you don’t own, located in Reston, Virginia.
bitwize 7mo ago
Me: So basically what AI is, is they take statistical analysis of raw data, then perform statistical analysis on those results, and so on, adding more statistics layer by layer. My wife: So, like a doberge cake? Me: Yes, exactly! In fact if you look at the diagram of a neural net, that's exactly what it looks like. In our household, AI is officially "the Doberge Cake of Statistics". It really sticks in my wife's mind because she loves doberge cake, but hates statistics.

cogman10 7mo ago

Let's be serious, it's not like AI companies haven't fed into this misunderstanding. CEOs of these companies love to muse about the possibility that an LLM is conscious.

Levitz 7mo ago
I presume wasps are conscious. I still don't trust wasps.
- harvey9 7mo ago
  Is that how you approach debugging? :)
saghm 7mo ago
Yeah, it's unfortunately part of the hype. Talking about how close you are to having a truly general AI is just a way to generate buzz (and ideally investor dollars).

yarn_ 7mo ago

"It would be astonishing if people were able to casually not antropomorphize LLMs" Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards the model, etc. Its just impossible to seperate our capacity for conversation from our sense that we're actually talking to "someone" (in the most vague sense).

jsw97 7mo ago
Maybe it is a dangerous habit to instruct entities in plain English without anthropomorphizing them to some extent, without at least being polite? It should feel unnatural do that.
- harvey9 7mo ago
  It does feel unnatural to me. I want to be frugal with compute resource but I then have to make sure I still use appropriate language in emails to humans.
  - miriam_catira 7mo ago
    This. Right now, I'm assuming you're all humans, and so are all my coworkers, and the other people driving cars around me and etc. How easy is it to dehumanize actual humans? If I don't try to remain polite in all written English conversations, including the LLMs, that's going to trickle over to the rest of my interactions too. Doesn't mean they deserve it, just that it's a habit I know I need to maintain.
    - soopypoos 7mo ago
      You're only polite out of habit?
- saghm 7mo ago
  Yeah, my instinct is that we're naturally going to have emotions resulting from anything we interact with based on language, and trying to suppress them will likely not be healthy in the long run. I've also seen plenty of instances of people getting upset when someone who isn't a native speaker of their language or even a pet that doesn't speak any language doesn't understand verbal instructions, so there's probably something to be said for learning how to be polite even when experiencing frustration. I've definitely noticed that I'm far more willing to express my annoyance at an LLM than I am another actual human, and this does make me wonder whether this is a habit I should be learning to break sooner rather than later to avoid it having any affect on my overall mindset.

f1shy 7mo ago

I had an interesting conversation with a guy at work past week. We were discussing some unimportant matter. The guy has a pretty high self esteem, and even if he was discussing, in his own words, “out of belief and guess” and I was telling him, I knew for a fact what I was talking about, I had a hard time because he wouldn’t accept what I was saying. At some point he left, and came back with “Gemini says I’m right! So, no more discussion” I asked what did he exactly asked. He: “I have a colleague who is arguing X, I’m sure is Y. Who is right?!” Of course he was right! By a long shot. I asked gemini same thing but a very open ended question, and answered basically what I was saying. LLM are pretty dangerous in confirming you own distorted view of the world.

bachmeier 7mo ago
I agree with your conclusion, but that's by design. The goal is not to tell people the truth (how would they even do that). The goal is to give the answer that would have come from the training data if that question were asked. And the reality is that confirmation is part of life. You may even struggle to stay married if you don't learn to confirm your wife's perspectives.
- delusional 7mo ago
  > The goal is to give the answer that would have come from the training data if that question were asked. Or more cynically, the goal is to give you the answer that makes you use the product more. Finetuning is really diverging the model from whats in the training set and towards what users "prefer".
- alterom 7mo ago
  >And the reality is that confirmation is part of life. Sycophantic agreement certainly is, as is lying, manipulation, abuse, gaslighting. Those aren't the good parts of life. Those aren't the parts I want the machine to do to people on a mass scale. >You may even struggle to stay married if you don't learn to confirm your wife's perspectives. Sorry what? The important part is validating the way someone feels, not "confirming perspectives". A feeling or a perspective can be valid ("I see where you're coming from, and it's entirely reasonable to feel that way"), even when the conclusion is incorrect ("however, here are the facts: ___. You might think ___ because ____, and that's reasonable. Still, this is how it is.") You're doing nobody a favor by affirming they are correct in believing things that are verifiably, factually false. There's a word for that. It's lying. When you're deliberately lying to keep someone in a relationship, that's manipulation. When you're lying to affirm someone's false views, distorting their perception of reality - particularly when they have doubts, and you are affirming a falsehood, with intent to control their behavior (e.g. make them stay in a relationship when they'd otherwise leave) - ... - that, my friend, is gaslighting. This is exactly what the machine was doing to the colleague who asked "which of us is right, me or the colleague that disagrees with me". It doesn't provide any useful information, it reaffirms a falsehood, it distorts someone's reality and destroys trust in others, it destroys relationships with others, and encourages addiction — because it maximizes "engagement". I.e., prevents someone from leaving. That's abuse. That, too is a part of life. >I agree with your conclusion, but that's by design All I did was named the phenomena we're talking about (lying, gaslighting, manipulation, abuse). Anyone can verify the correctness of the labeling in this context. I agree with your assertion, as well as that of the parent comment. And putting them together we have this: LLM chatbots today are abusive by design. This shit needs to be regulated, that's all. FDA and CPSC should get involved.

borski 7mo ago

This is the best I’ve ever heard this put.

Sharlin 7mo ago
Thank you!

godelski 7mo ago

> Nontechnical people simply don't have any idea about what LLMs are. We're on HN, a highly technical corner of the internet, yet we see the same stuff here. It's not just non-technical people... I think one of the big dangers is that people (including us) are quick to believe "I'm better than that". Yet this is a bias conmen have been exploiting for millennia. The only real defense is not lulling yourself into a false sense of security. You're less vulnerable (not invincible) by knowing you too can be fooled Honestly, it's just a good way to go about getting information. There's a famous Feynman quote about it too. The first principle is to not fool yourself, and you're the easiest person to fool

simonh 7mo ago
There’s nobody who knows how to fool you better than yourself.
- themafia 7mo ago
  There is extreme value to be had in astroturfing Hacker News.

delusional 7mo ago

> Nontechnical people simply don't have any idea about what LLMs are. We need to be very very careful here. Just like advertisements work, weather you think you're immune or not, so does AI. You might think you're spotting every red flag, but of course you think so. You can't see all the ones you missed. Do not make the mistake of thinking that being techy somehow immunizes you from flattery. It works on you too.

Sharlin 7mo ago
Of course not. It was meant as an implication, not equivalence.

themafia 7mo ago

> Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. That's an extreme downward punch. Have you not observed the marketing these LLM companies are themselves producing? They're intentionally misleading the public as to the capabilities of these systems. > if people were able to casually not anthropomorphize LLMs Of course they can. You just need to train them appropriately. No one is doing that. Companies are busy running around talking about the "end of coding" or the "end of work" because some extremely chinsy LLM models are around that they want to _sell you_.

pseudohadamard 7mo ago

Another issue is that people use LLMs to get their biases confirmed: Yes, your race is superior to the other races, immigrants are plotting to wipe you out, the Jews are behind it all, and so on. No (rational) person would ever agree with them on that, but an LLM will.