5 ms·

Yeah anyone who has used ChatGPT for more than 30 minutes of asking it to write poetry about King Charles rapping with Tupac and other goofy stuff has realized

Yeah anyone who has used ChatGPT for more than 30 minutes of asking it to write poetry about King Charles rapping with Tupac and other goofy stuff has realized that it is essentially trained to assume that whatever you're saying to it is true and to not say anything negative to you. It can't write stories without happy endings, and it can't recognize when you ask it a question that contains a false premise. In marketing, I assume if you ask a fake target demographic if it will like your new product that is pogs but with blockchain technology, it will pretty much always say yes

Zambyte 2y ago
[flagged]
cj 2y ago
I’ve noticed this in article summaries. It does seem to have some weird biases. I’ve been able to get around that by directly asking it for pros/cons or “what are the downsides” or “identify any inconsistencies” or “where are the main risks”… etc There’s also a complexity threshold where it performs much better if you break a simple question down into multiple parts. You can basically to prompt-based transformations of your own input to break down information and analyze it in different ways prior to using all of that information to finally answer a higher level question. I wish ChatGPT could do this behind the scenes. Prompt itself “what questions should I ask myself that would help me answer this question?” And go through all those steps without exposing it to the user. Or maybe it can or already does, but it still seems like I get significantly better results when I do it manually and walk ChatGPT through the thought process myself.
- Propelloni 2y ago
  If you can do that and do it, for what do you need to ask the chatbot? Genuine question, because in my mind that's the heavy lifting you do there and you will get to a conclusion in the process. All the bot can do is agree with you and that serves what purpose?
- markovs_gun 2y ago
  Another interesting case with this is an instance I had with Google Assistant's AI summary feature for group chats. In the group chat, my mom said that my grandma was in the hospital and my sister said she was going to go visit her. In the AI summary, my grandma was on vacation and my sister was in the hospital. Completely useless.
IIAOPSW 2y ago
Yes, but with the caveat of in some very specific cases no. I spent a good deal of time trying to get it to believe there was a concept in the law of "praiseworthy homicide". I even had (real) citations to a law textbook. It refused to believe me. Given the massive selling point of ChatGPT to the legal profession, and the importance of actually being right, OpenAI certainly reduces the "high trait agree-ability" in favor of accuracy in this particular area.
- AnthonBerg 2y ago
  There’s a way to apply the concept of praiseworthy homicide – metaphorically – to your battle with it.
fragmede 2y ago
here's a story with a sad ending, called sad musical farewell. https://chatgpt.com/share/0d651c67-166f-4cef-bc8c-1f4d5747bd86 https://chatgpt.com/share/0d651c67-166f-4cef-bc8c-1f4d5747bd...
- Zambyte 2y ago
  Apparently counter examples are very unappreciated. I also gave a counter example for each of their claims, but my comment got flagged immediately. https://news.ycombinator.com/item?id=41187549 https://news.ycombinator.com/item?id=41187549
  - mrguyorama 2y ago
    As a probabilistic machine, singular counterexamples do not mean much. Two people can ask LLMs the same question and get different results.
- markovs_gun 2y ago
  I should have clarified that I meant that it has trouble writing stories with bad endings unless you ask for them directly and specifically, which can be burdensome if you're trying to get it to write a story about something specific that would naturally have a sad ending.
Terr_ 2y ago
> it is essentially trained to assume that whatever you're saying to it is true and to not say anything negative to you. Oh, it's actually worse than that: A given LLM probably has zero concept of "entities", let alone "you are one entity and I am another" or "statements can be truths or lies." There is merely one combined token stream, and dream-predicting the next tokens. While that output prediction often resembles a conversation we expect between two entities with boundaries, that says more about effective mimicry than about its internal operation.
- ithkuil 2y ago
  I agree there is limited modeling going on but the smoking gun is not on the fact that all there is to an LLM is mere "next token prediction". In order to successfully predict the next token the model needs to reach a significant level of "understanding" of the preceding context and the next token is the "seed" of a much longer planned response. Now, it's true that this "understanding" is not even close to what humans would call understanding (hence the quotes) and that the model behaviour is heavily biased towards productions that "sound convincing" or "sound human". Nevertheless LLMs perform an astounding amount of computation in order to produce that next token and that computation happens in a high dimensional space that captures a lot of "features" of the world derived from an unfathomably large and diverse training set. And there is still room for improvement in collecting. cleaning and/or synthesizing an even better training corpus for LLMs. Whether the current architecture of LLMs will ever be able to truly model the world is an open question but I don't think the question can be resolved just by pointing out that all the model does is produce the next token. That's just an effective way researchers found to build a channel with the external world (humans and the training set) and transform to and from the high-dimensional reasoning space and legible text.
  - borggambit 2y ago
    I think this is just wrong. Any "understanding" is a mirage. Case in point. prompt: "Is it possible to combine statistical mechanics, techno music production and pizza?" If the model had even the slightest understanding of the world the answer would just be no. Instead gpt4o: "Yes, it is possible to combine statistical mechanics, techno music production, and pizza, though it might require some creative thinking. Here’s how these three seemingly unrelated things could be connected" then gives a list of complete nonsense. The trick is that it can't say no because it doesn't understand ANYTHING. It has no understanding of the difference between combining pizza with two completely disparate nonsense subjects/items with combining pizza with two other food items. The later would seem to have a mirage, high dimensional, understanding from data of "food" though in the response.

Zambyte 2y ago

[flagged]

cj 2y ago

I’ve noticed this in article summaries. It does seem to have some weird biases. I’ve been able to get around that by directly asking it for pros/cons or “what are the downsides” or “identify any inconsistencies” or “where are the main risks”… etc There’s also a complexity threshold where it performs much better if you break a simple question down into multiple parts. You can basically to prompt-based transformations of your own input to break down information and analyze it in different ways prior to using all of that information to finally answer a higher level question. I wish ChatGPT could do this behind the scenes. Prompt itself “what questions should I ask myself that would help me answer this question?” And go through all those steps without exposing it to the user. Or maybe it can or already does, but it still seems like I get significantly better results when I do it manually and walk ChatGPT through the thought process myself.

Propelloni 2y ago
If you can do that and do it, for what do you need to ask the chatbot? Genuine question, because in my mind that's the heavy lifting you do there and you will get to a conclusion in the process. All the bot can do is agree with you and that serves what purpose?
markovs_gun 2y ago
Another interesting case with this is an instance I had with Google Assistant's AI summary feature for group chats. In the group chat, my mom said that my grandma was in the hospital and my sister said she was going to go visit her. In the AI summary, my grandma was on vacation and my sister was in the hospital. Completely useless.

IIAOPSW 2y ago

Yes, but with the caveat of in some very specific cases no. I spent a good deal of time trying to get it to believe there was a concept in the law of "praiseworthy homicide". I even had (real) citations to a law textbook. It refused to believe me. Given the massive selling point of ChatGPT to the legal profession, and the importance of actually being right, OpenAI certainly reduces the "high trait agree-ability" in favor of accuracy in this particular area.

AnthonBerg 2y ago
There’s a way to apply the concept of praiseworthy homicide – metaphorically – to your battle with it.

fragmede 2y ago

here's a story with a sad ending, called sad musical farewell. https://chatgpt.com/share/0d651c67-166f-4cef-bc8c-1f4d5747bd86 https://chatgpt.com/share/0d651c67-166f-4cef-bc8c-1f4d5747bd...

Zambyte 2y ago
Apparently counter examples are very unappreciated. I also gave a counter example for each of their claims, but my comment got flagged immediately. https://news.ycombinator.com/item?id=41187549 https://news.ycombinator.com/item?id=41187549
- mrguyorama 2y ago
  As a probabilistic machine, singular counterexamples do not mean much. Two people can ask LLMs the same question and get different results.
markovs_gun 2y ago
I should have clarified that I meant that it has trouble writing stories with bad endings unless you ask for them directly and specifically, which can be burdensome if you're trying to get it to write a story about something specific that would naturally have a sad ending.

Terr_ 2y ago

> it is essentially trained to assume that whatever you're saying to it is true and to not say anything negative to you. Oh, it's actually worse than that: A given LLM probably has zero concept of "entities", let alone "you are one entity and I am another" or "statements can be truths or lies." There is merely one combined token stream, and dream-predicting the next tokens. While that output prediction often resembles a conversation we expect between two entities with boundaries, that says more about effective mimicry than about its internal operation.

ithkuil 2y ago
I agree there is limited modeling going on but the smoking gun is not on the fact that all there is to an LLM is mere "next token prediction". In order to successfully predict the next token the model needs to reach a significant level of "understanding" of the preceding context and the next token is the "seed" of a much longer planned response. Now, it's true that this "understanding" is not even close to what humans would call understanding (hence the quotes) and that the model behaviour is heavily biased towards productions that "sound convincing" or "sound human". Nevertheless LLMs perform an astounding amount of computation in order to produce that next token and that computation happens in a high dimensional space that captures a lot of "features" of the world derived from an unfathomably large and diverse training set. And there is still room for improvement in collecting. cleaning and/or synthesizing an even better training corpus for LLMs. Whether the current architecture of LLMs will ever be able to truly model the world is an open question but I don't think the question can be resolved just by pointing out that all the model does is produce the next token. That's just an effective way researchers found to build a channel with the external world (humans and the training set) and transform to and from the high-dimensional reasoning space and legible text.
- borggambit 2y ago
  I think this is just wrong. Any "understanding" is a mirage. Case in point. prompt: "Is it possible to combine statistical mechanics, techno music production and pizza?" If the model had even the slightest understanding of the world the answer would just be no. Instead gpt4o: "Yes, it is possible to combine statistical mechanics, techno music production, and pizza, though it might require some creative thinking. Here’s how these three seemingly unrelated things could be connected" then gives a list of complete nonsense. The trick is that it can't say no because it doesn't understand ANYTHING. It has no understanding of the difference between combining pizza with two completely disparate nonsense subjects/items with combining pizza with two other food items. The later would seem to have a mirage, high dimensional, understanding from data of "food" though in the response.