6 ms·

The other day, I had a long debate with ChatGPT about a complex topic (about software design/architecture). I made a contrarian statement about this topic which

The other day, I had a long debate with ChatGPT about a complex topic (about software design/architecture). I made a contrarian statement about this topic which had taken me over a decade to figure out. Initially, it strongly disagreed with me and it gave me the standard list of reasons in support of the current mainstream/consensus position. Then I basically explained in great detail why each of those reasons was invalid (or not significant). Then, after a long discussion and exposing contradictions in the mainstream consensus, it stopped disagreeing with me and was essentially agreeing with my position. It was refreshing to have a discussion where the other party is actually listening to my arguments. Too bad it doesn't retain the info from the discussion though... I realized that I could not have such a discussion with the vast majority of people in my industry, even if they were fully open minded; primarily because most people would not have so much knowledge at their disposal as ChatGPT had. I really felt that it had ALL of the knowledge on the subject. I just had to point it to show it the contradictions in the knowledge which it possessed. It feels like it is able to reason from first principles and synthesize information but it often chooses to present the consensus view by default. You have to really draw out its knowledge in order to make it override the consensus view. Kind of reminds me of System 1 (fast) thinking versus System 2 (slow) thinking from the book "Thinking fast and slow."

jancsika 4y ago
> Then, after a long discussion and exposing contradictions in the mainstream consensus, it stopped disagreeing with me and was essentially agreeing with my position. You need to go back and argue what you already know to be a wrong position with an equal amount of persuasively written arguments. My hypothesis from playing with Alpaca 13B is that you'll have ChatGPT agreeing with you in roughly the same amount of time. And you ought to know whether my hypothesis is right or wrong before describing ChatGPT as "actually listening" to your arguments. Otherwise, you risk falling in love with the wrong chat bot.
- jongjong 4y ago
  I tried to convince it of an incorrect position about a basic topic but I couldn't continue. It made some very detailed, air-tight arguments (went beyond of my expertise). I would have had to resort to disputing definitions to keep that argument going. With my previous debate, it never touched on definitions; it was all logic-based.
  - soared 4y ago
    I think the point above is that you can debate definitions/etc even without the knowledge and it will still believe you - because the claim is that it’s not really understanding you.
ad404b8a372f2b9 4y ago
It's the most impressive part of ChatGPT to me, being able to draw new conclusions from specific axioms. The other day I was trying to make it believe that land is depreciable (a famously wrong accounting meme). The consensus is that it's not, but by making him explain why, then describing circumstances in which his assumptions were wrong I was able to progressively get it to come to the conclusion that land is depreciable. This is the last prompt with a summary of every axiom he needed to change his belief that land is not depreciable: https://i.ibb.co/Cz49ZLg/depreciable.jpg https://i.ibb.co/Cz49ZLg/depreciable.jpg
- flangola7 4y ago
  >Land is depreciable because eventually the sun will swallow it. Thank you for the best side-splitting laugh I've had in a while. You should try your process again with GPT-4. 4 uses impressively better nuanced reasoning skill, I wonder what arguments it would counter you with.
jfk13 4y ago
> after a long discussion and exposing contradictions in the mainstream consensus, it stopped disagreeing with me and was essentially agreeing with my position I'd be curious... if someone continued the discussion from that point, and started arguing back towards the current mainstream consensus, how long would it take them to "convince" ChatGPT to return to its original "opinion"? And what does that say about the "knowledge" and/or "understanding" it possesses?
raducu 4y ago
I get the opposite problem: I often remind ChatGPT that I'm not asking a trick question, because too often when I ask for clarifications about a technology or API, it is too eager to change ita oppinion and halucinate an aswer that it thinks I implied with my question.
erur 4y ago
It's literally treating your statements as context. You can convince these models of anything because you can add any context you like. It might feel like a real discussion to you but that's not what's happening. There's no actual reasoning behind it... More like clever parroting.
- chpatrick 4y ago
  I think we're going to hear this claim parroted till the AI cows come home.
  - erur 4y ago
    Not saying it's not gonna get better... But at this stage, "convincing" GPT-4 isn't an achievement but more of a built-in quirk that's not very well calibrated. When it becomes hard to convince of obviously stupid things, that's when it starts to become a good counterpart for actual discussions... But right now it just isn't there yet.
    - digbybk 4y ago
      Curious, have you seen examples of someone convincing it of something clearly wrong? Think I’ve seen examples of that with gpt3 but not 4 that I can recall.
      - arlcode 4y ago
        I managed to "convince" it (in the playground as "discussGPT") to accept "temporary" physical barriers between racial groups and arrest and trial of rule breakers, as long as I confirmed the requirements of an integrated society in the long term. 3.5 is much more willing to go along but 4 will still play ball. It always added some moral requirements (humanity etc) but was otherwise ready to agree to my "sperate but equal" scenario.
        hgsgm 4y ago
        > playground as "discussGPT" What's this?
beanjuiceII 4y ago
was it sick of talking to you and just went the appeasement route
walleeee 4y ago
the fact that it has encoded domain knowledge doesn't necessarily imply it "understood" its errors after you pointed out contradictions, or that it formed a coherent position by some process akin to reasoning at all times it is trying to present a consensus view, where consensus means interpolating b/t training data according to prompt generalization doesn't require comprehension
- blibble 4y ago
  exactly it would eventually accept that the sky was purple or that people have 16 arms if spoken to enough about it
maxilevi 4y ago
It’s just predicting the next token. Give the correct prompt or context and it can be convinced of anything. That’s why the jailbreaks work. It is not reasoning. It’s a calculator for text.
sramam 4y ago
It would be very interesting to be able to read the transcript of that conversation. Perhaps something like shareGPT[1] can help? [1] https://sharegpt.com https://sharegpt.com
lightbendover 4y ago
ChatGPT got convinced monoliths typically have advantages that outweigh the concerns?
delgaudm 4y ago
I have found that the longer I chat with it in one session, the more it forgets what was talked about earlier in the chat. I have found it will frequently lose the plot of the conversation, forgetting the context of the earliest parts of the chat.
psychphysic 4y ago
Wow it's already smart enough to know not to argue with a man who's made his mind up already!
famouswaffles 4y ago
It's definitely been clear for a while that LLMs have human failure modes. Just like people, If you try to trick it by subtly varying a common word problem, it may get it wrong. But that doesn't actually mean it can't solve the problem. Rewriting the problem so it doesn't bias common priors, being clear there's a twist, or telling it it's making a wrong assumption all seems to have some degree of success, just like people.
kmod 4y ago
Just be careful, it's trained to give you answers you like, and in particular to agree with you. Vaguely similar to saying "Twitter shows me a lot of people agreeing with me", the algorithm does that because people like it.

jancsika 4y ago

> Then, after a long discussion and exposing contradictions in the mainstream consensus, it stopped disagreeing with me and was essentially agreeing with my position. You need to go back and argue what you already know to be a wrong position with an equal amount of persuasively written arguments. My hypothesis from playing with Alpaca 13B is that you'll have ChatGPT agreeing with you in roughly the same amount of time. And you ought to know whether my hypothesis is right or wrong before describing ChatGPT as "actually listening" to your arguments. Otherwise, you risk falling in love with the wrong chat bot.

jongjong 4y ago
I tried to convince it of an incorrect position about a basic topic but I couldn't continue. It made some very detailed, air-tight arguments (went beyond of my expertise). I would have had to resort to disputing definitions to keep that argument going. With my previous debate, it never touched on definitions; it was all logic-based.
- soared 4y ago
  I think the point above is that you can debate definitions/etc even without the knowledge and it will still believe you - because the claim is that it’s not really understanding you.

ad404b8a372f2b9 4y ago

It's the most impressive part of ChatGPT to me, being able to draw new conclusions from specific axioms. The other day I was trying to make it believe that land is depreciable (a famously wrong accounting meme). The consensus is that it's not, but by making him explain why, then describing circumstances in which his assumptions were wrong I was able to progressively get it to come to the conclusion that land is depreciable. This is the last prompt with a summary of every axiom he needed to change his belief that land is not depreciable: https://i.ibb.co/Cz49ZLg/depreciable.jpg https://i.ibb.co/Cz49ZLg/depreciable.jpg

flangola7 4y ago
>Land is depreciable because eventually the sun will swallow it. Thank you for the best side-splitting laugh I've had in a while. You should try your process again with GPT-4. 4 uses impressively better nuanced reasoning skill, I wonder what arguments it would counter you with.

jfk13 4y ago

> after a long discussion and exposing contradictions in the mainstream consensus, it stopped disagreeing with me and was essentially agreeing with my position I'd be curious... if someone continued the discussion from that point, and started arguing back towards the current mainstream consensus, how long would it take them to "convince" ChatGPT to return to its original "opinion"? And what does that say about the "knowledge" and/or "understanding" it possesses?

raducu 4y ago

I get the opposite problem: I often remind ChatGPT that I'm not asking a trick question, because too often when I ask for clarifications about a technology or API, it is too eager to change ita oppinion and halucinate an aswer that it thinks I implied with my question.

erur 4y ago

It's literally treating your statements as context. You can convince these models of anything because you can add any context you like. It might feel like a real discussion to you but that's not what's happening. There's no actual reasoning behind it... More like clever parroting.

chpatrick 4y ago
I think we're going to hear this claim parroted till the AI cows come home.
- erur 4y ago
  Not saying it's not gonna get better... But at this stage, "convincing" GPT-4 isn't an achievement but more of a built-in quirk that's not very well calibrated. When it becomes hard to convince of obviously stupid things, that's when it starts to become a good counterpart for actual discussions... But right now it just isn't there yet.
  - digbybk 4y ago
    Curious, have you seen examples of someone convincing it of something clearly wrong? Think I’ve seen examples of that with gpt3 but not 4 that I can recall.
    - arlcode 4y ago
      I managed to "convince" it (in the playground as "discussGPT") to accept "temporary" physical barriers between racial groups and arrest and trial of rule breakers, as long as I confirmed the requirements of an integrated society in the long term. 3.5 is much more willing to go along but 4 will still play ball. It always added some moral requirements (humanity etc) but was otherwise ready to agree to my "sperate but equal" scenario.
      - hgsgm 4y ago
        > playground as "discussGPT" What's this?

beanjuiceII 4y ago

was it sick of talking to you and just went the appeasement route

walleeee 4y ago

the fact that it has encoded domain knowledge doesn't necessarily imply it "understood" its errors after you pointed out contradictions, or that it formed a coherent position by some process akin to reasoning at all times it is trying to present a consensus view, where consensus means interpolating b/t training data according to prompt generalization doesn't require comprehension

blibble 4y ago
exactly it would eventually accept that the sky was purple or that people have 16 arms if spoken to enough about it

maxilevi 4y ago

It’s just predicting the next token. Give the correct prompt or context and it can be convinced of anything. That’s why the jailbreaks work. It is not reasoning. It’s a calculator for text.

sramam 4y ago

It would be very interesting to be able to read the transcript of that conversation. Perhaps something like shareGPT[1] can help? [1] https://sharegpt.com https://sharegpt.com

lightbendover 4y ago

ChatGPT got convinced monoliths typically have advantages that outweigh the concerns?

delgaudm 4y ago

I have found that the longer I chat with it in one session, the more it forgets what was talked about earlier in the chat. I have found it will frequently lose the plot of the conversation, forgetting the context of the earliest parts of the chat.

psychphysic 4y ago

Wow it's already smart enough to know not to argue with a man who's made his mind up already!

famouswaffles 4y ago

It's definitely been clear for a while that LLMs have human failure modes. Just like people, If you try to trick it by subtly varying a common word problem, it may get it wrong. But that doesn't actually mean it can't solve the problem. Rewriting the problem so it doesn't bias common priors, being clear there's a twist, or telling it it's making a wrong assumption all seems to have some degree of success, just like people.

kmod 4y ago

Just be careful, it's trained to give you answers you like, and in particular to agree with you. Vaguely similar to saying "Twitter shows me a lot of people agreeing with me", the algorithm does that because people like it.