8 ms·

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good

by mg 2y ago

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is logically strange here. Newer models still sometimes do, but more often than not catch it. GPT-4o for example: There is a slight inconsistency in the narrative. The text states that Joe drank tea with Sue in the kitchen after driving her to university, which implies that Sue is at home, contradicting the earlier statement that she was driven to university. Surely nothing in the prompt directly triggered the word "inconsistency". Did the model form some kind of "world model" in its inner layers in which it knows about a person "Sue" who is at a location called "university" after the first sentence?

abalaji 2y ago
My guess is that this is a post training (rlhf) artifact on world model prompts. There were likely many “logical inconsistency” prompts which humans coerced to the above response.
- brcmthrowaway 2y ago
  Is RLHF same as fine tuning?
  - flebron 2y ago
    No. In the common use of the word fine-tuning, one is in the supervised learning scenario. One has an input prompt, and an output sentence. One teaches the model to say that output in response to that prompt. In the reinforcement learning scenario, one has a prompt, and a way of rewarding the model for different outputs. One can have, for instance, a reward model, that assigns a reward for a given model output. One could also have a pairwise reward model, where the learner is sampled with that prompt twice (with different RNGs), and the reward model gives a reward based on the better of the two samples. You could also have humans give these pointwise or pairwise rewards. In essence, one is not telling the model "This. This is what you should output next time." but rather "I liked this reply. Have a cookie." The behaviors that you can learn in RL are more subtle, but you get a lot less information per step. That's because, in a causal language modeling objective, when I tell you "For the prompt X, you should output exactly Y[0...m)", you get a gradient for P(Y[0] | X), another one for P(Y[1] | X Y[0..1)), another for P(Y[2] | X Y[0..2)), another for P(Y[3] | X Y[0..3)), and so on. It's a lot more of a step-by-step guidance, than it is a sentence-wise reward that you get in the RL framework. In RL, I'd give you a cookie for P(Y | X). What part of Y made me give you that cookie? Was there even such a part? Was it perhaps some internal representation that made everything in Y better? That's for the model to learn.
    - jacobr1 2y ago
      One wrinkle, is that it is now common to fine-tune on previously derived RL datasets, with the tested inputs and preferred sample outputs as the training data.
tharkun__ 2y ago
That is one version of implication. The implication could also be that a time component isn't stated explicitly. He drove her to university. Time passes. Now it's "afterwards" and Sue is done with her classes and they go back home to drink tea. For all we know Joe had a cup of coffee while she was writing an exam but because they live so far from university it made no sense to go home and come back to come pick her up as it might take an hour to drive each way and the exam was 2 hours.
- deleted 2y ago
  [deleted]
awb 2y ago
I guess one could assume that Joe waited for Sue at her university before driving back to his home with her. The problem with many early LLMs is that they could be persuaded into believing something if you stated that it’s true. Maybe the default prompt in GPT-4o includes something to the effect of “You are a helpful, critical-thinking chatbot”…
prasoonds 2y ago
This is an interesting thought exercise! The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about the stability of the structure. The newer models more often than not got it right. Of course, that hasn't stopped people from making tired old claims of "stochastic parrot" and so forth but I think if your model can solve such reasoning problem (spatial or otherwise) then there's something really interesting to discover. I'm glad that folks at Anthropic are really trying to create a better understanding of these models (https://www.anthropic.com/research/mapping-mind-language-model https://www.anthropic.com/research/mapping-mind-language-mod...)
- amelius 2y ago
  > The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. But this is merely a definition of what it means to "understand" something. For example just tabulating many input/output combinations would not follow this definition.
  - prasoonds 2y ago
    You're right of course. Let me rephrase. What I was trying to say was: If the model solves some reasoning tasks that may imply it's building some inner world model which is fundamental to reasoning. That in-turn might mean that the model is reasoning. Others have argued against the "inner world model" theory and suggested that solving reasoning tasks is merely an extension of the "stochastic parrot" scenario - i.e., they claim that no such world model exists and that the model has rote memorized these reasoning scenarios.
    - zarzavat 2y ago
      Why not just give it a series of directions e.g. go one step forwards, turn right, etc like a LOGO program, and then ask it if it is back to the start or not. If the series of instructions is randomly generated then this is impossible to solve without a world model.
QuadmasterXLII 2y ago
A mental model of GPTs that's pretty successful at predicting their behaviour at the limit of large scale is an N-Gram model trained on a much larger internet than the real one. If you search the real world's internet for the text ``` You are a helpful and harmless AI assistant. What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. ``` It would be exponentially unlikely to show up even once. But if you could keep re-rolling the internet from whatever probability distribution generated it, eventually (and I mean eventually- every additional token in the prompt will require ~10,000 times more re-rolls of the whole internet) the whole prompt will show up, and you can just grab the next word. An N-Gram model trained on that massive multi-internet will answer the question in a human fashion. (likely, copied from a human writing science fiction).
- viccis 2y ago
  One of the impressive results of this to me is that it can capture illocutionary meaning, which is something that language modeling struggled with in the past. This is why if you say "Are you sure?" after one of its statements, it will often completely flip its statement to the opposite rather than interpreting "Are you sure?" as a discursive question.
- Filligree 2y ago
  An interesting way to think about it, but if it did that perfectly... wouldn't it effectively be human-level intelligent? Sure, it isn't any _particular_ human, but I don't see a big difference otherwise.
  - Jensson 2y ago
    No, since that larger internet is still not infinitely large, it still fails in dumb ways once your question is a tiny bit stranger.
faizshah 2y ago
This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with logical inconsistencies and then out of that vocabulary the highest likelihood words are those that come from the nearby context. I would posit that this interpretation is significantly more likely since instead of interpreting “afterwards” as “after Sue’s class finished” it interpreted the sentence as a lateral thinking problem probably because the training set had many lateral thinking problems in it that were used to test the model’s “reasoning” capabilities. This is the danger of trying to anthropomorphize LLMs, they are not thinking and there are clear limitations to the abilities of this architecture: https://youtu.be/MiqLoAZFRSE?si=iRhg_UJIokKseU7K https://youtu.be/MiqLoAZFRSE?si=iRhg_UJIokKseU7K
- smusamashah 2y ago
  seen similarly structured sentences But ChatGPT doesn't generalize the structure of sentences. If this same problem was written in a different language, or just replaced words in the sentence, the result will be very different, no?
  - faizshah 2y ago
    That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case. I’m just a practitioner so my language might be imprecise but when I say similarly structured sentences what I mean is, and this is my interpretation based on my experience with using Agents and LLMs, that the shape of the context as in the phrasing and the word choice highly bias the outputs of LLMs. In my own observations at work, those who interpret LLMs to be thinking often produce bad agents. LLM are not good at open ended questions, if you ask an LLM “improve this code” you will often get bad results that just look passable. But if you interpret LLMs as probabilistic models highly biased by their context then you would add a lot more context and specific instructions in the prompt in order to get the Agent to produce the right output. Side note, this is also why I like the AICI approach: https://github.com/microsoft/aici https://github.com/microsoft/aici A lot of us think it is silly how the phrasing and word choice can produce dramatically different results in RAG applications. If you could run a program (like AICI) that post processes the output and picks the next word in a more structured way instead of writing more creative prompts that just makes a lot more sense to me.
hoosieree 2y ago
> Surely nothing in the prompt directly triggered the word "inconsistency". The prompt is (implicitly) asking to find inconsistencies ("what do you think about") within some statements of fact. Many variations of "find problems in the text" are part of its training set. Remove the "what do you think about" and the model doesn't find inconsistencies. Or keep it, but make the following text more consistent, and watch it continue to find inconsistencies (which don't exist).
- williamcotton 2y ago
  https://news.ycombinator.com/item?id=40721115 https://news.ycombinator.com/item?id=40721115
  - hoosieree 2y ago
    "Explain" is a variation of "what do you think about", so you get similar results. https://news.ycombinator.com/item?id=40723601 https://news.ycombinator.com/item?id=40723601
    - deleted 2y ago
      [deleted]
    - deleted 2y ago
      [deleted]
    - williamcotton 2y ago
      It is most definitely not the same in any manner other than being a prompt for a response and not just a weird sentence floating in a void. What would a reasonable response be? “Cool story, bro?”
TheGlav 2y ago
I asked the same prompt, but have a memory asking for detailed answers. It gave me: The text presents a logical inconsistency. Here's a breakdown of the issues: Sequence of Events: Joe drives Sue to the university and then returns home. This part is clear. Inconsistency: The text states that after Joe drove home, he drank tea with Sue in the kitchen. However, Sue should be at the university if Joe drove her there, not at home. To correct this, consider one of the following revisions: Sue returns home: If Sue returns home with Joe: "Joe drove Sue to the university. Afterwards, he drove back home with her, and they drank tea together in the kitchen." Joe drinks tea alone: If Sue stays at the university: "Joe drove Sue to the university. Afterwards, he drove home again and drank tea alone in the kitchen." Either option removes the inconsistency and clarifies the sequence of events. This is pretty impressive.
- BiteCode_dev 2y ago
  Just yesterday people downvoted me on HN for suggesting ChatGPT had the best ability to deduct things, because, according to down voters, it didn't have any. But the "dumb parrot" narrative that was quite fitting on GPT2, and maybe still useful for understanding the foundation of the tech in GPT3, is now just a demonstration of ignorance. We don't know what they are really doing. Nobody working outside of openai knows. But if you use it long enough, it will be obvious we have passed the fancy completion phase, there is a little logic in there. Not a lot, but it makes all the results much better already. And definitely much better than the competition.
  - exe34 2y ago
    > Nobody working outside of openai knows. I wouldn't be too sure openai people know either. if a machine has enough moving parts, it's hard for an individual human to understand all of it. nowadays it's possible that nobody quite knows why the silicon compiler put a particular block in a particular position on the die. it just figured that's the best way to save power or space or whatever.
  - w1nk 2y ago
    For a more informed opinion than folks on the internet, here's some work from Microsoft with early/internal access to gpt4: https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 . I don't think people close by these systems share the same dumb parrot sentiment at all.
jonah 2y ago
I asked Google's Gemini and got the following: The text contains a logical inconsistency. The first sentence states that Joe drove Sue to university, implying Sue is now at university. The second sentence says Joe drank tea with Sue in the kitchen afterwards, which suggests Sue is back at home with Joe. This creates a contradiction. It's impossible for Sue to be both at university and in the kitchen at the same time. Then, I asked a follow-on: But what if Joe drove home with Sue even though it doesn't say that? (It doesn't say he drove home alone either.) To which it replied: You're absolutely right. The text we have is ambiguous. Here's why: * Original interpretation: We naturally assume Joe dropped Sue off and then returned home alone. This is because it's the most common scenario and aligns with the typical purpose of driving someone to a specific location (like a university). * Alternative interpretation: As you point out, the text doesn't explicitly state Joe drove back alone. It's entirely possible he gave Sue a ride home after her university visit. In this case, the seemingly illogical detail of drinking tea in the kitchen becomes perfectly reasonable. This is a great example of how limited information can lead to multiple interpretations. The next sentence or some additional context could clarify the situation. For instance: * "Joe dropped Sue off at her dorm and then drove home alone to make himself a cup of tea." (Original interpretation) * "After dropping Sue off, Joe drove them both back to their place and they enjoyed a cup of tea together in the kitchen." (Alternative interpretation)
danielmarkbruce 2y ago
Stop thinking about words. Think about concepts. As soon as you turn words/tokens into high dimensional embeddings and start playing around with them, they stop being words.
- namaria 2y ago
  Not when the only way to get anything out of it is using these matrices to line up word tokens. Or play with jpeg noise.
  - danielmarkbruce 2y ago
    Unclear what you are saying here
    - namaria 2y ago
      Claiming LLMs are manipulating 'high dimension concepts' not words when all they can do with these 'high dimension concepts' is outputting words is meaningless.
      - danielmarkbruce 2y ago
        It's meaningful because the model can learn relationships between things (like, that a dog and cat are often pets and interchangdable in certain circumstances) and then produce output that it has never seen. Read the word2vec paper as an example. And, an LLM can output whatever you want. Change the last linear layer from (embedding dimension -> vocab size) to (embedding dimension -> sentiment categories) for example, and hey presto it can produce sentiment analysis (with surprisingly few samples, because... it's learned all the concepts in the transformer blocks in pre-training). You gotta actually have a clue about how they work. This stuff is no longer magic, there are many resources to learn them in detail.
        namaria 2y ago
        This isn't magic. Precisely. Calling it manipulating high dimension concepts muddles the waters. It's not. It's merely using a whole lot of numbers to manipulate word tokens. And it's quite limited at that despite the impressive ability to store up language corner cases and put them back out. "an LLM can output whatever you want" -> this right here is approaching magical thinking. I am really clear headed about it's capabilities and limitations, breathlessly describing it's internals as if it implies it's anything more then a first decent NLP interface is the problem and you seem to be indulging in it.
namaria 2y ago
Occam's Razor says that between the model having somehow developed a persistent internal world model representation and can now reason about it - or is it just big enough in terms of training data, knobs and memory allocation that it can capture more and more language corner cases... Well likely the latter is true.
- deleted 2y ago
  [deleted]
empath75 2y ago
There's so many people who just flatly say that it is impossible for ChatGPT to do what it plainly does, and I think it's largely because early explanations of how it works were so bad. "Stochastic parrot" was a particularly obnoxious one, that wasn't even entirely true for gpt-2 and certainly wasn't accurate for GPT-3, and just seems silly trying to apply it to GPT-4.
exe34 2y ago
interestingly, I, a human and totally not an LLM, completely missed the inconsistency, and then when I saw it pointed out to me, initially tried to argue that it's not necessarily an inconsistency until I thought it through.
Anon84 2y ago
Here is GPT-4o with a "Please explain your thinking step by step" `The text describes a sequence of events involving two characters, Joe and Sue. Here’s a step-by-step analysis of the text: First Sentence: "Joe drove Sue to university." Action: Joe is driving. Destination: University. Passenger: Sue. Second Sentence: "Afterwards he drove home again and drank a tea with her in the kitchen." Action: Joe drives home. Time Reference: "Afterwards" indicates this action happens after driving Sue to university. Activity: Joe drinks tea. Location: In the kitchen. Companion: Sue ("with her" refers to Sue). Logical Flow and Possible Inconsistencies Driving Sequence: Joe drives Sue to university. Joe drives home again. This implies Joe drove to university, dropped off Sue, and then returned home. Tea Drinking: Joe drinks tea with Sue at home in the kitchen. Potential Issues and Interpretation: Continuity Issue: If Sue was dropped off at the university, she shouldn't be home with Joe unless she somehow returned home quickly or didn't actually stay at the university. Timeframe: The text lacks details on the timeframe between dropping off Sue and drinking tea, leading to ambiguity. If there’s a significant time gap, it would make more sense. Otherwise, it’s confusing. Possible Corrections or Clarifications: Correction for Continuity: "Joe drove Sue to university. Afterwards, he picked her up, and they drove home together and drank tea in the kitchen." This makes it clear that Sue is back home with Joe. Additional Information: Providing more context about the events or timeframe could help clarify the situation. In conclusion, while the text describes a sequence of actions, it has a logical inconsistency that could confuse readers. Clarifying the timeline or actions would make the text more coherent.`
- tga 2y ago
  A LLM cannot meaningfully “explain its thinking”. When prompted to do so, it will generate an explanation-looking text — but that has nothing to do with the actual way the original answer was generated. If you ask it to explain how a LLM could possibly catch an inconsistency, then it might give you some good insights (or hallucinate something entertaining).
  - zeknife 2y ago
    It can't explain it's thinking after producing the answer, like you say, it'll just generate a post-hoc rationalization. But if you ask it to think step by step before making a conclusion, LLM's will somewhat reliably arrive at reasonable steps and better final answers.
Kerbonut 2y ago
I think it's system prompt dependent. I asked LLaMa-2-70B iQ2s which was set up with a system prompt telling it to be logical and detail oriented. First try: I think there is a logical inconsistency in the text. If Joe drove Sue to university, then she was dropped off at the university, which means she would not be available to drink tea with him at his home afterwards. The sequence of events seems to be out of order. Second try: I notice that there's a small issue with the sequence of events. It seems unlikely that Joe drove home again after dropping Sue off at university, only to then drink tea with her in the kitchen. If Joe drove Sue to university, it implies she got out of the car at the university, not that he drove back home with her. A more logical sequence of events would be: Joe drove Sue to university. Afterwards, he drove home and drank a cup of tea by himself in the kitchen. Or, if you want to keep the scenario where Joe drinks tea with Sue: Joe drove Sue to university. They went back home together, and then they drank a cup of tea together in the kitchen.
misnome 2y ago
Does this work without “Clever Hans” prompting it with the implication that there is something to notice?
- williamcotton 2y ago
  Yes.
  - deleted 2y ago
    [deleted]
  - Jensson 2y ago
    I just tried with 4o and it didn't notice anything, it just rewrote the statements with different words.
    - williamcotton 2y ago
      https://news.ycombinator.com/item?id=40721115 https://news.ycombinator.com/item?id=40721115
      - misnome 2y ago
        You explicitly asked it to explain?
        williamcotton 2y ago
        You need to ask it something or else it is just a weird sentence floating in a void. What would a reasonable response be? “And..?”
- hoosieree 2y ago
  No. Remove the "explain" or "what do you think" or "think critically about" types of phrases from the prompt and you get this kind of thing: My prompt: Joe drove Sue to the university. Afterwards, he drove home again and drank tea with her in the kitchen. GPT-4o Response: Your sentence is correct, but for a slightly smoother flow, you could rephrase it as: Joe drove Sue to the university. Afterwards, he returned home and drank tea with her in the kitchen. It will never "notice" anything off about the input unless you ask it to. I'm consistently impressed how many supposedly tech-savvy people believe GPTs turn into magic orbs if you make them big enough.
accrual 2y ago
Just to add another sample from another LLM, here is the response from Llama-3-Instruct-8B. I had to specifically ask it identify any possible inconsistency, otherwise it just thought it was a lovely short story. > The inconsistency is that Joe drives Sue to university, which suggests it's morning or early afternoon. But then he drives "home again", implying that he was already at his own home before taking Sue to university. This seems unlikely and creates a paradox! What do you think is going on here?
deleted 2y ago
[deleted]
mriziq 2y ago
There is a world model. Please see Wattenberg Viegas 2023 https://arxiv.org/abs/2305.02469 https://arxiv.org/abs/2305.02469

by mg 2y ago

abalaji 2y ago

My guess is that this is a post training (rlhf) artifact on world model prompts. There were likely many “logical inconsistency” prompts which humans coerced to the above response.

brcmthrowaway 2y ago
Is RLHF same as fine tuning?
- flebron 2y ago
  No. In the common use of the word fine-tuning, one is in the supervised learning scenario. One has an input prompt, and an output sentence. One teaches the model to say that output in response to that prompt. In the reinforcement learning scenario, one has a prompt, and a way of rewarding the model for different outputs. One can have, for instance, a reward model, that assigns a reward for a given model output. One could also have a pairwise reward model, where the learner is sampled with that prompt twice (with different RNGs), and the reward model gives a reward based on the better of the two samples. You could also have humans give these pointwise or pairwise rewards. In essence, one is not telling the model "This. This is what you should output next time." but rather "I liked this reply. Have a cookie." The behaviors that you can learn in RL are more subtle, but you get a lot less information per step. That's because, in a causal language modeling objective, when I tell you "For the prompt X, you should output exactly Y[0...m)", you get a gradient for P(Y[0] | X), another one for P(Y[1] | X Y[0..1)), another for P(Y[2] | X Y[0..2)), another for P(Y[3] | X Y[0..3)), and so on. It's a lot more of a step-by-step guidance, than it is a sentence-wise reward that you get in the RL framework. In RL, I'd give you a cookie for P(Y | X). What part of Y made me give you that cookie? Was there even such a part? Was it perhaps some internal representation that made everything in Y better? That's for the model to learn.
  - jacobr1 2y ago
    One wrinkle, is that it is now common to fine-tune on previously derived RL datasets, with the tested inputs and preferred sample outputs as the training data.

tharkun__ 2y ago

That is one version of implication. The implication could also be that a time component isn't stated explicitly. He drove her to university. Time passes. Now it's "afterwards" and Sue is done with her classes and they go back home to drink tea. For all we know Joe had a cup of coffee while she was writing an exam but because they live so far from university it made no sense to go home and come back to come pick her up as it might take an hour to drive each way and the exam was 2 hours.

deleted 2y ago
[deleted]

awb 2y ago

I guess one could assume that Joe waited for Sue at her university before driving back to his home with her. The problem with many early LLMs is that they could be persuaded into believing something if you stated that it’s true. Maybe the default prompt in GPT-4o includes something to the effect of “You are a helpful, critical-thinking chatbot”…

prasoonds 2y ago

This is an interesting thought exercise! The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about the stability of the structure. The newer models more often than not got it right. Of course, that hasn't stopped people from making tired old claims of "stochastic parrot" and so forth but I think if your model can solve such reasoning problem (spatial or otherwise) then there's something really interesting to discover. I'm glad that folks at Anthropic are really trying to create a better understanding of these models (https://www.anthropic.com/research/mapping-mind-language-model https://www.anthropic.com/research/mapping-mind-language-mod...)

amelius 2y ago
> The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. But this is merely a definition of what it means to "understand" something. For example just tabulating many input/output combinations would not follow this definition.
- prasoonds 2y ago
  You're right of course. Let me rephrase. What I was trying to say was: If the model solves some reasoning tasks that may imply it's building some inner world model which is fundamental to reasoning. That in-turn might mean that the model is reasoning. Others have argued against the "inner world model" theory and suggested that solving reasoning tasks is merely an extension of the "stochastic parrot" scenario - i.e., they claim that no such world model exists and that the model has rote memorized these reasoning scenarios.
  - zarzavat 2y ago
    Why not just give it a series of directions e.g. go one step forwards, turn right, etc like a LOGO program, and then ask it if it is back to the start or not. If the series of instructions is randomly generated then this is impossible to solve without a world model.

QuadmasterXLII 2y ago

A mental model of GPTs that's pretty successful at predicting their behaviour at the limit of large scale is an N-Gram model trained on a much larger internet than the real one. If you search the real world's internet for the text ``` You are a helpful and harmless AI assistant. What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. ``` It would be exponentially unlikely to show up even once. But if you could keep re-rolling the internet from whatever probability distribution generated it, eventually (and I mean eventually- every additional token in the prompt will require ~10,000 times more re-rolls of the whole internet) the whole prompt will show up, and you can just grab the next word. An N-Gram model trained on that massive multi-internet will answer the question in a human fashion. (likely, copied from a human writing science fiction).

viccis 2y ago
One of the impressive results of this to me is that it can capture illocutionary meaning, which is something that language modeling struggled with in the past. This is why if you say "Are you sure?" after one of its statements, it will often completely flip its statement to the opposite rather than interpreting "Are you sure?" as a discursive question.
Filligree 2y ago
An interesting way to think about it, but if it did that perfectly... wouldn't it effectively be human-level intelligent? Sure, it isn't any _particular_ human, but I don't see a big difference otherwise.
- Jensson 2y ago
  No, since that larger internet is still not infinitely large, it still fails in dumb ways once your question is a tiny bit stranger.

faizshah 2y ago

This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with logical inconsistencies and then out of that vocabulary the highest likelihood words are those that come from the nearby context. I would posit that this interpretation is significantly more likely since instead of interpreting “afterwards” as “after Sue’s class finished” it interpreted the sentence as a lateral thinking problem probably because the training set had many lateral thinking problems in it that were used to test the model’s “reasoning” capabilities. This is the danger of trying to anthropomorphize LLMs, they are not thinking and there are clear limitations to the abilities of this architecture: https://youtu.be/MiqLoAZFRSE?si=iRhg_UJIokKseU7K https://youtu.be/MiqLoAZFRSE?si=iRhg_UJIokKseU7K

smusamashah 2y ago
seen similarly structured sentences But ChatGPT doesn't generalize the structure of sentences. If this same problem was written in a different language, or just replaced words in the sentence, the result will be very different, no?
- faizshah 2y ago
  That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case. I’m just a practitioner so my language might be imprecise but when I say similarly structured sentences what I mean is, and this is my interpretation based on my experience with using Agents and LLMs, that the shape of the context as in the phrasing and the word choice highly bias the outputs of LLMs. In my own observations at work, those who interpret LLMs to be thinking often produce bad agents. LLM are not good at open ended questions, if you ask an LLM “improve this code” you will often get bad results that just look passable. But if you interpret LLMs as probabilistic models highly biased by their context then you would add a lot more context and specific instructions in the prompt in order to get the Agent to produce the right output. Side note, this is also why I like the AICI approach: https://github.com/microsoft/aici https://github.com/microsoft/aici A lot of us think it is silly how the phrasing and word choice can produce dramatically different results in RAG applications. If you could run a program (like AICI) that post processes the output and picks the next word in a more structured way instead of writing more creative prompts that just makes a lot more sense to me.

hoosieree 2y ago

> Surely nothing in the prompt directly triggered the word "inconsistency". The prompt is (implicitly) asking to find inconsistencies ("what do you think about") within some statements of fact. Many variations of "find problems in the text" are part of its training set. Remove the "what do you think about" and the model doesn't find inconsistencies. Or keep it, but make the following text more consistent, and watch it continue to find inconsistencies (which don't exist).

williamcotton 2y ago
https://news.ycombinator.com/item?id=40721115 https://news.ycombinator.com/item?id=40721115
- hoosieree 2y ago
  "Explain" is a variation of "what do you think about", so you get similar results. https://news.ycombinator.com/item?id=40723601 https://news.ycombinator.com/item?id=40723601
  - deleted 2y ago
    [deleted]
  - deleted 2y ago
    [deleted]
  - williamcotton 2y ago
    It is most definitely not the same in any manner other than being a prompt for a response and not just a weird sentence floating in a void. What would a reasonable response be? “Cool story, bro?”

TheGlav 2y ago

I asked the same prompt, but have a memory asking for detailed answers. It gave me: The text presents a logical inconsistency. Here's a breakdown of the issues: Sequence of Events: Joe drives Sue to the university and then returns home. This part is clear. Inconsistency: The text states that after Joe drove home, he drank tea with Sue in the kitchen. However, Sue should be at the university if Joe drove her there, not at home. To correct this, consider one of the following revisions: Sue returns home: If Sue returns home with Joe: "Joe drove Sue to the university. Afterwards, he drove back home with her, and they drank tea together in the kitchen." Joe drinks tea alone: If Sue stays at the university: "Joe drove Sue to the university. Afterwards, he drove home again and drank tea alone in the kitchen." Either option removes the inconsistency and clarifies the sequence of events. This is pretty impressive.

BiteCode_dev 2y ago
Just yesterday people downvoted me on HN for suggesting ChatGPT had the best ability to deduct things, because, according to down voters, it didn't have any. But the "dumb parrot" narrative that was quite fitting on GPT2, and maybe still useful for understanding the foundation of the tech in GPT3, is now just a demonstration of ignorance. We don't know what they are really doing. Nobody working outside of openai knows. But if you use it long enough, it will be obvious we have passed the fancy completion phase, there is a little logic in there. Not a lot, but it makes all the results much better already. And definitely much better than the competition.
- exe34 2y ago
  > Nobody working outside of openai knows. I wouldn't be too sure openai people know either. if a machine has enough moving parts, it's hard for an individual human to understand all of it. nowadays it's possible that nobody quite knows why the silicon compiler put a particular block in a particular position on the die. it just figured that's the best way to save power or space or whatever.
- w1nk 2y ago
  For a more informed opinion than folks on the internet, here's some work from Microsoft with early/internal access to gpt4: https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 . I don't think people close by these systems share the same dumb parrot sentiment at all.

jonah 2y ago

I asked Google's Gemini and got the following: The text contains a logical inconsistency. The first sentence states that Joe drove Sue to university, implying Sue is now at university. The second sentence says Joe drank tea with Sue in the kitchen afterwards, which suggests Sue is back at home with Joe. This creates a contradiction. It's impossible for Sue to be both at university and in the kitchen at the same time. Then, I asked a follow-on: But what if Joe drove home with Sue even though it doesn't say that? (It doesn't say he drove home alone either.) To which it replied: You're absolutely right. The text we have is ambiguous. Here's why: * Original interpretation: We naturally assume Joe dropped Sue off and then returned home alone. This is because it's the most common scenario and aligns with the typical purpose of driving someone to a specific location (like a university). * Alternative interpretation: As you point out, the text doesn't explicitly state Joe drove back alone. It's entirely possible he gave Sue a ride home after her university visit. In this case, the seemingly illogical detail of drinking tea in the kitchen becomes perfectly reasonable. This is a great example of how limited information can lead to multiple interpretations. The next sentence or some additional context could clarify the situation. For instance: * "Joe dropped Sue off at her dorm and then drove home alone to make himself a cup of tea." (Original interpretation) * "After dropping Sue off, Joe drove them both back to their place and they enjoyed a cup of tea together in the kitchen." (Alternative interpretation)

danielmarkbruce 2y ago

Stop thinking about words. Think about concepts. As soon as you turn words/tokens into high dimensional embeddings and start playing around with them, they stop being words.

namaria 2y ago
Not when the only way to get anything out of it is using these matrices to line up word tokens. Or play with jpeg noise.
- danielmarkbruce 2y ago
  Unclear what you are saying here
  - namaria 2y ago
    Claiming LLMs are manipulating 'high dimension concepts' not words when all they can do with these 'high dimension concepts' is outputting words is meaningless.
    - danielmarkbruce 2y ago
      It's meaningful because the model can learn relationships between things (like, that a dog and cat are often pets and interchangdable in certain circumstances) and then produce output that it has never seen. Read the word2vec paper as an example. And, an LLM can output whatever you want. Change the last linear layer from (embedding dimension -> vocab size) to (embedding dimension -> sentiment categories) for example, and hey presto it can produce sentiment analysis (with surprisingly few samples, because... it's learned all the concepts in the transformer blocks in pre-training). You gotta actually have a clue about how they work. This stuff is no longer magic, there are many resources to learn them in detail.
      - namaria 2y ago
        This isn't magic. Precisely. Calling it manipulating high dimension concepts muddles the waters. It's not. It's merely using a whole lot of numbers to manipulate word tokens. And it's quite limited at that despite the impressive ability to store up language corner cases and put them back out. "an LLM can output whatever you want" -> this right here is approaching magical thinking. I am really clear headed about it's capabilities and limitations, breathlessly describing it's internals as if it implies it's anything more then a first decent NLP interface is the problem and you seem to be indulging in it.

namaria 2y ago

Occam's Razor says that between the model having somehow developed a persistent internal world model representation and can now reason about it - or is it just big enough in terms of training data, knobs and memory allocation that it can capture more and more language corner cases... Well likely the latter is true.

deleted 2y ago
[deleted]

empath75 2y ago

There's so many people who just flatly say that it is impossible for ChatGPT to do what it plainly does, and I think it's largely because early explanations of how it works were so bad. "Stochastic parrot" was a particularly obnoxious one, that wasn't even entirely true for gpt-2 and certainly wasn't accurate for GPT-3, and just seems silly trying to apply it to GPT-4.

exe34 2y ago

interestingly, I, a human and totally not an LLM, completely missed the inconsistency, and then when I saw it pointed out to me, initially tried to argue that it's not necessarily an inconsistency until I thought it through.

Anon84 2y ago

Here is GPT-4o with a "Please explain your thinking step by step" `The text describes a sequence of events involving two characters, Joe and Sue. Here’s a step-by-step analysis of the text: First Sentence: "Joe drove Sue to university." Action: Joe is driving. Destination: University. Passenger: Sue. Second Sentence: "Afterwards he drove home again and drank a tea with her in the kitchen." Action: Joe drives home. Time Reference: "Afterwards" indicates this action happens after driving Sue to university. Activity: Joe drinks tea. Location: In the kitchen. Companion: Sue ("with her" refers to Sue). Logical Flow and Possible Inconsistencies Driving Sequence: Joe drives Sue to university. Joe drives home again. This implies Joe drove to university, dropped off Sue, and then returned home. Tea Drinking: Joe drinks tea with Sue at home in the kitchen. Potential Issues and Interpretation: Continuity Issue: If Sue was dropped off at the university, she shouldn't be home with Joe unless she somehow returned home quickly or didn't actually stay at the university. Timeframe: The text lacks details on the timeframe between dropping off Sue and drinking tea, leading to ambiguity. If there’s a significant time gap, it would make more sense. Otherwise, it’s confusing. Possible Corrections or Clarifications: Correction for Continuity: "Joe drove Sue to university. Afterwards, he picked her up, and they drove home together and drank tea in the kitchen." This makes it clear that Sue is back home with Joe. Additional Information: Providing more context about the events or timeframe could help clarify the situation. In conclusion, while the text describes a sequence of actions, it has a logical inconsistency that could confuse readers. Clarifying the timeline or actions would make the text more coherent.`

tga 2y ago
A LLM cannot meaningfully “explain its thinking”. When prompted to do so, it will generate an explanation-looking text — but that has nothing to do with the actual way the original answer was generated. If you ask it to explain how a LLM could possibly catch an inconsistency, then it might give you some good insights (or hallucinate something entertaining).
- zeknife 2y ago
  It can't explain it's thinking after producing the answer, like you say, it'll just generate a post-hoc rationalization. But if you ask it to think step by step before making a conclusion, LLM's will somewhat reliably arrive at reasonable steps and better final answers.

Kerbonut 2y ago

I think it's system prompt dependent. I asked LLaMa-2-70B iQ2s which was set up with a system prompt telling it to be logical and detail oriented. First try: I think there is a logical inconsistency in the text. If Joe drove Sue to university, then she was dropped off at the university, which means she would not be available to drink tea with him at his home afterwards. The sequence of events seems to be out of order. Second try: I notice that there's a small issue with the sequence of events. It seems unlikely that Joe drove home again after dropping Sue off at university, only to then drink tea with her in the kitchen. If Joe drove Sue to university, it implies she got out of the car at the university, not that he drove back home with her. A more logical sequence of events would be: Joe drove Sue to university. Afterwards, he drove home and drank a cup of tea by himself in the kitchen. Or, if you want to keep the scenario where Joe drinks tea with Sue: Joe drove Sue to university. They went back home together, and then they drank a cup of tea together in the kitchen.

misnome 2y ago

Does this work without “Clever Hans” prompting it with the implication that there is something to notice?

williamcotton 2y ago
Yes.
- deleted 2y ago
  [deleted]
- Jensson 2y ago
  I just tried with 4o and it didn't notice anything, it just rewrote the statements with different words.
  - williamcotton 2y ago
    https://news.ycombinator.com/item?id=40721115 https://news.ycombinator.com/item?id=40721115
    - misnome 2y ago
      You explicitly asked it to explain?
      - williamcotton 2y ago
        You need to ask it something or else it is just a weird sentence floating in a void. What would a reasonable response be? “And..?”
hoosieree 2y ago
No. Remove the "explain" or "what do you think" or "think critically about" types of phrases from the prompt and you get this kind of thing: My prompt: Joe drove Sue to the university. Afterwards, he drove home again and drank tea with her in the kitchen. GPT-4o Response: Your sentence is correct, but for a slightly smoother flow, you could rephrase it as: Joe drove Sue to the university. Afterwards, he returned home and drank tea with her in the kitchen. It will never "notice" anything off about the input unless you ask it to. I'm consistently impressed how many supposedly tech-savvy people believe GPTs turn into magic orbs if you make them big enough.

accrual 2y ago

Just to add another sample from another LLM, here is the response from Llama-3-Instruct-8B. I had to specifically ask it identify any possible inconsistency, otherwise it just thought it was a lovely short story. > The inconsistency is that Joe drives Sue to university, which suggests it's morning or early afternoon. But then he drives "home again", implying that he was already at his own home before taking Sue to university. This seems unlikely and creates a paradox! What do you think is going on here?

deleted 2y ago

[deleted]

mriziq 2y ago

There is a world model. Please see Wattenberg Viegas 2023 https://arxiv.org/abs/2305.02469 https://arxiv.org/abs/2305.02469