5 ms·

This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral th

This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with logical inconsistencies and then out of that vocabulary the highest likelihood words are those that come from the nearby context. I would posit that this interpretation is significantly more likely since instead of interpreting “afterwards” as “after Sue’s class finished” it interpreted the sentence as a lateral thinking problem probably because the training set had many lateral thinking problems in it that were used to test the model’s “reasoning” capabilities. This is the danger of trying to anthropomorphize LLMs, they are not thinking and there are clear limitations to the abilities of this architecture: https://youtu.be/MiqLoAZFRSE?si=iRhg_UJIokKseU7K https://youtu.be/MiqLoAZFRSE?si=iRhg_UJIokKseU7K

smusamashah 2y ago
seen similarly structured sentences But ChatGPT doesn't generalize the structure of sentences. If this same problem was written in a different language, or just replaced words in the sentence, the result will be very different, no?
- faizshah 2y ago
  That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case. I’m just a practitioner so my language might be imprecise but when I say similarly structured sentences what I mean is, and this is my interpretation based on my experience with using Agents and LLMs, that the shape of the context as in the phrasing and the word choice highly bias the outputs of LLMs. In my own observations at work, those who interpret LLMs to be thinking often produce bad agents. LLM are not good at open ended questions, if you ask an LLM “improve this code” you will often get bad results that just look passable. But if you interpret LLMs as probabilistic models highly biased by their context then you would add a lot more context and specific instructions in the prompt in order to get the Agent to produce the right output. Side note, this is also why I like the AICI approach: https://github.com/microsoft/aici https://github.com/microsoft/aici A lot of us think it is silly how the phrasing and word choice can produce dramatically different results in RAG applications. If you could run a program (like AICI) that post processes the output and picks the next word in a more structured way instead of writing more creative prompts that just makes a lot more sense to me.
calf 2y ago
It could be neither here nor there, because a neural network model is not necessarily a statistical engine just as real brains should not be reductively thought of as such. (E.g. The usual argument is that these black boxes were trained on huge samples to optimize a result, I am not persuaded by that line of reasoning.)
mg 2y ago
You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her mother in the kitchen. In my experiments, the model now no longer claims there is an inconsistency. If it is only based on sentence structure, what makes the model output "inconsistency" for the first prompt but not for the second?
- faizshah 2y ago
  If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure. ``` Can you summarize joe’s schedule for my journal? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Sure! Here is a summary of Joe's schedule for your journal: --- \*Joe's Schedule:\* - Drove Sue to university - Returned home - Drank tea with Sue in the kitchen ——- ```
  - williamcotton 2y ago
    Me: Joe drove Sue to university. Afterwords he drove home again and drank a tea with her in the kitchen. Explain. ChatGPT: This statement seems to contain an ambiguity regarding Sue's location, suggesting some confusion about the timeline of events. Here are a few possibilities that could clarify the scenario: 1. *Sue Returns with Joe*: - Joe drove Sue to university. - Sue either decided not to stay at university or completed what she needed to do quickly. - Joe drove Sue back home. - They both drank tea together in the kitchen. 2. *Misstatement or Mistake*: - The statement might contain a mistake, implying that Joe and Sue met at home later but doesn't clearly explain how Sue got back from the university. - It could be intended to mean that Joe drove Sue to university and then drove home alone, and Sue somehow returned home later to join Joe for tea. 3. *Joe Visits University*: - Joe drove Sue to university. - Joe stayed at the university, possibly waiting for Sue to finish. - After she was done, they both drove back home together. - They then drank tea in the kitchen. Based on the original statement, the first scenario seems most plausible, suggesting Sue quickly returned home with Joe.
empath75 2y ago
There's a danger in flatly stating what it's doing _without testing your assumptions_. Did you actually try and trip it up and see if it falls for it? Reword his prompt in different ways to get it to fail?

smusamashah 2y ago

seen similarly structured sentences But ChatGPT doesn't generalize the structure of sentences. If this same problem was written in a different language, or just replaced words in the sentence, the result will be very different, no?

faizshah 2y ago
That’s right, if LLMs were really thinking/forming world models etc. we would expect them to be robust against word choice or phrasing. But in practice anyone using RAG can tell you that that is not the case. I’m just a practitioner so my language might be imprecise but when I say similarly structured sentences what I mean is, and this is my interpretation based on my experience with using Agents and LLMs, that the shape of the context as in the phrasing and the word choice highly bias the outputs of LLMs. In my own observations at work, those who interpret LLMs to be thinking often produce bad agents. LLM are not good at open ended questions, if you ask an LLM “improve this code” you will often get bad results that just look passable. But if you interpret LLMs as probabilistic models highly biased by their context then you would add a lot more context and specific instructions in the prompt in order to get the Agent to produce the right output. Side note, this is also why I like the AICI approach: https://github.com/microsoft/aici https://github.com/microsoft/aici A lot of us think it is silly how the phrasing and word choice can produce dramatically different results in RAG applications. If you could run a program (like AICI) that post processes the output and picks the next word in a more structured way instead of writing more creative prompts that just makes a lot more sense to me.

calf 2y ago

It could be neither here nor there, because a neural network model is not necessarily a statistical engine just as real brains should not be reductively thought of as such. (E.g. The usual argument is that these black boxes were trained on huge samples to optimize a result, I am not persuaded by that line of reasoning.)

mg 2y ago

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her mother in the kitchen. In my experiments, the model now no longer claims there is an inconsistency. If it is only based on sentence structure, what makes the model output "inconsistency" for the first prompt but not for the second?

faizshah 2y ago
If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure. ``` Can you summarize joe’s schedule for my journal? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Sure! Here is a summary of Joe's schedule for your journal: --- \*Joe's Schedule:\* - Drove Sue to university - Returned home - Drank tea with Sue in the kitchen ——- ```
- williamcotton 2y ago
  Me: Joe drove Sue to university. Afterwords he drove home again and drank a tea with her in the kitchen. Explain. ChatGPT: This statement seems to contain an ambiguity regarding Sue's location, suggesting some confusion about the timeline of events. Here are a few possibilities that could clarify the scenario: 1. *Sue Returns with Joe*: - Joe drove Sue to university. - Sue either decided not to stay at university or completed what she needed to do quickly. - Joe drove Sue back home. - They both drank tea together in the kitchen. 2. *Misstatement or Mistake*: - The statement might contain a mistake, implying that Joe and Sue met at home later but doesn't clearly explain how Sue got back from the university. - It could be intended to mean that Joe drove Sue to university and then drove home alone, and Sue somehow returned home later to join Joe for tea. 3. *Joe Visits University*: - Joe drove Sue to university. - Joe stayed at the university, possibly waiting for Sue to finish. - After she was done, they both drove back home together. - They then drank tea in the kitchen. Based on the original statement, the first scenario seems most plausible, suggesting Sue quickly returned home with Joe.

empath75 2y ago

There's a danger in flatly stating what it's doing _without testing your assumptions_. Did you actually try and trip it up and see if it falls for it? Reword his prompt in different ways to get it to fail?