7 ms·

You can see it in various tasks where the answer for a text generator would be incorrect, yet simple reasoning or understanding yields the correct answer. For e

by jfim 3y ago

You can see it in various tasks where the answer for a text generator would be incorrect, yet simple reasoning or understanding yields the correct answer. For example, the everything fits in the boat variant of the farmer across the river problem: https://chat.openai.com/share/7d2de1c4-1cd6-4d7d-97e4-645830c0647a https://chat.openai.com/share/7d2de1c4-1cd6-4d7d-97e4-645830...

throwaway4233 3y ago
I do not think that answer would correctly solve the original riddle. The LLM blindly just gave an answer without raising the question of which pair of objects together would be a problem if left alone.
- jfim 3y ago
  Good catch, it actually leaves the goat with the cabbage in the answer above.
fragmede 3y ago
It figures it out if you point out the difference much like a human would when faced with a riddle they didn't get. https://chat.openai.com/share/34df3eb0-c41c-4c5b-80d2-48f5d00d200a https://chat.openai.com/share/34df3eb0-c41c-4c5b-80d2-48f5d0... I'll be honest, the only reason I knew to read your variation of the farmer-boat problem carefully is because you set me up for success - I knew it wasn't going to be the original. Curiously, I haven't been able to get it to one-shot the question, even by prefixing it with instructions.
- godelski 3y ago
  Except the problem here is that you spoiled the solution. It is quite difficult to provide hints that don't spoil. Or rather, don't provide hints but rather tell it that it is wrong. Here, I'll demonstrate it. First I'll prod it with vague responses about it just generally being wrong. Trying to leak no information to the model. We'll try to slowly add a bit more and then use your followups. Notice that the model cannot escape the overfit regime without your strong hints. You told the model specifically what to consider. You told it that it is a trick question of a trick question. This is not how a human would handle the situation. To my followups they wouldn't spit out the same answers. They'd actually likely followup with questions if they were confused. Which is a behavior I've never seen from an LLM: asking clarifying questions. https://chat.openai.com/share/57ab9bca-326d-45cb-9257-7fb8c2333a73 https://chat.openai.com/share/57ab9bca-326d-45cb-9257-7fb8c2...
godelski 3y ago
I love this example! I've been using a different one and GPT has been getting better at it (unconvinced they aren't training on my reporting that it gets the wrong answer. Or that I've posted this question dozens of times here and some have gotten pretty high up. But it's been a year now and we're still in stochastic phase). The question I ask is variants of "Which weighs more, a pound of feathers or a kilogram of bricks?" Similar to yours it is just a slight variant on a standard logic puzzle. For those wondering why ask these questions, we're testing for overfitting. Yes, you can overfit even if your training and validation curves don't diverge. If the model fails these questions, they've clearly overfit. But the solution space is very large and complex, so overfitting in one regime doesn't mean it didn't underfit another. GPT 3.5: https://chat.openai.com/share/328cc39d-7fb3-4726-92a2-29437c962e0c https://chat.openai.com/share/328cc39d-7fb3-4726-92a2-29437c... LLaMA 2 70B chat: https://hf.co/chat/r/KX5H3P2 https://hf.co/chat/r/KX5H3P2 Falcon 180B Chat: https://hf.co/chat/r/V8XxdYh https://hf.co/chat/r/V8XxdYh None of these models can consistently get the answer right. GPT used to actually explain to me the difference between a pound and kilogram (correctly) and then use that answer to justify why they are the same. Such an answer is very clearly a demonstration of a lack of understanding as it isn't even remotely self consistent. LLMs are powerful and amazing technologies. But we can also critique them. Hyping up models hinders the ability to improve them because every thing (LLMs, humans, governments, whatever) has limitations and is worthy of critique. But criticism isn't saying something is garbage and too many people confuse this.
- jfim 3y ago
  Agreed, there's definitely a lot of value in LLMs even if they don't really have an understanding of the world.
  - godelski 3y ago
    It is quite interesting to me that people latch onto the world model so much as if it is highly significant or a lack of a world model invalidates LLMs. Personally I still believe they're magical (in the proverbial sense) and amazing tools even though I also believe they are just complex stochastic machines that have very little understanding of the things they are doing. That just tells me how cool and amazing that statistics is. I honestly don't know anyone that thinks LLMs are useless things, despite knowing a large number of people that are highly critical of them. But those are very different metrics. I don't know why we can't have nuance in things, why they need to either be a gift from god or a pile of shit.
glitchcrab 3y ago
But that isn't a correct solution - it leaves the cabbage with the goat.
- jfim 3y ago
  The correct solution is that everything fits in the boat, the farmer can just go across the river with everything at once.

by jfim 3y ago

throwaway4233 3y ago

I do not think that answer would correctly solve the original riddle. The LLM blindly just gave an answer without raising the question of which pair of objects together would be a problem if left alone.

jfim 3y ago
Good catch, it actually leaves the goat with the cabbage in the answer above.

fragmede 3y ago

It figures it out if you point out the difference much like a human would when faced with a riddle they didn't get. https://chat.openai.com/share/34df3eb0-c41c-4c5b-80d2-48f5d00d200a https://chat.openai.com/share/34df3eb0-c41c-4c5b-80d2-48f5d0... I'll be honest, the only reason I knew to read your variation of the farmer-boat problem carefully is because you set me up for success - I knew it wasn't going to be the original. Curiously, I haven't been able to get it to one-shot the question, even by prefixing it with instructions.

godelski 3y ago
Except the problem here is that you spoiled the solution. It is quite difficult to provide hints that don't spoil. Or rather, don't provide hints but rather tell it that it is wrong. Here, I'll demonstrate it. First I'll prod it with vague responses about it just generally being wrong. Trying to leak no information to the model. We'll try to slowly add a bit more and then use your followups. Notice that the model cannot escape the overfit regime without your strong hints. You told the model specifically what to consider. You told it that it is a trick question of a trick question. This is not how a human would handle the situation. To my followups they wouldn't spit out the same answers. They'd actually likely followup with questions if they were confused. Which is a behavior I've never seen from an LLM: asking clarifying questions. https://chat.openai.com/share/57ab9bca-326d-45cb-9257-7fb8c2333a73 https://chat.openai.com/share/57ab9bca-326d-45cb-9257-7fb8c2...

godelski 3y ago

I love this example! I've been using a different one and GPT has been getting better at it (unconvinced they aren't training on my reporting that it gets the wrong answer. Or that I've posted this question dozens of times here and some have gotten pretty high up. But it's been a year now and we're still in stochastic phase). The question I ask is variants of "Which weighs more, a pound of feathers or a kilogram of bricks?" Similar to yours it is just a slight variant on a standard logic puzzle. For those wondering why ask these questions, we're testing for overfitting. Yes, you can overfit even if your training and validation curves don't diverge. If the model fails these questions, they've clearly overfit. But the solution space is very large and complex, so overfitting in one regime doesn't mean it didn't underfit another. GPT 3.5: https://chat.openai.com/share/328cc39d-7fb3-4726-92a2-29437c962e0c https://chat.openai.com/share/328cc39d-7fb3-4726-92a2-29437c... LLaMA 2 70B chat: https://hf.co/chat/r/KX5H3P2 https://hf.co/chat/r/KX5H3P2 Falcon 180B Chat: https://hf.co/chat/r/V8XxdYh https://hf.co/chat/r/V8XxdYh None of these models can consistently get the answer right. GPT used to actually explain to me the difference between a pound and kilogram (correctly) and then use that answer to justify why they are the same. Such an answer is very clearly a demonstration of a lack of understanding as it isn't even remotely self consistent. LLMs are powerful and amazing technologies. But we can also critique them. Hyping up models hinders the ability to improve them because every thing (LLMs, humans, governments, whatever) has limitations and is worthy of critique. But criticism isn't saying something is garbage and too many people confuse this.

jfim 3y ago
Agreed, there's definitely a lot of value in LLMs even if they don't really have an understanding of the world.
- godelski 3y ago
  It is quite interesting to me that people latch onto the world model so much as if it is highly significant or a lack of a world model invalidates LLMs. Personally I still believe they're magical (in the proverbial sense) and amazing tools even though I also believe they are just complex stochastic machines that have very little understanding of the things they are doing. That just tells me how cool and amazing that statistics is. I honestly don't know anyone that thinks LLMs are useless things, despite knowing a large number of people that are highly critical of them. But those are very different metrics. I don't know why we can't have nuance in things, why they need to either be a gift from god or a pile of shit.

glitchcrab 3y ago

But that isn't a correct solution - it leaves the cabbage with the goat.

jfim 3y ago
The correct solution is that everything fits in the boat, the farmer can just go across the river with everything at once.