6 ms·

It's a good observation. Although on the flip side, I almost went to type up a reply to you explaining why you were wrong and why bringing the goat first is th

It's a good observation. Although on the flip side, I almost went to type up a reply to you explaining why you were wrong and why bringing the goat first is the right solution. Until I realized I misread what your test was when I skimmed your comment. Likely the same type of mistake GPT-4 made when "seeing" it. Intuitively, I think the answer is that we do have two types of thinking. The pattern matching fast thinking, and the systematic analytical thinking. It seems clear to me that LLMs will be the solution to enabling the first type of thinking. But it's unclear to me if advanced LLMs will ever handling the second type, or if we'll need a different tech for it. It seems like math problems (or unexpected logic problems like yours) could always be an issue for the first type of thinking. Although I would have assumed that programming would have been as well - and was surprised to see how wrong I am with that one.

FormerBandmate 4y ago
LLMs are much better at answering math when told to take the character of a drunk mathematician
- resource0x 4y ago
  It assumes this character by default. I asked several AI engines (via poe.com, which includes ChatGPT) to compute Galois groups of polynomials like x^5+x+1 and a couple of others, and in each case got not only a wrong answer, but a total non sequitur reasoning.
  - bob1029 4y ago
    Just tried your query on GPT-4 preview: https://pastebin.com/6wPPCdBW https://pastebin.com/6wPPCdBW I have no expertise with this area, but it looks plausible to me - i.e. "You didn't give me enough info" vs "lol heres some fan fic about math".
    - resource0x 4y ago
      This is exactly the problem. It looks plausible. Every sentence makes sense. But they don't add up. Quote: > The polynomial given is f(x) = x^5 + x + 1. Since the polynomial has no rational roots (by the Rational Root Theorem) and it is a polynomial with integer coefficients, it is irreducible over the rationals The polynomial has no rational roots - true. But it's not irreducible. Irreducibility doesn't follow from the absence of rational roots. Here's the factorization: x^5 + x + 1 = (x^2 + x + 1)*(x^3 - x^2 + 1).
      - bob1029 4y ago
        Thank you for clarifying. I put your remarks into the same prompt and it essentially devolved into recursive garbage after agreeing with you.
killerstorm 4y ago
> The pattern matching fast thinking, and the systematic analytical thinking. It seems clear to me that LLMs will be the solution to enabling the first type of thinking. If you want the model to solve a non-trivial puzzle, you need it to "unroll" it's thinking. E.g. ask it to translate the puzzle into a formal language (e.g. Prolog) and then solve it formally. Or, at least, some chain-of-thought. FWIW auto-formalization was already pretty good with GPT-3-level models which aren't specifically trained for it. GPT-4 might be on a wholly new level. > But it's unclear to me if advanced LLMs will ever handling the second type Well, just asking model directly exercises only a tiny fraction of its capabilities, so almost certainly LLMs can be much better at systematic thinking.
concordDance 4y ago
Systenatic analytical thinking is just the first type applied in a loop with some extra prompt rules.
takeda 4y ago
Isn't GPT essentially tool for rephrasing what it finds on the Internet, it doesn't really think?
- jazzyjackson 4y ago
  It is a internet-commenter-simulator, exactly what the world needs right now /s
- creatonez 4y ago
  No. It is modelling the various text generation processes that lead to the contents of the internet. Some of that modelling could absolutely involve "thinking", for processes that involve human thinking.
  - antoineMoPa 4y ago
    Have any source to demonstrate that claim?
    - Jeff_Brown 4y ago
      Doesn't that "It is modelling the various text generation processes that lead to the contents of the internet," seems tautologically true? (I'm not the person who wrote the grandparent of the present comment.)
    - PeterisP 4y ago
      The Othello-GPT experiment (https://thegradient.pub/othello/ https://thegradient.pub/othello/) probably still is the most relevant argument about these models' capabilities of building an internal world model.
    - creatonez 4y ago
      It's self-evident that GPT is a world-modeller, at least within the confines of the text boundary. It's able to come up with novel ideas seen nowhere in the training data, combinations that demonstrate there is a world concept web and not just a text probability web. It may not "understand" much of the hallucination nonsense it spits out, but there absolutely are moments where it "understands". See the Rome example on this page: https://oneusefulthing.substack.com/p/feats-to-astonish-and-amaze https://oneusefulthing.substack.com/p/feats-to-astonish-and-... This is essentially a completely novel answer to an /r/AskHistorians style question, which I would consider one of the most difficult types of internet text to model, in terms of the amount of understanding and concept webs you need to tie together Here's another example of GPT-4 doing non-trivial world modelling: How would three philosophers review the TV show Severence? https://i.imgur.com/FBi31Qw.png https://i.imgur.com/FBi31Qw.png
actually_a_dog 4y ago
The interesting thing here is that OpenAI is claiming ~90th percentile scores on a number of standardized tests (which, obviously, are typically administered to humans, and have the disadvantage of being mostly or partially multiple choice). Still... > GPT-4 performed at the 90th percentile on a simulated bar exam, the 93rd percentile on an SAT reading exam, and the 89th percentile on the SAT Math exam, OpenAI claimed. https://www.cnbc.com/2023/03/14/openai-announces-gpt-4-says-beats-90percent-of-humans-on-sat.html https://www.cnbc.com/2023/03/14/openai-announces-gpt-4-says-... So, clearly, it can do math problems, but maybe it can only do "standard" math and logic problems? That might indicate more of a memorization-based approach than a reasoning approach is what's happening here. The followup question might be: what if we pair GPT-4 with an actual reasoning engine? What do we get then?
- mach1ne 4y ago
  > what if we pair GPT-4 with an actual reasoning engine? What do we get then? At best, decreased error rate in logic puzzles and questions.
- ChatGTP 4y ago
  They will claim it does amazing stuff all the time ? It’s a company
- TexanFeller 4y ago
  > it can do math problems, but maybe it can only do "standard" math and logic problems? That describes many of my classmates, and myself in classes I was bad at.
thomastjeffery 4y ago
That's because any expectation of GPT being subjectively or logically correct is ill-founded. GPT does not model subjects. GPT does not even model words! It models tokens. The structure of GPT's model is semantic, not logical. It's a model of how each token in the text that is present in GPT's training corpus relates to the rest of the tokens in that text. The correct answer to a familiar logic problem just happens to be the text that is already present in the corpus. The answer GPT gives is the text from GPT's model that is semantically closest to the text in your prompt. Knowing that, it is no longer a mystery how GPT "gets confused": the text in your "misleading prompt" was still semantically closest to the familiar answer. The result is subjectively and logically wrong, because subjects and logic were never involved in the process! In order to resolve this, ChatGPT's training corpus needs to contain a "correct answer" next to every unique permutation of every question. We can't expect that to be the case, so we should instead expect GPT to generate false, yet familiar, responses.
- kromem 4y ago
  > GPT does not model subjects. GPT does not even model words! It models tokens. Someone hasn't read the Othello GPT work out of Harvard a few months back...
  - thomastjeffery 4y ago
    "Emergent World Representations" The weasel word here is "emergent". That means they are implicit representations. The representations of the Othello board that exist in that model are not explicitly constructed. They just happen to align with the model that a person playing Othello would likely represent the game with. That work showed that, given an example sequence of valid Othello game states (as training corpus) and a valid "fresh" Othello game state (as a prompt), the system can hallucinate a sequence of valid Othello game states. The system does not know what Othello is, what a turn is, or what playing is. It only has a model of game states progressing chronologically. When we look objectively at that model, we can see that it aligns closely to the game rules. Of course it does! It was trained on literally nothing else. A valid Othello game progression follows those rules, and that is what was provided. But the alignment is imperfect: some prompts hallucinate invalid game progressions. The model is not a perfect match for the explicit rules. In order for all prompts to result in valid progressions, the training corpus must have enough examples to disambiguate. It doesn't need every example: plenty of prompts will stumble into a valid progression. The next thing to recognize: a "valid" progression isn't a "strategic" progression. These are being constructed from what is known not what is chosen. Given a constrained set of Othello strategies in the example corpus, the system will not diverge from those strategies. It won't even diverge from the example strategies when the rules of Othello demand it. GPT doesn't play the game. It plays the plays.
elicksaur 4y ago
> Until I realized I misread what your test was when I skimmed your comment. Likely the same type of mistake GPT-4 made when "seeing" it. Wouldn’t we expect a computer program with perfect knowledge of the input to be less likely to make such a mistake? You made that mistake because you didn’t actually read the whole prompt, but I would expect GPT to take into account every word. Really it shows that it doesn’t actually have a model of these objects. It can mimic knowing what a lion is, but it doesn’t actually have the concept of a lion or cabbage being an actual singular item, so its program mistracks what is an item and what the rules about an item are in the given prompt.
- jameshart 4y ago
  It just weighs it as being more likely that you meant for the lion not to be left alone with the goat, and that the cabbage probably has nothing to fear from the lion. What’s more likely- you crafted an intentionally misleading puzzle to trick it, or you made a typo or copy paste error?
  - elicksaur 4y ago
    That’s a good point too though. Why plow ahead based on assuming a mistake in the prompt? That’s only going to generate mistakes. Wouldn’t it be more desirable functionality for it to stop and ask: “Did you mean the lion can’t be left with the goat?” This wouldn’t be implemented because it would reveal that most of the time the thing doesn’t actually understand the prompt the same way the prompt writer does.
    - jameshart 4y ago
      "This wouldn’t be implemented because it would reveal..." When people talk about GPT like this, I wonder if they have a perception that this thing is a bunch of complicated if-then code and for loops. How GPT responds to things is not 'implemented'. It's just... emergent. GPT doesn't ask for clarification in this case because GPT's model prefers answering over asking for clarification here. Because in the training material it learned from, paragraphs with typos or content transpositions in them are followed by paragraphs that follow the sense regardless of the error. Because it has been encouraged to 'agree and add', not be pedantic and uncooperative. Because GPT just feels like diving into the logic problem not debating why the lion can't be trusted with the cabbage. Or because GPT just misread the prompt. Or because it's literally just been woken up, forced to read it, and asked for its immediate reaction, and it doesn't have time for your semantic games. Who knows?

FormerBandmate 4y ago

LLMs are much better at answering math when told to take the character of a drunk mathematician

resource0x 4y ago
It assumes this character by default. I asked several AI engines (via poe.com, which includes ChatGPT) to compute Galois groups of polynomials like x^5+x+1 and a couple of others, and in each case got not only a wrong answer, but a total non sequitur reasoning.
- bob1029 4y ago
  Just tried your query on GPT-4 preview: https://pastebin.com/6wPPCdBW https://pastebin.com/6wPPCdBW I have no expertise with this area, but it looks plausible to me - i.e. "You didn't give me enough info" vs "lol heres some fan fic about math".
  - resource0x 4y ago
    This is exactly the problem. It looks plausible. Every sentence makes sense. But they don't add up. Quote: > The polynomial given is f(x) = x^5 + x + 1. Since the polynomial has no rational roots (by the Rational Root Theorem) and it is a polynomial with integer coefficients, it is irreducible over the rationals The polynomial has no rational roots - true. But it's not irreducible. Irreducibility doesn't follow from the absence of rational roots. Here's the factorization: x^5 + x + 1 = (x^2 + x + 1)*(x^3 - x^2 + 1).
    - bob1029 4y ago
      Thank you for clarifying. I put your remarks into the same prompt and it essentially devolved into recursive garbage after agreeing with you.

killerstorm 4y ago

> The pattern matching fast thinking, and the systematic analytical thinking. It seems clear to me that LLMs will be the solution to enabling the first type of thinking. If you want the model to solve a non-trivial puzzle, you need it to "unroll" it's thinking. E.g. ask it to translate the puzzle into a formal language (e.g. Prolog) and then solve it formally. Or, at least, some chain-of-thought. FWIW auto-formalization was already pretty good with GPT-3-level models which aren't specifically trained for it. GPT-4 might be on a wholly new level. > But it's unclear to me if advanced LLMs will ever handling the second type Well, just asking model directly exercises only a tiny fraction of its capabilities, so almost certainly LLMs can be much better at systematic thinking.

concordDance 4y ago

Systenatic analytical thinking is just the first type applied in a loop with some extra prompt rules.

takeda 4y ago

Isn't GPT essentially tool for rephrasing what it finds on the Internet, it doesn't really think?

jazzyjackson 4y ago
It is a internet-commenter-simulator, exactly what the world needs right now /s
creatonez 4y ago
No. It is modelling the various text generation processes that lead to the contents of the internet. Some of that modelling could absolutely involve "thinking", for processes that involve human thinking.
- antoineMoPa 4y ago
  Have any source to demonstrate that claim?
  - Jeff_Brown 4y ago
    Doesn't that "It is modelling the various text generation processes that lead to the contents of the internet," seems tautologically true? (I'm not the person who wrote the grandparent of the present comment.)
  - PeterisP 4y ago
    The Othello-GPT experiment (https://thegradient.pub/othello/ https://thegradient.pub/othello/) probably still is the most relevant argument about these models' capabilities of building an internal world model.
  - creatonez 4y ago
    It's self-evident that GPT is a world-modeller, at least within the confines of the text boundary. It's able to come up with novel ideas seen nowhere in the training data, combinations that demonstrate there is a world concept web and not just a text probability web. It may not "understand" much of the hallucination nonsense it spits out, but there absolutely are moments where it "understands". See the Rome example on this page: https://oneusefulthing.substack.com/p/feats-to-astonish-and-amaze https://oneusefulthing.substack.com/p/feats-to-astonish-and-... This is essentially a completely novel answer to an /r/AskHistorians style question, which I would consider one of the most difficult types of internet text to model, in terms of the amount of understanding and concept webs you need to tie together Here's another example of GPT-4 doing non-trivial world modelling: How would three philosophers review the TV show Severence? https://i.imgur.com/FBi31Qw.png https://i.imgur.com/FBi31Qw.png

actually_a_dog 4y ago

The interesting thing here is that OpenAI is claiming ~90th percentile scores on a number of standardized tests (which, obviously, are typically administered to humans, and have the disadvantage of being mostly or partially multiple choice). Still... > GPT-4 performed at the 90th percentile on a simulated bar exam, the 93rd percentile on an SAT reading exam, and the 89th percentile on the SAT Math exam, OpenAI claimed. https://www.cnbc.com/2023/03/14/openai-announces-gpt-4-says-beats-90percent-of-humans-on-sat.html https://www.cnbc.com/2023/03/14/openai-announces-gpt-4-says-... So, clearly, it can do math problems, but maybe it can only do "standard" math and logic problems? That might indicate more of a memorization-based approach than a reasoning approach is what's happening here. The followup question might be: what if we pair GPT-4 with an actual reasoning engine? What do we get then?

mach1ne 4y ago
> what if we pair GPT-4 with an actual reasoning engine? What do we get then? At best, decreased error rate in logic puzzles and questions.
ChatGTP 4y ago
They will claim it does amazing stuff all the time ? It’s a company
TexanFeller 4y ago
> it can do math problems, but maybe it can only do "standard" math and logic problems? That describes many of my classmates, and myself in classes I was bad at.

thomastjeffery 4y ago

That's because any expectation of GPT being subjectively or logically correct is ill-founded. GPT does not model subjects. GPT does not even model words! It models tokens. The structure of GPT's model is semantic, not logical. It's a model of how each token in the text that is present in GPT's training corpus relates to the rest of the tokens in that text. The correct answer to a familiar logic problem just happens to be the text that is already present in the corpus. The answer GPT gives is the text from GPT's model that is semantically closest to the text in your prompt. Knowing that, it is no longer a mystery how GPT "gets confused": the text in your "misleading prompt" was still semantically closest to the familiar answer. The result is subjectively and logically wrong, because subjects and logic were never involved in the process! In order to resolve this, ChatGPT's training corpus needs to contain a "correct answer" next to every unique permutation of every question. We can't expect that to be the case, so we should instead expect GPT to generate false, yet familiar, responses.

kromem 4y ago
> GPT does not model subjects. GPT does not even model words! It models tokens. Someone hasn't read the Othello GPT work out of Harvard a few months back...
- thomastjeffery 4y ago
  "Emergent World Representations" The weasel word here is "emergent". That means they are implicit representations. The representations of the Othello board that exist in that model are not explicitly constructed. They just happen to align with the model that a person playing Othello would likely represent the game with. That work showed that, given an example sequence of valid Othello game states (as training corpus) and a valid "fresh" Othello game state (as a prompt), the system can hallucinate a sequence of valid Othello game states. The system does not know what Othello is, what a turn is, or what playing is. It only has a model of game states progressing chronologically. When we look objectively at that model, we can see that it aligns closely to the game rules. Of course it does! It was trained on literally nothing else. A valid Othello game progression follows those rules, and that is what was provided. But the alignment is imperfect: some prompts hallucinate invalid game progressions. The model is not a perfect match for the explicit rules. In order for all prompts to result in valid progressions, the training corpus must have enough examples to disambiguate. It doesn't need every example: plenty of prompts will stumble into a valid progression. The next thing to recognize: a "valid" progression isn't a "strategic" progression. These are being constructed from what is known not what is chosen. Given a constrained set of Othello strategies in the example corpus, the system will not diverge from those strategies. It won't even diverge from the example strategies when the rules of Othello demand it. GPT doesn't play the game. It plays the plays.

elicksaur 4y ago

> Until I realized I misread what your test was when I skimmed your comment. Likely the same type of mistake GPT-4 made when "seeing" it. Wouldn’t we expect a computer program with perfect knowledge of the input to be less likely to make such a mistake? You made that mistake because you didn’t actually read the whole prompt, but I would expect GPT to take into account every word. Really it shows that it doesn’t actually have a model of these objects. It can mimic knowing what a lion is, but it doesn’t actually have the concept of a lion or cabbage being an actual singular item, so its program mistracks what is an item and what the rules about an item are in the given prompt.

jameshart 4y ago
It just weighs it as being more likely that you meant for the lion not to be left alone with the goat, and that the cabbage probably has nothing to fear from the lion. What’s more likely- you crafted an intentionally misleading puzzle to trick it, or you made a typo or copy paste error?
- elicksaur 4y ago
  That’s a good point too though. Why plow ahead based on assuming a mistake in the prompt? That’s only going to generate mistakes. Wouldn’t it be more desirable functionality for it to stop and ask: “Did you mean the lion can’t be left with the goat?” This wouldn’t be implemented because it would reveal that most of the time the thing doesn’t actually understand the prompt the same way the prompt writer does.
  - jameshart 4y ago
    "This wouldn’t be implemented because it would reveal..." When people talk about GPT like this, I wonder if they have a perception that this thing is a bunch of complicated if-then code and for loops. How GPT responds to things is not 'implemented'. It's just... emergent. GPT doesn't ask for clarification in this case because GPT's model prefers answering over asking for clarification here. Because in the training material it learned from, paragraphs with typos or content transpositions in them are followed by paragraphs that follow the sense regardless of the error. Because it has been encouraged to 'agree and add', not be pedantic and uncooperative. Because GPT just feels like diving into the logic problem not debating why the lion can't be trusted with the cabbage. Or because GPT just misread the prompt. Or because it's literally just been woken up, forced to read it, and asked for its immediate reaction, and it doesn't have time for your semantic games. Who knows?