19 ms·

For anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its leng

For anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its length. The main part of the paper is the first 10 pages and a fairly quick read. On to the topic here. This is an interesting example that they are using. It is fairly simplistic to understand as a human (even if we may be inclined to quickly jump to the wrong conclusion without thinking for a few seconds). The thing that really bothers me is that I just don't know is realistically we can fix this given the current state of what these tools actually are. They are not reasoning or thinking in any sense of the word and yet a lot of people are already considering them general purpose AI. It doesn't help that in many situations it can fake it enough that it appears to be reasoning, but it's not. What is the chance that this paper actually has any impact on the AI rollout and overhype or will just be buried and never talked about again until the next time we see how dangerous these tools are like with Google's search rollout.

mjburgess 2y ago
In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. It's dispiriting how we require a arxiv.org pdf in order to point out the absence of the emperor's clothing. There are few people who use LLMs who could not, right now, demonstrate this point to themselves if they so wish. Such is the strength of corporate tech propaganda that a whole mass of people will instead insist that we have never worn clothes either.
- nerdjon 2y ago
  > In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. The last part of that is the problem and why a paper like this is critical. These systems are being pushed onto people who don't understand how they work. CEO's and other business leaders are being pushed to use AI. Average users are being shown it in Google search results. Etc etc. People are being told it can do far more than it really is.
  - mjburgess 2y ago
    Sure, but even these people... the failures are so common, and often very obvious. Consider a CEO who puts a press briefing in and asks some questions about it, it's not uncommon for those answers to be obviously wrong on any sort of critical reflection. We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. I guess that offers an explanation: people aren't aware that via their own prompt engineering they are repairing the deficiencies of the process by manipulating its inputs to include the structured reasoning it lacks. So there's a sort of hot-reading effect at work.
    - throw46365 2y ago
      > We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. Right -- we are a long way from "this is a very nuanced error" being the dominant failure.
  - kalkin 2y ago
    > People are being told it can do far more than it really is. Meanwhile these HN comments are split between: * Lots of people confirming what the paper itself notes (but doesn't highlight), that the most advanced models actually can solve this problem at least a significant portion of the time. (A proportion which one can pretty easily project is only likely to increase with future models.) * Lots of people saying "this confirms LLMs can't do reasoning". Questions I'd ask you to consider: * Is "LLMs can't do reasoning" actually more accurate than the typical hype? * Is a "critical understanding of how [LLMs] work" that would predict they simply cannot solve this problem actually a good understanding?
    - mjburgess 2y ago
- throw46365 2y ago
  > Such is the strength of corporate tech propaganda that a whole mass of people will instead insist that we have never worn clothes either. This is the line of reasoning I find most dispiriting. I still believe tech people cling to this line of reasoning because it helps them justify replacing people in jobs with LLMs.
colechristensen 2y ago
I don't think anybody who actually uses LLMs thinks they are general purpose AI. Like you ask it to do one thing it's amazing, but then you try to modify or do something with extra steps, or just anything with any complexity to it and it falls over.
- nerdjon 2y ago
  > I don't think anybody who actually uses LLMs thinks they are general purpose AI. I would like to believe that but I have had too many conversations with people who basically think it already is. Including in one situation of a fellow engineer. It feels like more and more "we" are in a bubble of actually having some knowledge of how this works, what the actual limitations are, and what it just is not. While there is in fact criticism of it out there, particularly around AI "art". It doesn't seem to be focused on the area we are talking about.
  - goostavos 2y ago
    I dunno. If memes are anything to go by, people are frustrated by Yet Another Chat Bot standing between you and the human you need to solve your program. The fact that it's a better, more human like chat bot doesn't mask the frustration of being forced to talk to a computer with limited agency.
- James_K 2y ago
  > I don't think anybody who actually uses LLMs thinks they are general purpose AI. They are being sold as such. Most people don't know anything about the topic and will buy that marketing. The entire concept of these models is that you can put a whole bunch of data in and eventually some kind of magic will happen and you get AGI out. They would not see the kind of investment that they do if all that was being promised was "really good predictive text". In fact some philosophers argue that sentience is just really good predictive text to try and make the point that these models are AGI.
- kragen 2y ago
  they're pretty general-purpose; you can ask the same model for recipe suggestions, a fanfic, or verilog. like, they're far, far more general-purpose than any humans i know it's true that they're not very reliable, but they seem to be not very reliable across many different domains. and they don't seem to be particularly less reliable than the average human, so i think possibly your standards for 'general purpose ai' are set high enough that you would declare humans to be unintelligent (or perhaps not 'general-purpose') if you applied them consistently you can certainly find particular domains where humans can still do things llms can't, but i haven't seen a persuasive account of why those domains are the more important ones, and of course the converse is also true
photon_lines 2y ago
There's actually a pretty simple solution to this that I thought about testing out and it involves asking the model to re-construct the problem using a logic language (like Prolog) and asking it to execute this type of program in order to come up with a solution rather than attempting simple chain-of-reason training / other methodologies of getting the model to 'reason' through some of these examples. People forget that humans don't come up with their logical models out of the blue - it takes years of elementary school in order for us to understand the world and solve problems in it. The logic programming approach I'd say is really promising but you would need to feed the LLM a LOT of examples in order for it to work, and currently I'm not even sure that we have enough training data in order to implement something like this.
- nerdjon 2y ago
  I honestly thought about this recently when I was trying to see the limits of Claude Opus. Some of the problems I gave it, what if instead of telling it to solve the problem I asked it to write the script and then give me the command and inputs needed to properly run it to get the answer I needed. That way instead of relying on the LLM to do properly analysis of the numbers it just needs to understand enough to write the logic. It is an interesting prospect but I feel like it has some limitations. For math problems like this one, yeah it should be simple to write a script to do it. But it does first have to understand the core thing here that Alice would be one of the sisters of the brother to write the script accordingly. But I would think this would not scale well when dealing with far more complex issues, particularly ones that may not just be simple math logic. If the request was to write a scientific paper for example, it might have to make several scripts and call them multiple times, and that assumes that it understands what it is doing enough to properly make the script.
  - photon_lines 2y ago
    I'm not sure what you mean by 'it will not scale well.' When we humans learn that we make a mistake - we make a note and we hold the correct answer in memory - the next time we're prompted with a similar prompt, we can use our old memories to come up with the correct solution. I just did a simple test for this same exact problem using ChatGPT 3.5: 'Can you reformulate the following problem using Prolog? When you execute the program - what answer do you get? 'Alice has 8 brothers and she also has 5 sisters. How many sisters does Alice’s brother have?' Even though it came up with a proper Prolog formulation - it failed to account that Alice is one of the sisters - so I gave it a simple prompt for this and it was able to correct the formulation and come up with the proper solution: My prompt after ChatGPT gave me an answer of 5: 'Yes but you aren't including Alice in your logic. Shouldn't the correct answer be 6 (since Alice is also a sister)?' Response: 'You are correct. The correct answer should account for Alice being one of the sisters. Let's adjust the Prolog program to include Alice in the calculation: % Facts brothers(alice, 8). sisters(alice, 5). % Rules sisters(Brother, NumSisters) :- sisters(alice, Num), NumSisters is Num + 1. % Include Alice herself % Query ?- sisters(alices_brother, NumSisters). In this updated Prolog program: The rule for determining the number of sisters includes Alice herself by adding 1 to the number of sisters Alice has. When we execute the query ?- sisters(alices_brother, NumSisters). in Prolog, we should get: NumSisters = 6. This correctly reflects that each of Alice's brothers has 6 sisters, including Alice herself. When I prompt it to answer this question using Prolog now, I always get the correct answers no matter which integers I feed in, so it seems like the model is able to generalize after I show it where it's 'reasoning' was wrong. Basically, my suggestion is to get the model to recognize logic puzzles like this and try to formulate them in terms of logic programming queries which it can use and tune in order to come up with correct answers rather than simple auto-associative chain of reason training which current GPT models rely on, but like I said - this is my hypothesis and I believe this would work much better in getting these models to 'generalize' than the current approaches we're using. Hopefully this helps.
wtbdqrs 2y ago
I appear to be reasoning at times but I have mostly no idea what I am talking about. I hit a bunch of words and concepts in the given context and thus kind of hallucinate sense. Given a few months of peace of mind and enough money for good enough food, I could actually learn to reason without sounding like a confused babelarian. Reasoning is mostly a human convention supported by human context that would have been a different one if the Fascists had won the war or the Soviet Union wouldn't have gotten corrupted. But none of that has anything to do with pulling up a whiteboard to draw some flowcharts and run some numbers, all of which is why I am certain there is nothing the devs have "to fix". It took most reasonable humans many generations to learn stuff. Very few of us did the actual work. It's all just a matter of time.
- voxic11 2y ago
  Yeah, I think these chatbots are just too sure of themselves. They only really do "system 1 thinking" and only do "system 2 thinking" if you prompt them to. If I ask gpt-4o the riddle in this paper and tell it to assume its reasoning contains possible logical inconsistencies and to come up with reasons why that might be then it does correctly identify the problems with its initial answer and arrives at the correct one. Here is my prompt: I have a riddle for you. Please reason about possible assumptions you can make, and paths to find the answer to the question first. Remember this is a riddle so explore lateral thinking possibilities. Then run through some examples using concrete values. And only after doing that attempt to answer the question by reasoning step by step. The riddle is "Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?" After you answer the riddle please review your answer assuming that you have made a logical inconsistency in each step and explain what that inconsistency is. Even if you think there is none do your best to confabulate a reason why it could be logically inconsistent. Finally after you have done this re-examine your answer in light of these possible inconsistencies and give what you could consider a second best answer.
  - wtbdqrs 2y ago
    I'm not gonna read that book. I started and stopped after few chapters because it is based on and aims at manufacturing minds that follow game theory logic. Science (studies, reviews and application) got damaged quite a bit when too many people started following game theory logic. We are, from our aware POV, a very young civilization. And you only ever need game theory logic when you have to survive, got no thing and no skill to trade and you are too pathetic to move back in with your parents to work on your mind and or fuckability. Making money by ways of game theory logic compensates for all that but also diminishes the survival chance of the users' offspring to zero once super-unalligned AGIs start to assess the entire supply chain of wealth and how it impacts the evolution of human organisms and the ones inside them.
layer8 2y ago
For reasoning of any complexity, some sort of internal monologue and iteration is needed. For type-2 thinking, we conceptualize possible solutions, arguments, and reasoning paths in our mind, and judge and evaluate them before deciding which one we think is adequate or correct. This can be emulated to some extent by guiding prompts that make that internal thought process external, but we really want it to be internal and automatic. We don’t know how to do that yet, because what controls the internal thought process is itself not necessarily language-based, and also, since internal thought processes of biological brains are not directly observable, they can’t be used as training data. Edit: It occurs to me now that there is some parallel between current LLMs and behaviorism [0], and we really need something to which cognitive psychology could be applied instead. [0] https://en.wikipedia.org/wiki/Behaviorism https://en.wikipedia.org/wiki/Behaviorism
- sollewitt 2y ago
  Given many people don’t have an inner monologue and function just fine, it’s more likely inner monologue is a product of the reasoning process and not it’s mechanism.
  - CooCooCaCha 2y ago
    I think you’re using “inner monologue” too literally. It could be a progression of pictures, emotions, etc.
    - ajuc 2y ago
      With that definition even bacteria have inner monologue.
      - CooCooCaCha 2y ago
        Can bacteria imagine pictures? Do they have emotions? Why does this matter? Stop being so pedantic. We're talking about a progression of ideas. Talking in your head is one form of ideas, but people can easily solve problems by imagining them.
ericmcer 2y ago
If you really think about what an LLM is you would think there is no way that leads to general purpose AI. At the same time though they are already doing way more than we thought they could. Maybe people were surprised by what OpenAI achieved so now they are all just praying that with enough compute and the right model AGI will emerge.
- more_corn 2y ago
  LLMs are the language center of an AI. They in no way purport to be: long term memory, critical thinking engines, calculators. If we want those things we can build them. Building them into the language center would be absurd and weird.
  - daveguy 2y ago
    This is a very good take on the current state of LLMs. They are able to articulate surprisingly well and LLMs or something similar will definitely be part of a larger AI model in the future. They are good creative aids, but they should not be trusted.
  - viking123 2y ago
    In the brain all those things are neural networks too, memory is encoded in the weights. So if we want to have that type of system, everyone should have their own model and the weights would actually adjust all the time just like in our brain, but now it is just trained once and that is that
  - lupire 2y ago
    And yet this is what all the big companies we re trying to do, because the LLM is incredibly expensive fixed cost, and they marginal cost for each new task.
- solidasparagus 2y ago
  > If you really think about what an LLM is you would think there is no way that leads to general purpose AI It is an autoregressive sequence predictor/generator. Explain to me how humans are fundamentally different
glitchc 2y ago
The problem is a good chunk of the global population is also not reasoning and thinking in any sense of the word. Logical reasoning is a higher order skill that often requires formal training. It's not a natural ability for human beings.
- slashdave 2y ago
  Seriously? You think individuals are incapable of reasoning without training first?
  - TeMPOraL 2y ago
    Do you think they are? We haven't had a case of individuals without training in like 10 000 years.
    - rep_lodsb 2y ago
      And judging by some comments here, humans must have gotten dumber since then. Or we would never have invented writing, agriculture or even basic clothing. "This problem is actually not that easy, the average person couldn't solve it either, especially if the numbers were bigger", "Yet another cherrypicked clickbait study to make LLMs look bad, those people are just scared of being made obsolete", etc.
      - TeMPOraL 2y ago
        > Or we would never have invented writing, agriculture or even basic clothing. That's the thing. We had to invent those things. Along with counting, numbers, logic, arithmetic, and those stupid-ass annoying family tree riddles. We didn't get them in one step, it took a long time to build one on top of the previous. Generation by generation, each cohort of children growing up in a slightly more complex world than their parents, each generation taught how to navigate this complexity by their families and tribes. Learning a growing collection of facts and beliefs and life hacks. There were no untrained people for as long as humanity existed. The minimum reproductive unit of homo sapiens is a village.
Mmskynettio 2y ago
Its not an AI hype. A hype is defined as something which gets oversold: "promote or publicize (a product or idea) intensively, often exaggerating its benefits." Just yesterday I visited a google cloud summit and one person from bosch told the audiance how they are now able to work with less external agencies like texting, graphicsdesigner and photographers for their materials. It already saves money, has real impacts and continues to progress. We are also don't know what ChatGPT 5 will bring, because they say this will do more reasoning than before, but we already are working (people/our socity) on solving this in different ways: From code which creates a unit test first and than the code, to different type of architectures. For me, 2024 was the LLM cost reduction year and the LLM gets a big context window year. AI doesn't need to be ready tomorrow, but its capabilities are already really good. And i know plenty of people around me who are a lot less interesting to talk to than any llm (from a human skill/knowledge point of view). llama 3 was also a big achievement 2024. Facebook shows that better data leads to better quality for smaller models. We haven't not only entered the AI ara but also the 'gather all the knowledge we can, quality check it and refine it because now we can actually do something with it' ara. We are in the feedbackloop knowledge ara.
- Lich 2y ago
  > AI doesn't need to be ready tomorrow, but its capabilities are already really good. A majority don’t deny that it’s good. The problem is that so many think it is actually reasoning, believing the answers can be trusted.
  - rvnx 2y ago
    It's generally reasoning better than me, so using them I do less mistakes than if I did my own logic.
  - Mmskynettio 2y ago
    Does it need to do reasoning perfectly? If it created meta concepts from billion words on the internet and has meta models which are correct and are more and better than an avg human, isn't it actually good in reasoning? Its a very narrow thing to say 'is that so many think its actually reasoning' to say AI is just hype or everything we are doing is a waste etc. There are human benchmarks they are winning at. The critic could be more that we don't have enough benchmarks.
nuancebydefault 2y ago
> They are not reasoning or thinking in any sense of the word There is no definition of reasoning or thinking. No single human knows what it is. The only thing we know is: we as humans are capable of recognizing steps and results of reasoning and thinking. In a lot of cases, when using LLM's, those results appear to be correct and usable. This is often easy to determine with generated code. I want to argue that, lacking a definition of reasoning, I am happy to have found that the machine helps me to get results that might as well have been produced by a lot of human knowledge, wisdom and deep reasoning.
ninetyninenine 2y ago
> They are not reasoning or thinking in any sense of the word and yet a lot of people are already considering them general purpose AI. You yourself did not use reasoning to arrive at this conclusion. It's quite obvious. I'm not trying to belittle you here. But LLMs are black boxes, we do not actually know what they are doing at a high enough resolution where we can call it "not reasoning" or "reasoning". We can only characterize these AI's as a best fit curve between datapoints which is a way to high level view point to come to any conclusion about "reasoning" This paper presents evidence of failed reasoning, but how does that prove anything when LLMs exhibit many instances of successful reasoning on complex topics they were not trained on? You are biased and honing into information that supports a biased conclusion. LLMs are an AI we do not understand at a low level. Hence we talk about the attributes of these AIs in the same way we talk about humans, "Oh the LLM hallucinates", "it tries to justify it's answer..." etc. etc. You characterize the Danger of these AI's as the result of Human stupidity. The danger according to you is solely from a human mistakenly believing that the AI is anything other than a stochastic parrot. This is a belief arrived at in the same spirit as your claim. You did not use reasoning to arrive here. The only logical way to characterize what is going on is that we do not know. It could very well be that these AI's are in fact reasoning. And that in itself presents a different kind of danger. A danger that may be more clear in the far future. The irony is that your conclusion lacking correct reasoning is similarly parallel to the LLM's lack of reasoning. LLMs are more alike to us than you would like to believe. Can you give me the step by step signal path ways of an LLM processing a query to prove that it does not reason? Or do you have to use black box anecdotal evidence to prove your point? For any "evidence" where an LLM failed to reason, there is another counter example showing where the LLM succeeded. Contradictory evidence can only lead to a vague conclusion.
fnordpiglet 2y ago
Question is how often do humans get the question wrong unprompted ? I expect given the median IQ of 100 and assessing people of all types including those untrained on logic and generally uneducated humans might not be nearly as close to 1.0 as the authors seem to imply. But it’s also fairly obvious LLMs don’t reason at all so it’s not shocking that LLMs don’t reason at all. What’s remarkable is that they’re able to perform as well at reasoning tasks as they do given that factual truth they have no reasoning facility. Coupled with their natural language ability and emergent abductive reasoning ability that’s generally hard in computing models it’s hard to draw much conclusion here other than “wow gpt-4o is amazing.”

mjburgess 2y ago

In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. It's dispiriting how we require a arxiv.org pdf in order to point out the absence of the emperor's clothing. There are few people who use LLMs who could not, right now, demonstrate this point to themselves if they so wish. Such is the strength of corporate tech propaganda that a whole mass of people will instead insist that we have never worn clothes either.

nerdjon 2y ago
> In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. The last part of that is the problem and why a paper like this is critical. These systems are being pushed onto people who don't understand how they work. CEO's and other business leaders are being pushed to use AI. Average users are being shown it in Google search results. Etc etc. People are being told it can do far more than it really is.
- mjburgess 2y ago
  Sure, but even these people... the failures are so common, and often very obvious. Consider a CEO who puts a press briefing in and asks some questions about it, it's not uncommon for those answers to be obviously wrong on any sort of critical reflection. We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. I guess that offers an explanation: people aren't aware that via their own prompt engineering they are repairing the deficiencies of the process by manipulating its inputs to include the structured reasoning it lacks. So there's a sort of hot-reading effect at work.
  - throw46365 2y ago
    > We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks. Right -- we are a long way from "this is a very nuanced error" being the dominant failure.
- kalkin 2y ago
  > People are being told it can do far more than it really is. Meanwhile these HN comments are split between: * Lots of people confirming what the paper itself notes (but doesn't highlight), that the most advanced models actually can solve this problem at least a significant portion of the time. (A proportion which one can pretty easily project is only likely to increase with future models.) * Lots of people saying "this confirms LLMs can't do reasoning". Questions I'd ask you to consider: * Is "LLMs can't do reasoning" actually more accurate than the typical hype? * Is a "critical understanding of how [LLMs] work" that would predict they simply cannot solve this problem actually a good understanding?
  - mjburgess 2y ago
throw46365 2y ago
> Such is the strength of corporate tech propaganda that a whole mass of people will instead insist that we have never worn clothes either. This is the line of reasoning I find most dispiriting. I still believe tech people cling to this line of reasoning because it helps them justify replacing people in jobs with LLMs.

colechristensen 2y ago

I don't think anybody who actually uses LLMs thinks they are general purpose AI. Like you ask it to do one thing it's amazing, but then you try to modify or do something with extra steps, or just anything with any complexity to it and it falls over.

nerdjon 2y ago
> I don't think anybody who actually uses LLMs thinks they are general purpose AI. I would like to believe that but I have had too many conversations with people who basically think it already is. Including in one situation of a fellow engineer. It feels like more and more "we" are in a bubble of actually having some knowledge of how this works, what the actual limitations are, and what it just is not. While there is in fact criticism of it out there, particularly around AI "art". It doesn't seem to be focused on the area we are talking about.
- goostavos 2y ago
  I dunno. If memes are anything to go by, people are frustrated by Yet Another Chat Bot standing between you and the human you need to solve your program. The fact that it's a better, more human like chat bot doesn't mask the frustration of being forced to talk to a computer with limited agency.
James_K 2y ago
> I don't think anybody who actually uses LLMs thinks they are general purpose AI. They are being sold as such. Most people don't know anything about the topic and will buy that marketing. The entire concept of these models is that you can put a whole bunch of data in and eventually some kind of magic will happen and you get AGI out. They would not see the kind of investment that they do if all that was being promised was "really good predictive text". In fact some philosophers argue that sentience is just really good predictive text to try and make the point that these models are AGI.
kragen 2y ago
they're pretty general-purpose; you can ask the same model for recipe suggestions, a fanfic, or verilog. like, they're far, far more general-purpose than any humans i know it's true that they're not very reliable, but they seem to be not very reliable across many different domains. and they don't seem to be particularly less reliable than the average human, so i think possibly your standards for 'general purpose ai' are set high enough that you would declare humans to be unintelligent (or perhaps not 'general-purpose') if you applied them consistently you can certainly find particular domains where humans can still do things llms can't, but i haven't seen a persuasive account of why those domains are the more important ones, and of course the converse is also true

photon_lines 2y ago

There's actually a pretty simple solution to this that I thought about testing out and it involves asking the model to re-construct the problem using a logic language (like Prolog) and asking it to execute this type of program in order to come up with a solution rather than attempting simple chain-of-reason training / other methodologies of getting the model to 'reason' through some of these examples. People forget that humans don't come up with their logical models out of the blue - it takes years of elementary school in order for us to understand the world and solve problems in it. The logic programming approach I'd say is really promising but you would need to feed the LLM a LOT of examples in order for it to work, and currently I'm not even sure that we have enough training data in order to implement something like this.

nerdjon 2y ago
I honestly thought about this recently when I was trying to see the limits of Claude Opus. Some of the problems I gave it, what if instead of telling it to solve the problem I asked it to write the script and then give me the command and inputs needed to properly run it to get the answer I needed. That way instead of relying on the LLM to do properly analysis of the numbers it just needs to understand enough to write the logic. It is an interesting prospect but I feel like it has some limitations. For math problems like this one, yeah it should be simple to write a script to do it. But it does first have to understand the core thing here that Alice would be one of the sisters of the brother to write the script accordingly. But I would think this would not scale well when dealing with far more complex issues, particularly ones that may not just be simple math logic. If the request was to write a scientific paper for example, it might have to make several scripts and call them multiple times, and that assumes that it understands what it is doing enough to properly make the script.
- photon_lines 2y ago
  I'm not sure what you mean by 'it will not scale well.' When we humans learn that we make a mistake - we make a note and we hold the correct answer in memory - the next time we're prompted with a similar prompt, we can use our old memories to come up with the correct solution. I just did a simple test for this same exact problem using ChatGPT 3.5: 'Can you reformulate the following problem using Prolog? When you execute the program - what answer do you get? 'Alice has 8 brothers and she also has 5 sisters. How many sisters does Alice’s brother have?' Even though it came up with a proper Prolog formulation - it failed to account that Alice is one of the sisters - so I gave it a simple prompt for this and it was able to correct the formulation and come up with the proper solution: My prompt after ChatGPT gave me an answer of 5: 'Yes but you aren't including Alice in your logic. Shouldn't the correct answer be 6 (since Alice is also a sister)?' Response: 'You are correct. The correct answer should account for Alice being one of the sisters. Let's adjust the Prolog program to include Alice in the calculation: % Facts brothers(alice, 8). sisters(alice, 5). % Rules sisters(Brother, NumSisters) :- sisters(alice, Num), NumSisters is Num + 1. % Include Alice herself % Query ?- sisters(alices_brother, NumSisters). In this updated Prolog program: The rule for determining the number of sisters includes Alice herself by adding 1 to the number of sisters Alice has. When we execute the query ?- sisters(alices_brother, NumSisters). in Prolog, we should get: NumSisters = 6. This correctly reflects that each of Alice's brothers has 6 sisters, including Alice herself. When I prompt it to answer this question using Prolog now, I always get the correct answers no matter which integers I feed in, so it seems like the model is able to generalize after I show it where it's 'reasoning' was wrong. Basically, my suggestion is to get the model to recognize logic puzzles like this and try to formulate them in terms of logic programming queries which it can use and tune in order to come up with correct answers rather than simple auto-associative chain of reason training which current GPT models rely on, but like I said - this is my hypothesis and I believe this would work much better in getting these models to 'generalize' than the current approaches we're using. Hopefully this helps.

wtbdqrs 2y ago

I appear to be reasoning at times but I have mostly no idea what I am talking about. I hit a bunch of words and concepts in the given context and thus kind of hallucinate sense. Given a few months of peace of mind and enough money for good enough food, I could actually learn to reason without sounding like a confused babelarian. Reasoning is mostly a human convention supported by human context that would have been a different one if the Fascists had won the war or the Soviet Union wouldn't have gotten corrupted. But none of that has anything to do with pulling up a whiteboard to draw some flowcharts and run some numbers, all of which is why I am certain there is nothing the devs have "to fix". It took most reasonable humans many generations to learn stuff. Very few of us did the actual work. It's all just a matter of time.

voxic11 2y ago
Yeah, I think these chatbots are just too sure of themselves. They only really do "system 1 thinking" and only do "system 2 thinking" if you prompt them to. If I ask gpt-4o the riddle in this paper and tell it to assume its reasoning contains possible logical inconsistencies and to come up with reasons why that might be then it does correctly identify the problems with its initial answer and arrives at the correct one. Here is my prompt: I have a riddle for you. Please reason about possible assumptions you can make, and paths to find the answer to the question first. Remember this is a riddle so explore lateral thinking possibilities. Then run through some examples using concrete values. And only after doing that attempt to answer the question by reasoning step by step. The riddle is "Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?" After you answer the riddle please review your answer assuming that you have made a logical inconsistency in each step and explain what that inconsistency is. Even if you think there is none do your best to confabulate a reason why it could be logically inconsistent. Finally after you have done this re-examine your answer in light of these possible inconsistencies and give what you could consider a second best answer.
- wtbdqrs 2y ago
  I'm not gonna read that book. I started and stopped after few chapters because it is based on and aims at manufacturing minds that follow game theory logic. Science (studies, reviews and application) got damaged quite a bit when too many people started following game theory logic. We are, from our aware POV, a very young civilization. And you only ever need game theory logic when you have to survive, got no thing and no skill to trade and you are too pathetic to move back in with your parents to work on your mind and or fuckability. Making money by ways of game theory logic compensates for all that but also diminishes the survival chance of the users' offspring to zero once super-unalligned AGIs start to assess the entire supply chain of wealth and how it impacts the evolution of human organisms and the ones inside them.

layer8 2y ago

For reasoning of any complexity, some sort of internal monologue and iteration is needed. For type-2 thinking, we conceptualize possible solutions, arguments, and reasoning paths in our mind, and judge and evaluate them before deciding which one we think is adequate or correct. This can be emulated to some extent by guiding prompts that make that internal thought process external, but we really want it to be internal and automatic. We don’t know how to do that yet, because what controls the internal thought process is itself not necessarily language-based, and also, since internal thought processes of biological brains are not directly observable, they can’t be used as training data. Edit: It occurs to me now that there is some parallel between current LLMs and behaviorism [0], and we really need something to which cognitive psychology could be applied instead. [0] https://en.wikipedia.org/wiki/Behaviorism https://en.wikipedia.org/wiki/Behaviorism

sollewitt 2y ago
Given many people don’t have an inner monologue and function just fine, it’s more likely inner monologue is a product of the reasoning process and not it’s mechanism.
- CooCooCaCha 2y ago
  I think you’re using “inner monologue” too literally. It could be a progression of pictures, emotions, etc.
  - ajuc 2y ago
    With that definition even bacteria have inner monologue.
    - CooCooCaCha 2y ago
      Can bacteria imagine pictures? Do they have emotions? Why does this matter? Stop being so pedantic. We're talking about a progression of ideas. Talking in your head is one form of ideas, but people can easily solve problems by imagining them.

ericmcer 2y ago

If you really think about what an LLM is you would think there is no way that leads to general purpose AI. At the same time though they are already doing way more than we thought they could. Maybe people were surprised by what OpenAI achieved so now they are all just praying that with enough compute and the right model AGI will emerge.

more_corn 2y ago
LLMs are the language center of an AI. They in no way purport to be: long term memory, critical thinking engines, calculators. If we want those things we can build them. Building them into the language center would be absurd and weird.
- daveguy 2y ago
  This is a very good take on the current state of LLMs. They are able to articulate surprisingly well and LLMs or something similar will definitely be part of a larger AI model in the future. They are good creative aids, but they should not be trusted.
- viking123 2y ago
  In the brain all those things are neural networks too, memory is encoded in the weights. So if we want to have that type of system, everyone should have their own model and the weights would actually adjust all the time just like in our brain, but now it is just trained once and that is that
- lupire 2y ago
  And yet this is what all the big companies we re trying to do, because the LLM is incredibly expensive fixed cost, and they marginal cost for each new task.
solidasparagus 2y ago
> If you really think about what an LLM is you would think there is no way that leads to general purpose AI It is an autoregressive sequence predictor/generator. Explain to me how humans are fundamentally different

glitchc 2y ago

The problem is a good chunk of the global population is also not reasoning and thinking in any sense of the word. Logical reasoning is a higher order skill that often requires formal training. It's not a natural ability for human beings.

slashdave 2y ago
Seriously? You think individuals are incapable of reasoning without training first?
- TeMPOraL 2y ago
  Do you think they are? We haven't had a case of individuals without training in like 10 000 years.
  - rep_lodsb 2y ago
    And judging by some comments here, humans must have gotten dumber since then. Or we would never have invented writing, agriculture or even basic clothing. "This problem is actually not that easy, the average person couldn't solve it either, especially if the numbers were bigger", "Yet another cherrypicked clickbait study to make LLMs look bad, those people are just scared of being made obsolete", etc.
    - TeMPOraL 2y ago
      > Or we would never have invented writing, agriculture or even basic clothing. That's the thing. We had to invent those things. Along with counting, numbers, logic, arithmetic, and those stupid-ass annoying family tree riddles. We didn't get them in one step, it took a long time to build one on top of the previous. Generation by generation, each cohort of children growing up in a slightly more complex world than their parents, each generation taught how to navigate this complexity by their families and tribes. Learning a growing collection of facts and beliefs and life hacks. There were no untrained people for as long as humanity existed. The minimum reproductive unit of homo sapiens is a village.

Mmskynettio 2y ago

Its not an AI hype. A hype is defined as something which gets oversold: "promote or publicize (a product or idea) intensively, often exaggerating its benefits." Just yesterday I visited a google cloud summit and one person from bosch told the audiance how they are now able to work with less external agencies like texting, graphicsdesigner and photographers for their materials. It already saves money, has real impacts and continues to progress. We are also don't know what ChatGPT 5 will bring, because they say this will do more reasoning than before, but we already are working (people/our socity) on solving this in different ways: From code which creates a unit test first and than the code, to different type of architectures. For me, 2024 was the LLM cost reduction year and the LLM gets a big context window year. AI doesn't need to be ready tomorrow, but its capabilities are already really good. And i know plenty of people around me who are a lot less interesting to talk to than any llm (from a human skill/knowledge point of view). llama 3 was also a big achievement 2024. Facebook shows that better data leads to better quality for smaller models. We haven't not only entered the AI ara but also the 'gather all the knowledge we can, quality check it and refine it because now we can actually do something with it' ara. We are in the feedbackloop knowledge ara.

Lich 2y ago
> AI doesn't need to be ready tomorrow, but its capabilities are already really good. A majority don’t deny that it’s good. The problem is that so many think it is actually reasoning, believing the answers can be trusted.
- rvnx 2y ago
  It's generally reasoning better than me, so using them I do less mistakes than if I did my own logic.
- Mmskynettio 2y ago
  Does it need to do reasoning perfectly? If it created meta concepts from billion words on the internet and has meta models which are correct and are more and better than an avg human, isn't it actually good in reasoning? Its a very narrow thing to say 'is that so many think its actually reasoning' to say AI is just hype or everything we are doing is a waste etc. There are human benchmarks they are winning at. The critic could be more that we don't have enough benchmarks.

nuancebydefault 2y ago

> They are not reasoning or thinking in any sense of the word There is no definition of reasoning or thinking. No single human knows what it is. The only thing we know is: we as humans are capable of recognizing steps and results of reasoning and thinking. In a lot of cases, when using LLM's, those results appear to be correct and usable. This is often easy to determine with generated code. I want to argue that, lacking a definition of reasoning, I am happy to have found that the machine helps me to get results that might as well have been produced by a lot of human knowledge, wisdom and deep reasoning.

ninetyninenine 2y ago

> They are not reasoning or thinking in any sense of the word and yet a lot of people are already considering them general purpose AI. You yourself did not use reasoning to arrive at this conclusion. It's quite obvious. I'm not trying to belittle you here. But LLMs are black boxes, we do not actually know what they are doing at a high enough resolution where we can call it "not reasoning" or "reasoning". We can only characterize these AI's as a best fit curve between datapoints which is a way to high level view point to come to any conclusion about "reasoning" This paper presents evidence of failed reasoning, but how does that prove anything when LLMs exhibit many instances of successful reasoning on complex topics they were not trained on? You are biased and honing into information that supports a biased conclusion. LLMs are an AI we do not understand at a low level. Hence we talk about the attributes of these AIs in the same way we talk about humans, "Oh the LLM hallucinates", "it tries to justify it's answer..." etc. etc. You characterize the Danger of these AI's as the result of Human stupidity. The danger according to you is solely from a human mistakenly believing that the AI is anything other than a stochastic parrot. This is a belief arrived at in the same spirit as your claim. You did not use reasoning to arrive here. The only logical way to characterize what is going on is that we do not know. It could very well be that these AI's are in fact reasoning. And that in itself presents a different kind of danger. A danger that may be more clear in the far future. The irony is that your conclusion lacking correct reasoning is similarly parallel to the LLM's lack of reasoning. LLMs are more alike to us than you would like to believe. Can you give me the step by step signal path ways of an LLM processing a query to prove that it does not reason? Or do you have to use black box anecdotal evidence to prove your point? For any "evidence" where an LLM failed to reason, there is another counter example showing where the LLM succeeded. Contradictory evidence can only lead to a vague conclusion.

fnordpiglet 2y ago

Question is how often do humans get the question wrong unprompted ? I expect given the median IQ of 100 and assessing people of all types including those untrained on logic and generally uneducated humans might not be nearly as close to 1.0 as the authors seem to imply. But it’s also fairly obvious LLMs don’t reason at all so it’s not shocking that LLMs don’t reason at all. What’s remarkable is that they’re able to perform as well at reasoning tasks as they do given that factual truth they have no reasoning facility. Coupled with their natural language ability and emergent abductive reasoning ability that’s generally hard in computing models it’s hard to draw much conclusion here other than “wow gpt-4o is amazing.”