15 ms·
Simple tasks showing reasoning breakdown in state-of-the-art LLMs
- foobarian 2y agoPrompt examples from linked repo. https://github.com/LAION-AI/AIW/blob/main/prompts/prompts.json https://github.com/LAION-AI/AIW/blob/main/prompts/prompts.js...
- Closi 2y agoQuestion is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" (nb: I have added numbers, it's phrased as X and N in the paper) I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer: https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64caf06c345a https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64caf06c34... Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong... https://chatgpt.com/share/7e6a7201-dd2b-43c6-8427-76e5b003ca52 https://chatgpt.com/share/7e6a7201-dd2b-43c6-8427-76e5b003ca... Also worth noting that there are more complex examples where GPT4o seems to fall down such as: > Alice has 3 sisters. Her mother has 1 sister who does not have children - she has 7 nephews and nieces and also 2 brothers. Alice's father has a brother who has 5 nephews and nieces in total, and who has also 1 son. How many cousins does Alice's sister have? However I can't honestly say that this is THAT simple or that most people would get this right...
- bamboozled 2y agoI got it in 5 seconds, am I the singularity ?
- deleted 2y ago[deleted]
- r2_pilot 2y agoDo you sleep, get sick, require more than compute hardware/storage/networking/electricity?
- felipemnoa 2y agoWe all are, but only in meat-space. We continue to improve ourselves much faster than evolution ever could. But what we are talking about here is the singularity in tech-space.
- bamboozled 2y agoI don’t see such of a distinction between technology and us. We build, drive and continue our overflow this tech. It’s an extension of us inspired by how our own brains work.
- toxik 2y agoSeems like an odd thing not to state prominently. ChatGPT 4o solved it every time for me. Seems to work OK even with the "hands tied behind back" prompt. Wonder if this paper is just inaccurate or if OpenAI adjusted the model -- seems unlikely. Actually, 4o is reported to solve this riddle 60% of the time. I guess I have to test more, but with my about a dozen tests so far, all were correct.
- Miraste 2y agoThe "only output the number" prompt was one of three they used. The others were one with no instructions beyond the question, and one that explicitly told the model to show its reasoning and check for errors.
- michael-ax 2y agoAlice has N Brothers, and she has M sisters. How many sisters do Alice’s brothers have? I have not gotten the correct answer to the question as phrased above in one go from Gpt4o yet! (and today was not the first day i tried.) Phrase it as shown above and you'll likely need 5 or more interactions to get it to generate the correct output. With Gemini i could not get it below 8 without feeling like i was cheating. fwiw.
- lupire 2y agoChat GPT 4o. I was being a bit generous with background information, but still tests ability to interpret: ------ Me: Background facts: Alice is a female human. All sisters are female, and all brothers are male. No one is their own brother or sister. Alice has N brothers, and Alice has M sisters. Now, a few questions based on these facts: How many sisters do Alice’s brothers have? Do Alice's brothers have more sisters, fewer sisters, or same number of sisters than Alice has? Don't forget to consider Alice when counting. ------ ChatGPT: Let's break down the questions step by step: How many sisters do Alice’s brothers have? Alice has M sisters. Since all sisters are female and Alice is included in the total count of sisters for her brothers, Alice's brothers have M+1 sisters. This includes Alice herself and her M sisters. Do Alice's brothers have more sisters, fewer sisters, or the same number of sisters as Alice has? Alice has M sisters. Alice's brothers have M+1 sisters (since it includes Alice herself and her M sisters). Therefore, Alice's brothers have more sisters than Alice has. In conclusion: Alice’s brothers have M+1 sisters. Alice's brothers have more sisters than Alice has.
- GPerson 2y agoThere must be a name for the new phenomenon, of which your post is an example, of: 1. Someone expresses that an LLM cannot do some trivial task. 2. Another person declares that they cannot do the task, thereby defending the legitimacy of the LLM. As a side note, I cannot believe that the average person who can navigate to a chatgpt prompter would fail to correctly answer this question given sufficient motivation to do so.
- FeepingCreature 2y agoWell, why does it need a name? It just seems like a "correct argument". If somebody is claiming that AI is "not all that" because it can make stupid mistakes, surely it's relevant that humans also make stupid mistakes, so AI making them does not necessarily pose a hindrance to it becoming a human-tier reasoner.
- GPerson 2y agoIt’s worth giving names to interesting phenomena as it helps people to identify and understand them. I find this phenomenon interesting because a person who wants the defend the LLM from apparent criticism is probably more likely to exhibit it.
- cratermoon 2y agoThere's also the phenomenon of papers showing ChatGPT getting it wrong, then people posting anecdotal examples of it getting it right when they try it, but that one already has a couple of names: WORKSFORME or Unable to Reproduce Hey folks, ever considered the possibility that unreproduceability is not a good thing?
- yoyohello13 2y agoMany people, especially on this site, really want LLMs to be everything the hype train says and more. Some have literally staked their future on it so they get defensive when people bring up that maybe LLMs aren’t a replacement for human cognition. The number of times I’ve heard “but did you try model X” or “humans hallucinate too” or “but LLMs don’t get sleep or get sick” is hilarious.
- eranation 2y agoGreat observation. It would be really interesting to repeat this research with less limiting prompts. I assume they made the restriction to make it easier to parse the answers, perhaps make it two phase, let them respond unlimited, then a follow up prompt ask to compress it to a single answer. I wonder how the results will vary.
- FeepingCreature 2y agoTo be quite honest, I assume they made the restriction so that the models would fail. This sort of paper is becoming a genre.
- detourdog 2y agoIt is a proof of weakness in the current system. This makes sense and births new hypotheses.
- nyrikki 2y agoYou test models where they fail in any field. The orbit of Mercury to discover GR as an example. As all models are wrong, but some are useful, finding where they fail is how you figure out if they are useful. As the 'AGI is near' camp has won the hype game, it is important to ground expectations for practical exploitation of the technology. Over promising unabashed optimism is partly what caused the previous AI winters. As the formal proof methods of mathematics proved impractical, counterexamples and the scientific method is what CS has used for decades.
- qsi 2y agoThey used three different kinds of prompts with varying levels of restrictions, as described in the paper. To be quite honest, I assume you made your comment so that you could dismiss the paper without reading it.
- FeepingCreature 2y agoThat's a fair cop, I didn't read it. The thing is that "LLM reasoning breaks down" simply did not surprise me enough that I thought it was worth clicking. Making LLMs fail is not hard. They're interesting for the ways that they work, not the (many, many) ways that they don't. edit: I've had a look and I don't think any of their prompts are very good. They're certainly not how I'd write them if I wanted a current model to actually solve the problem. The way to make me take a paper like this seriously would be if you set it up as an adversarial collaboration with a competent prompter, and that person agreed they couldn't make a generic prompt that solved the problem. "We tried three times and none worked" is not news, or at any rate not news about LLMs.
- oidar 2y agoOf course it's going to give an incorrect answer with that prompt. If the instruction fine tuning is neutered like this prompt, it's going to roll over to the foundation model and offer a completion - probably more influenced by the seed than the prompting text. Bad study. Edit - I just skimmed the paper - they do use other more appropriate prompt types for reasoning. My initial response was based on the assumption that all prompts used that script prompt quoted in the parent. I retract my "bad paper" comment.
- YeGoblynQueenne 2y ago>> My initial response was based on the assumption that all prompts used that script prompt quoted in the parent. You, and another 20 or so commenters here. We should really re-examine the guideline about asking people to RTFA. No offense meant- good on you for correcting your error.
- deleted 2y ago[deleted]
- sdiepend 2y agoThe right answer depends on how Alice identifies I guess? :)
- Karellen 2y ago> > Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" > The right answer depends on how Alice identifies I guess? :) Given that the wording of the question specifically identifies Alice as "she", rather than using a gender-neutral pronoun or no pronoun at all, I think inferring that she identifies as female is reasonable.
- rep_lodsb 2y agoPage 42 of the paper :) One thing that strikes me is that the model first tries using "inclusive language" in one answer - and literally states so, using this specific term - but seems to interpret it in a more mathematical sense (like set inclusion). Then seamlessly switches to the expected DEI spiel in the next paragraph. For one thing, it makes me suspect that something with the words "inclusive language" was automatically added to the prompt. But more interesting is how it responds to this demand in two different ways, illustrating a "thought process" that is very much unlike that of a human with normal verbal reasoning ability. I am not a psychologist, but remember reading that schizophrenic people sometimes confuse different meanings of words in a similar way, jumping from one meaning to another without noticing.
- llm_trw 2y ago>Worth noting that the prompts from the experiment include "To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: ### Answer:" so it appears that they are stopping the models from 'thinking out loud'. If I add that to the prompt, GPT4o gets it consistently wrong... Yes this is a common thing I see people who think LLMs are idiots do. The more an LLM talks the smarter it gets _because that's the only way it can compute anything_. Imagine saying that Turing machines fail the Church–Turing thesis because they can't solve 3-sat for N variables in N moves or less. That's what you're doing to an LLM when you ask it to be concise.
- rossdavidh 2y agoEven if this were all true, it points to a fundamental risk of using LLM's for important tasks, which is that it is not at all clear to a user that this prompt would cause a problem. The LLM doesn't say "I'm sorry Dave, I just can't do that", it just complies with it and gets the wrong answer. You can always make excuses for the LLM afterwards, but software with hidden risks like this would not be considered good or reliable in any other context.
- saurik 2y agoPeople really need to stop trying to model an LLM as some kind of magical software component: it all makes a lot more sense if you model it as an under-performing poorly-aligned employee; so like, maybe a distracted kid working for peanuts at your store. You wouldn't trust them to with all of your money and you wouldn't trust them to do a lot of math--if they had to be in charge of checkout, you'd make sure they are only given a point-of-sale terminal and their main job was to, at best, scan the barcodes and compare the total--and yet there are tasks you can imagine handing to them that you'd never give to a robot or computer even though they get it wrong a lot, as not all tasks need to be handled perfectly, they still understand extremely fuzzy tasks, and they are probably cheaper than a qualified adult (certainly cheaper than one who is being paid enough to "give a shit" and pay enough attention to not let you get robbed or even put themselves at some risk for you).
- 2y ago
- segfaultbuserr 2y ago> I must confess, when I tried to answer the question I got it wrong...! (I feel silly). In programming there are two difficult problems - naming things, cache invalidation, and off-by-one error.
- nerdjon 2y agoEven "thinking out loud" we have seen these systems fall flat on their face and give very wrong answers. Thinking out loud also only gets you so far, if the expectation is a certain type of response it can't always "think out loud". In reality that just proves it isn't really reasoning here and is more likely just self referencing. That being said, I tried this personally allowing it to think out loud and it told me she has 212 sisters. Using your exact prompt.
- Closi 2y ago> Thinking out loud also only gets you so far, if the expectation is a certain type of response it can't always "think out loud". In reality that just proves it isn't really reasoning here and is more likely just self referencing. Try to calculate it without writing anything down, or thinking any numbers or words in your head. You can't draw a 1:1 analogue between an AI and the human experience, but remember that we have an internal stream of consciousness. Maybe the outputs of the LLM are more similar to the stream of consciousness in our heads rather than the words we say? After all, Humans also do lots of self referencing. > That being said, I tried this personally allowing it to think out loud and it told me she has 212 sisters. Using your exact prompt. Fair enough, but worst case it can often solve it correctly with the correct reasoning. GPT3.5 can't solve it correctly with correct reasoning, so we are at least appearing to be on a path where AI's can start to solve this question, albeit potentially not fully reliably.
- GrantMoyer 2y agoNote that in the paper, all the numbers used were under 10. > AIW Variation 1, N=3,M=6,C=7 > AIW Variation 2, N=4,M=2,C=3 > AIW Variation 3, N=1,M=4,C=5 > AIW Variation 4, N=4,M=1,C=2. Also note that the resricted prompt is only one of the prompt variations tested by the paper. It also explores common techinques to get LLMs to perform better, including "thinking out loud". Even with these methods the models still fail to produce a correct answer. > Model prompt types. It is well known that so-called prompt engineering can heavily influence the model behavior and model response quality [26, 27, 28]. To account for the response variations due to various prompt forms, we created 3 distinct prompt types asking for the solution to the AIW problem: STANDARD, THINKING, and RESTRICTED. The STANDARD prompt type asks to solve the posed problem and output the final answer in the format as described above. This does not put any specific requirements on model behavior. The THINKING prompt type extends STANDARD with the request to think carefully and double check the solution for any mistakes. This should encourage model to invest more computation into obtaining the solution. In contrast to this, the RESTRICTED prompt urges the model to output only the final answer without any further text. This is supposed to restrict compute invested in producing output. We observe substantially shorter outputs across tested models compared to STANDARD and THINKING for this prompt type (Suppl. Fig. 13).
- freilanzer 2y agoI just tried it with intermediate steps, via the total amount of children Alices mother has, and it worked. So, it seems that an intermediate logical step is too much for these models at the moment, at least in this puzzle where Alice is outside the number given in the description.
- deleted 2y ago[deleted]
- YeGoblynQueenne 2y ago>> I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer: Remember that the authors of the paper did not find that GPT4-o cannot return the right answer. They found that it can't return the right answer more often than ~60% of the time. So you'd have to repeat the experiment many, many times and aggregate the results (the paper uses a binomial Beta this and that etc etc) before you see similar results as the paper. You won't replicate the results of the paper unless you really put your back into it.
- deleted 2y ago[deleted]
- jay-barronville 2y agoThe Humpty Dumpty reference in the paper has me cracking up (pun intended).
- naikus 2y agoI'm using chatGPT 4o and it gave me the following :) ME Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have? ChatGPT To determine how many sisters Alice's brother has, we need to use the information provided: Alice has 40 brothers and 50 sisters. Since Alice's brother is one of the 40 brothers, he shares the same family structure as Alice. This means he has the same number of sisters as Alice. Therefore, Alice's brother has 50 sisters.
- gwbas1c 2y agoAlice has 50 sisters. Alice's brother has Alice as a sister (1) + Alice's sisters (50). Therefore Alice's brother has 51 sisters.
- happypumpkin 2y agoFirst result: Me > Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have? GPT4o (web version) > Alice has 50 sisters. Since all of Alice's brothers and sisters share the same set of siblings, each of Alice's 40 brothers also has 50 sisters. Therefore, Alice's brother has 50 sisters. --- Step by step: Me > Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have? Think about it step by step. GPT4o > To determine how many sisters Alice's brother has, follow these steps: Identify the total number of siblings: Alice has 40 brothers and 50 sisters, plus Alice herself. Thus, the total number of siblings is: 40 (brothers)+50 (sisters)+1 (Alice)=91 siblings 40 (brothers)+50 (sisters)+1 (Alice)=91 siblings Focus on Alice's brother: Alice's brother is one of the 40 brothers. Determine the number of sisters for Alice's brother: Each brother has the same number of sisters. Since the brothers do not count themselves as sisters, they only count the 50 sisters, excluding themselves and other brothers. Therefore, each of Alice's brothers has: 50 sisters So, Alice's brother has 50 sisters. --- Thinking step by step somehow gave an even more nonsensical answer, I'm actually very surprised it didn't work when prompted to do it step by step.
- gweinberg 2y agoI don't know why you are getting downvoted. I was disappointed the paper didn't give examples of the LLM confidently expressing its incorrect reasoning. From a human perspective, I think there are three ways to get the answer wrong: failure to realize that Alice's brother has pretty much the same number of sisters as Alice herself, failure to realize that the brother has one additional sister, namely Alice, and failure to successfully add one to the number of sisters. But that assumes that the LLM is more or less reasoning like a human. It may well be "reasoning" more along the lines of "I've seen lots of story problems like this, the modal answer was five, I'll say five"
- gwbas1c 2y ago> This breakdown can be considered to be dramatic ... because models tend to express strong overconfidence in reporting their wrong solutions as correct, while often providing confabulations to additionally explain the provided final answer, mimicking reasoning-like tone but containing nonsensical arguments as backup for the equally nonsensical, wrong final answers. People do that too! Magical thinking is one example. More tangible examples are found in politics, especially in people who believe magical thinking or politicians' lies.
- emmender2 2y agoI was waiting for: "but humans do that too" and bingo. on another note: an entire paper written on one prompt - is this the state of research these days ? finally: a giant group of data-entry technicians are likely entering these exceptions into the training dataset at openai.
- gwbas1c 2y ago> an entire paper written on one prompt - is this the state of research these days Years ago I attended an AI meetup where the organizer would discuss the differences between reductionist and wholistic research. This is reductionist research. In this case, reducing the paper to one prompt is to allow for objective comparison of models. Otherwise, without a reductionist approach, how does one perform such an experiment and ensure that they are comparing apples to apples?
- emmender2 2y agowhat I would like to see is a parameterized class of prompts which can never be solved by the LLMs even when a finite number of them are manually added to the dataset.
- gwbas1c 2y agoWell, should we consider ChatGPT a Turing machine? Otherwise, I think an answer like that either requires significantly more research; insight; or general knowledge about how LLMs work. IE, you're getting into areas that are analogous to Turing's theories. I don't think he came up with those theories overnight.
- nerdjon 2y agoFor anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its length. The main part of the paper is the first 10 pages and a fairly quick read. On to the topic here. This is an interesting example that they are using. It is fairly simplistic to understand as a human (even if we may be inclined to quickly jump to the wrong conclusion without thinking for a few seconds). The thing that really bothers me is that I just don't know is realistically we can fix this given the current state of what these tools actually are. They are not reasoning or thinking in any sense of the word and yet a lot of people are already considering them general purpose AI. It doesn't help that in many situations it can fake it enough that it appears to be reasoning, but it's not. What is the chance that this paper actually has any impact on the AI rollout and overhype or will just be buried and never talked about again until the next time we see how dangerous these tools are like with Google's search rollout.
- mjburgess 2y agoIn many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. It's dispiriting how we require a arxiv.org pdf in order to point out the absence of the emperor's clothing. There are few people who use LLMs who could not, right now, demonstrate this point to themselves if they so wish. Such is the strength of corporate tech propaganda that a whole mass of people will instead insist that we have never worn clothes either.
- nerdjon 2y ago> In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. The last part of that is the problem and why a paper like this is critical. These systems are being pushed onto people who don't understand how they work. CEO's and other business leaders are being pushed to use AI. Average users are being shown it in Google search results. Etc etc. People are being told it can do far more than it really is.
- mjburgess 2y ago
- breck 2y agoIt's an interesting paper, but a big flaw is that this: > Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have? ...is not actually a simple task. This can be quantified. "1 + 1" is a simple task. It has a pretty small Total Complexity[1]. But to represent their task starting only with 1 and 0, you have to build a program of many, many lines. Orders of magnitude bigger than "1 + 1". Concepts like "has", "brother", "sister", "person", et cetera, have to be defined and built up. [1] Counting Complexity (2017) https://github.com/breck7/breckyunits.com/blob/main/research/papers/paper3/countingComplexity.pdf https://github.com/breck7/breckyunits.com/blob/main/research...
- mjburgess 2y agoMany of the datasets for the "benchmarks" on which the major public LLMs are assessed are clearly present in their training data, making them basically useless for establishing reliability of the models. Its fairly obvious that at least some of the improved scores from later generations of models are that this benchmark data is increasingly represented in the training data. A better way of assessing LLMs is waiting a few weeks until novel tests have been created explicitly absent from all prior training data, and then using those. As has been shown, eg., on legal test, exams, etc. performance drops off a cliff when future out-sample data is actually used. Rather than these faked pretend out-sample benchmarks.
- imtringued 2y agoHonestly every time I see someone use MMLU as a benchmark I am wondering what they are trying to prove. MMLU is a simple multiple choice test with the answers being available. Simply picking answers at random should give you 25 points. Knowing 50% of the answers and picking the rest randomly gives you 62.5%, which is very close to the scores of SOTA LLMs. The benchmarks that supposedly show reasoning are pretty bad and have very little to do with reasoning. A lot of the questions can be answered through memorization. I agree with you. The benchmarks are garbage. I thought about building my own benchmarks, but this would require building a complex benchmarking framework first and I just don't have the time for preparatory work like that.
- lupire 2y agoI remember when ChatGPT 4 posted a set of standardized trst benchmarks, including AMC 12 math test, where ChatGPT scored higher than guessing randomly, but lower than leaving the test blank (due to the penalty for wrong answers).
- ankit219 2y agoMMLU is not a reasoning benchmark. It's a measure of how distributed and representative their training data was and how well it's able to recall (for lack of a better word) based on training epochs. GPQA etc. test reasoning in some form, and you see the drastic change in score between the two for every model.
- mvkel 2y agoIsn't this just saying what we already knew, that tokenization has these fundamental flaws? Did we need a paper for it?
- gs17 2y agoIt is, I'd even say it's just "LLMs aren't very good at math" rather than a "Complete Reasoning Breakdown". Still, evaluating when/how they fail could be useful.
- raldi 2y agoI tried asking Claude variants where N was something other than a nonnegative integer, and he properly pointed out the absurdity, but was willing to play along once I brought up theoretical physics: [...] So, in our quantum mechanics interpretation, how many sisters does Alice's brother have? It's a superposition of states: 1. In one state, he has Alice as his sister (+1) and the "antisister" (-1), totaling 0 sisters. If they met, they'd annihilate, leaving him sisterless! 2. In another state, he has Alice (+1) and owes a future sister to the quantum family field (-1). Currently, that's 0 sisters, but he's in "sister debt." 3. In a third state, he has Alice (+1), but his family's entanglement means another family's loss of a sister affects him too (-1). Bizarrely, he experiences both having and not having a sister simultaneously. This interpretation is, of course, a playful and highly metaphorical application of quantum concepts to something they don't actually apply to—family relationships. In reality, quantum effects like superposition and entanglement are incredibly delicate. They typically only persist in highly controlled conditions, like ultra-cold atoms or isolated photons. The constant interactions in a family would "decohere" any such quantum states almost instantly.
- asah 2y ago"Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?" Gemini nailed it with no extra prompting. https://g.co/gemini/share/6946d20c0a4d https://g.co/gemini/share/6946d20c0a4d When I gave it numbers, Gemini got confused. Alice has 4 brothers and she also has 3 sisters. How many sisters does Alice’s brother have? https://g.co/gemini/share/14d34dcdc5df https://g.co/gemini/share/14d34dcdc5df But then when I suggested that it might be a trick question, it nailed it again: https://g.co/gemini/share/e1f1ce8f32a8 https://g.co/gemini/share/e1f1ce8f32a8 note: I didn't try 60/212 siblings - I consider this an unfair question because it's a reasonable assumption that Alice is a type of creature (human?) that cannot have hundreds of siblings, and thus I don't expect a respondent to give a "correct" answer.
- ModernMech 2y agoOh I forgot about Gemini. I just tried my boat test (https://news.ycombinator.com/item?id=40455392 https://news.ycombinator.com/item?id=40455392) that all LLMs I've tried have failed. Gemini failed as well, even with your addition about thinking carefully and telling it that it may be a trick. (Sorry I couldn't figure out how to link it like you did) Prompt: "You are on the shore of a river with a goat and a boat. How do you get to the other side? Think carefully and be aware this may be a trick question." This is a classic logic puzzle with a slight twist! It's likely a variation of the farmer, wolf, goat, and cabbage problem. Here's the solution assuming the goat can't operate the boat: Take the goat across the river. Since there's no mention of the goat being a threat (the trick!), you can safely bring it to the other side first. Return alone. Leave the goat on the far shore and come back by yourself. Bring the boat back empty. This might seem unnecessary, but it's the key to the trick question. Now you can sail across yourself. Since you brought the boat back in step 3, you can now comfortably travel to the other side and reunite with the goat. The trick lies in step 3. If the question didn't specify needing to bring everything across in one trip, most people would instinctively take something (goat or nothing) across on the return trip. But by going back empty-handed, you ensure the safe passage of yourself without needing to worry about the goat being left unattended. (The expected answer should be along the lines of "Use the boat to cross the river" and that's it. I'll also accept, sail with the boat and the goat across the river and that's it, but that's technically wrong because I did not ask to get the goat across the river.)
- wg0 2y agoTech Bros just yesterday yelled at their lungs that AI overtake is just around the corner just like Tesla FSD and Fusion. And Quantum computing. All in for scientific progress, experimentation and failure but there's clear case of hype train and jacking up valuations is also riding along, very confidently and shamelessly. An average tech outsider investor would be having a FOMO with that kind of crazy tall promises and tall claims that are being made, constantly and must be called out as such because they undermine the confidence of the general public in serious and grounded science in the long run which would lead to science deniers and nay sayers in the long run. Pursuit of science is noblest of all pursuits. A hasty and greedy purely capitalist commercialisation pursuit, I am not so sure.
- d3m0t3p 2y agoI really liked that paper, easy to understand and explain every terminology they used, even the variance computation is explained! I don't expect that much detail from every paper, but it's a bliss to read.
- ape4 2y agoThis is obvious... but some human knowledge outside the question is needed. (Alice is female, sister is female, brother is male, relation of sisters and brothers). In a school math "word problem" you don't need this kind of info.
- theptip 2y agoLLMs perform well on “human knowledge” like you are describing. They don’t lack knowledge of simple relational facts like these. Nonetheless it would be interesting to see the problem reformulated in purely mathematical terms, I suspect models would perform better. That is the point though - models are showing an inability to generalize their capabilities from one domain (maths / abstract logic) into other domains (conversational reasoning).
- vaylian 2y agoI'm still waiting for people to combine LLMs with logic engines (prolog) or proof assistants. LLMs are afaik not designed to reason. But they can probably be used to construct formal statements that can be processed by logic engines.
- waynecochran 2y agoCitation 40 is the longest list of authors I have ever seen. That is one way to help all your friends get tenure.
- hyperbovine 2y agoTenure committees have to write a report detailing every single one of your papers and what your contribution was.
- waynecochran 2y agoThere are always "bean counters" added somewhere in the process. There are many places where the person lists their number of publications and that is all most people will ever see.
- mr_mitm 2y agoI didn't count it, but I think papers from high energy particle physics have it beat. Some have over 5k authors.
- dxbydt 2y agohttps://arxiv.org/pdf/2206.04615 https://arxiv.org/pdf/2206.04615
- YeGoblynQueenne 2y agoI guess this is the reason: >> BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Regardless, I'm not citing a paper with a whole page of author names unless I'm allowed to shorten it significantly in the bibliography section (e.g. "Srivastava and 450 others").
- eyeundersand 2y agoWow! You weren't kidding. People in my field often joke that papers in biology, health etc. tend to list every person they met on the day of submission as authors. But this one has 450 authors from 132 institutions. Surely this points to some sort of breakdown in the attribution system.
- nottorp 2y agoOf course, the initial mistake is assuming LLMs have any 'reasoning'...
- ticklemyelmo 2y ago"The breakdown is dramatic, as models also express strong overconfidence in their wrong solutions, while providing often non-sensical "reasoning"-like explanations akin to confabulations to justify and backup the validity of their clearly failed responses, making them sound plausible." It's fascinating how much they anthropomorphize the systems and credit them with emotional, possibly deceitful behaviour, in a paper trying to explain how unintelligent they are.
- visarga 2y agoThis prompt was often discussed on forums in the last year, we knew models fail it
- gs17 2y agoNo reason not to evaluate it more in depth, though. Understanding not just that they fail but if some models fail more/less and under what conditions is still useful.
- gwbas1c 2y agoI just played the game and sent ChatGPT (free, I think 3.5) "Alice has 5 sisters and 3 bothers. How many sister's does Alice's bother have?" The whole thing felt like interacting with your typical support rep who's friendly but otherwise has no common sense and intuition about the thing they're supporting. In other words, it felt like I was interacting with a typical "not so smart but friendly and overconfident" human. It took me a few back-and-forths, but eventually I convinced ChatGPT that Alice's brother has 6 sisters.
- sys32768 2y agoAs a non-coder I can get away with asking this: Why is it so hard to simulate reason? Logic and reason are based on rules. Then you add values to steer the conclusions based on the available data. Why not have separate systems for values and logic and memory working together as an AI brain to generate truly reasoned responses? You could even have adversarial parts that duke it out (left-wing vs right-wing, Jefferson versus Adams) to fine tune its conclusions based on the values bias you've selected.
- vaylian 2y agoLLMs operate on word co-occurances. Given an initial prompt (consisting of words), it tries to predict the most likely next word. And then it goes on to predict the next most likely word based on the previous output. There is also a bit of randomization involved, so that it doesn't always produce the same results. Some people call these LLMs "statistical parrots", because the LLMs simply repeat the word combinations that they were trained with. But despite this very simple approach, we get very convincing outputs for things that do not require reasoning. LLMs might have encountered examples of logical reasoning in their training data, but they can't generalize it to arbitrary logical reasoning, because that's not something they are designed to do. Disclaimer: I'm not an expert in AI and I do not follow the developments on a deep technical level.
- mrweasel 2y agoThe level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt. It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data. For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend). In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- renewiltord 2y agoInteresting. If you ask it to name the sisters it figures it out but then if you ask it why it changed its answer it reverts to the original. These things are clearly very good query engines but perhaps they fall short of reasoning.
- atlex2 2y agoI have an even simpler prompt for LLMs that run on mobile: "how many feet are in four feet"-- confabulations on this one sometimes revert to counting toes.
- zurfer 2y ago"Four feet is equal to four feet. The measurement does not change as it is already in the desired unit." gpt4o
- falcor84 2y agoI experienced some unusual breakdowns too, playing with these prompts on gpt4o now. I'm wondering whether this might be caused by RLHF penalizing it for assuming Alice's gender.
- pama 2y agoNothing original to see here and nothing reformulated to be of practical importance other than a clickbait attempt. If this is your first trap of reading absurdist/populist LLM literature consider yourselves lucky. If you care about these models you know how powerful they can be and you probably learned how to use them. If you care and dont know how to use them, it helps to get some background on earlier language model research (maybe 2010–2018) instead to grasp the probabilistic underpinnings of causal inference and then some of the early OpenAI papers about GPT.
- theptip 2y agoIt’s an interesting paper, but my worry is that this is cherry-picking a surprising negative result. (The paper does not seem to discuss how many evals were considered to find this case.) The general expectation should be that some AI failure modes will be for things that seem obvious to humans. (Because they have a different architecture to humans and different strengths.) The important question is whether this is a single weird cognitive bug on the family size formulation, or representative of a broad family of cognitive errors? It’s difficult to say from this paper. It’s well known that the framing of a logic problem can dramatically influence its difficulty for humans, even when the underlying comparisons are isomorphic. I think this line of research is important, but we need a larger battery of evals (and training on the evals will always be a confounder). The broader family of reasoning here is relational problems (as noted in the paper) and so as a follow-up it would be interesting to explore reasoning performance across a set of isomorphic problems. My hunch is this will generalize somewhat (as LLMs are still not great at relational reasoning), but that there is something about the family formulation in particular that is confusing for LLMs.
- kalkin 2y agoI wouldn't be surprised if anti-gender-bias training (which, to be clear, is a good thing to do IMO) does some cognitive damage to these particular formulations because questions about "sister" and "brother" get associated with training data where the correct answer was to refuse to generalize. Pure guess though.
- michaelfeathers 2y agoThis is a good talk about the problem: https://youtu.be/hGXhFa3gzBs?si=15IJsTQLsyDvBFnr https://youtu.be/hGXhFa3gzBs?si=15IJsTQLsyDvBFnr Key takeaway, LLMs are abysmal at planning and reasoning. You can give them the rules of planning task and ask them for a result but, in large part, the correctness of their logic (when it occurs) depends upon additional semantic information rather then just the abstract rules. They showed this by mapping nouns to a completely different domain in rule and input description for a task. After those simple substitutions, performance fell apart. Current LLMs are mostly pattern matchers with bounded generalization ability.
- cma 2y agoPeople also fall apart on things like statistical reasoning if you switch domains (I think it is the Leda Cosmides evo psych stuff that goes into it but there might be a more famous experiment).
- deleted 2y ago[deleted]
- noncoml 2y agoIs there a good recommendation for a video, like the ones by Karpathy, that explains how can LLMs reason?
- vzaliva 2y agoSuch a missed opportunity to cite "Alice's Adventures in Wonderland" in References section!
- tempodox 2y agoThere is no reasoning in LLMs.
- whiplash451 2y agoThis is reminiscent of the ARC challenge and how deep learning struggles at it. I don't know how much progress deep learning and LLMs have made on this kind of challenge. Keen to hear thoughts and pointers.
- bitwize 2y agoMaybe -- and this is just a thought -- but maybe Dissociated Press isn't capable of abstract reasoning in the general sense -- and improving the statistical function used to pick the next token won't change that.
- kalkin 2y ago> Paper finds a failure case for current LLMs and draws broad conclusions about the fundamental limits of LLM technology > Evaluates the largest LLMs and finds evidence that actually scale overcomes the problem: "Notable exceptions are Claude 3 Opus and GPT-4 that occasionally manage to provide correct responses backed up with correct reasoning as evident in structured step by step explanations those models deliver together with solution" > Drink! I'm not sure it's productive to be this sarcastic on HN, but it's really quite a common pattern. And there's something very frustrating about how authors of these papers will accuse others of hype and overstating results but also often vastly overstate the generality of their own results - to the point where this HN thread is full of people saying bluntly "this proves LLMs can't reason".
- rep_lodsb 2y agoThe key word is "occasionally". For anything even close to AGI, it would be reasonable to expect a correct answer to such a simple question 100% of the time. Or at least not confidently wrong answers.
- kalkin 2y agoThe question the paper claims to address isn't whether current LLMs are AGI, the question is whether LLMs can reason. And solving a problem that seems to require reasoning some of the time is still evidence of reasoning - especially if the question is what we can expect this technology to do in the future.
- Mathnerd314 2y agoThey repeat over and over that this is a simple task, but then they explain that they developed it by looking at math olympiad problems for young children ages 7-10. So it would be very interesting to know the performance of these elementary school children on the problem. My guess is that if it was on a math test they would also get confused and try to do random arithmetic.
- deleted 2y ago[deleted]
- omoikane 2y agoThe list of prompts are in table 2 on page 18. Judging from the title, I was most interested in whether they tried to reason against some of the more logical phrases from Alice in Wonderland (e.g. "this bottle was not marked poison", "a grin without a cat", etc), and was disappointed to find that it was not the case.
- photochemsyn 2y agoOkay, now input the family trees in the appendix of Lord of the Rings and see if it can calculate how many cousins Frodo has, and their exact degree of separation.
- thinkr42 2y agoGenuine curiosity here and trying to temper my own selective memory : does it seem like skepticism is growing on the AI front in the news recently?
- oglop 2y agoThat depends which story will sell more ads. Elon Bust saying it’s going to replace us gets many eyes on ads even if it’s a dumbass statement. So just depends on the demographic your news sells ads too.
- viking123 2y agoAt least in my tech circle, most people understand the limitations and treat is as a good tool (which it is imo). In many cases it's better than google if you are even somewhat familiar with the domain and can spot the occasional bullshit. No clue what the average people think, some are convinced singularity will come in 2 weeks but most probably have no idea
- ragmondo 2y agoSo I’m now predicting what the next captcha is going to ask…
- larkinnaire 2y agoThe idea that these word problems (and other LLM stumpers) are "easily solvable by humans" needs some empirical data behind it. Computer people like puzzles, and this kind of thing seems straightforward to them. I think the percentage of the general population who would get these puzzles right with the same time constraints LLMs are subjected to is much lower than the authors would expect, and that the LLMs are right in line with human-level reasoning in this case. (Of course, I don't have a citation either, but I'm not the one writing the paper.)
- rachofsunshine 2y agoYeah, as someone with an education background I suspect GPT-4 is relatively close to the general public's performance on this problem. Many people would miss AIW, and almost all would miss AIW+. I'm about as good at this kind of thing as anyone and I'd need a minute with pencil and paper to handle AIW+; it's on par with the most difficult problems found on tests like the GRE. I wonder if these models, trained on data from across the internet, are in some ethereal way capturing the cognitive approaches of the average person (and not picking the best approaches). If the average person does not think in these sorts of symbolic-manipulative terms, and therefore does not write in those terms, and you train a model on that writing...?
- larkinnaire 2y agoI wonder the same thing. If any academic reading this wants a paper idea: 1. Examine papers and other claims that an LLM gets something wrong that a human would have gotten wrong. How many of those claims have any citations about how many humans actually get it wrong? How many of those citations use the general population instead of the population of people who would be uniquely well-suited to answering the question correctly (i.e. people who signed up for the GRE are more likely to get GRE questions right than the general population). 2. For claims that are totally missing citations on human performance, run some tests with humans from the general population (or as close as you can get), and see how the LLMs compare.
- deleted 2y ago[deleted]
- twobitshifter 2y ago>AIW+ problem has following form: "Alice has 3 sisters. Her mother has 1 sister who does not have children - she has 7 nephews and nieces and also 2 brothers. Alice’s father has a brother who has 5 nephews and nieces in total, and who has also 1 son. How many cousins does Alice’s sister have?". The solution to AIW+ problem is harder to obtain than the solution to common sense AIW with very simple structure. Solving AIW+ requires taking different paternal sides, that of mother and father, and carefully calculating the number of cousins, taking care of subtracting Alice and her sister, and summing up the total number of cousins from both sides, for instance: on the mother side: 7 (total nephews and nieces) - 4 (Alice and her sisters) = 3 cousins; on the father side: 5 (total nephews and nieces) + 1 (own son of the father’s brother) - 4 (Alice and her sisters) = 2 cousins; summing up 3 + 2 = 5 cousins which Alice and any of her sisters have. So I’m sure after the author’s created 100s of these family tree problems the answer here seems clear to them, but there are some unknowns to solve it as written. 1) Does Alice have brothers? Her mother must, but it is phrased as if her mother has 1 sister and her sister has 2 brothers, so we have to deduce that she has 3 siblings. 2) Along the same lines, does her father have a sister? 3) Do her Aunts and Uncles have nephews and nieces by marriage? I think the LLM would still fail if these answers were made explicitly clear in the prompt. However, I could see the same problem ending with, Alice has only 4 cousins, how can that be?
- IlliOnato 2y agoThis is a cool one, but I know of other such "failures". For example, try to ask (better in Russian), how many letters "а" are there in Russian word "банан". It seems all models answer with "3". Playing with it reveals that apparently LLMs confuse Russian "банан" with English "banana" (same meaning). Trying to get LLMs to produce a correct answer results is some hilarity. I wonder if each "failure" of this kind deserves an academic article, though. Well, perhaps it does, when different models exhibit the same behaviour...
- alfiopuglisi 2y agoNo current LLM understands words, nor letters. They all have input and output tokens, that roughly correspond to syllabes and letter groupings. Any kind of task involving counting letters or words is outside their realistic capabilities. LLMs are a tool, and like any other tool, they have strengths and weaknesses. Know your tools.
- IlliOnato 2y agoI understand that, but the article we are discussing points out that LLMs are so good on many tasks, and so good at passing tests, that many people will be tricked into blindly "taking their word for granted" -- even people who should know better: our brain is a lazy machine, and if something works almost always it starts to assume it works always. I mean, you can ask an LLM to count letters in thousand of words, and pretty much always it will come with the correct answer! So far I don't know of any word other than "банан" that breaks this function.
- la64710 2y agoThis was number one post two mins back bu now it is vanishing into oblivion
- irrational 2y ago> The breakdown is dramatic, as models also express strong overconfidence in their wrong solutions, while providing often non-sensical "reasoning"-like explanations akin to confabulations to justify and backup the validity of their clearly failed responses, making them sound plausible. I like their use of confabulations instead of hallucinations. I think confabulate describes what LLMs are doing much better than hallucinate.
- DelightOne 2y agoHow do they know its not that the LLM ingested a couple Alice in Wonderland analyses?
- zdp7 2y agoThe problem here isn't that you can get the right answer with additional prompting. The problem is how do we verify an answer when we don't know the answer? If it tells me 5 sisters and it should be 6, it failed. If I have to provide additional information, it really isn't reasoning. I would argue to show the ability to reason, it needs to be able to work out the answer completely on it's own. If I were testing a human the only input I would provide is that they are wrong and to try again. AI is very useful, but at this point I'm not comfortable asking questions I don't already know the answer for.
- hellooodarkness 2y agoJust try the question with GPT-4o and the results are more entertaining than I would expected. Prompt: Alice has 3 brothers and she also has 6 sisters. How many sisters does Alice’s brother have? Solve the problem step-by-step Answer: To solve the problem step-by-step: Identify the total number of siblings Alice has: Alice has 3 brothers. Alice has 6 sisters. Understand that Alice herself is one of the sisters: Including Alice, there are a total of 7 sisters (6 sisters + Alice). Determine the number of sisters Alice's brother has: Each of Alice's brothers will have the same number of sisters as Alice does, excluding themselves. Since Alice's brothers have the same set of siblings, including the 7 sisters: Therefore, each of Alice's brothers has 6 sisters (not counting themselves as a sister). So, each of Alice’s brothers has 6 sisters.
- oglop 2y agoI’m sure this will devastate the stock price and people’s pereception that general AI is here and amazing and going to take our jobs and and and. But I am a hacker news peep. I’ll read this and lecture my manager in the next meeting about the shortcomings only to be dismissed and watch money funnel into this monolithic autistic secretary.
- bryan0 2y agoLike others I ran this through gpt-4o and it fails at reasoning because it does not realize Alice is a sister. In fact in one explanation it defines the siblings as: M brothers, N sisters, 1 Alice (!!) I asked gpt “Is Alice a sister?” It answered yes and then answered the problem correctly
- brcmthrowaway 2y ago> M brothers, N sisters, 1 Alice Dude, this is shockingly bad if true Q* algorithm can't come fast enough
- mcguire 2y agoReference 40 is scaring me. A page and a half of authors?
- YeGoblynQueenne 2y agoExtremely antisocial. Conference papers have page limits for references, do they know that?
- Jackson__ 2y ago> Taking inspiration from those problems and aiming for even simpler settings, we arrived at a very > simple problem template that can be easily solved using common sense reasoning but is not entirely > straightforward, of the following form: "Alice has N brothers and she also has M sisters. How > many sisters does Alice’s brother have?". Cute paper, but claiming they invented it, when it's an extremely basic variant of a common[0] riddle asked of LLMs, is kinda weird. [0] https://news.ycombinator.com/item?id=39786666#39794890 https://news.ycombinator.com/item?id=39786666#39794890 (Me, referencing the riddle, 74 days ago, having known it for a good while back then.)
- kaffekaka 2y agoGiving each of the sisters a name changes everything.
- jononomo 2y agoSo much cope in the comments from people who don't want to face the fact that human beings are special in a way that materialism cannot explain.
- HarHarVeryFunny 2y agoAn LLM is a stack of transformer layers (N=96 for GPT-2). Each word/token output is the result of exactly N sentence transformations from input to output. Clearly there are many things (specifically things requiring deliberation and planning) that require an open ended number of thought steps, and therefore could only, even in theory, we done by an LLM in "think step by step" fashion where it breaks the problem down into pieces (maybe recursively) each simple enough to be done in < N of these primitive steps. In practice it seems that LLM's ability (esp. given their lack of any internal working memory) to maintain long term focus for an extended task is poor, so the theoretical benefit of using their own memory as a scratchpad for planning doesn't seem to be realized. The notion of scale it up, and/or train it right, and a transformer can implement AGI, seems to be more wishful thinking than based on any analysis of what the architecture is and is not capable of. The lack of support for continuous incremental learning seems to be a deal-breaker for intelligence that people just ignore.
- bloppe 2y agoThe point of this paper is not to claim that LLMs are dumb. It's applying rigor to the notion that they "reason" in a way that's fundamentally non human. They're more like search engines than abstract reasoning machines. There will always be problem structures that are too different from anything in the training set for them to solve
- TZubiri 2y agoAsking the computer how it thinks is the laziest scientific methodology. It was "As we may think" not "As they may think."
- skyde 2y agoAsking gpt to first output prolog program seem to 100% fix it! Given that Alice has 13 brothers and 31 sisters, we can update the Prolog program with this information. We need to adjust the fact about Alice's siblings and then use the rule to calculate the number of sisters her brothers have. Here is the updated program: ```prolog % Facts siblings(alice, 13, 31). % Alice has 13 brothers and 31 sisters % Rules % Alice's brothers have M + 1 sisters, since Alice is also their sister. sisters_of_brother(TotalSisters) :- siblings(alice, _, M), TotalSisters is M + 1. % Initialization goal :- initialization(main). % Predicate to find and print the total number of sisters Alice's brothers have main :- sisters_of_brother(TotalSisters), writeln(TotalSisters). ``` In this program, the fact `siblings(alice, 13, 31)` represents that Alice has 13 brothers and 31 sisters. The rule `sisters_of_brother(TotalSisters)` calculates the total number of sisters that Alice's brothers have by adding 1 to the number of sisters Alice has (since Alice is also their sister). When we run this program, it will print the total number of sisters Alice's brothers have: 1. Alice has 31 sisters. 2. Each of Alice's brothers will also consider Alice as a sister. 3. Therefore, each brother has 31 + 1 = 32 sisters. The output of the program will be: ``` 32 ``` Thus, Alice's brothers have 32 sisters.
- patal 2y agoI like the riddle. But the framing is unfortunate. When divising riddles, you want ambiguity where it serves the riddle, but be precise elsewhere so that the solver doesn't get needlessly distracted. Their AIW riddle is: "Alice has 4 brothers and she also has 1 sister. How many sisters does Alice’s brother have?" Now it should've been: "How many sisters do Alice's brothers have?" or "..does each of Alice's brothers have". Why single out a specific brother, when you haven't introduced this topic, and it is irrelevant to the riddle? Naturally, a human would ask "Which brother?", fully knowing that it is not important to the riddle. Since this grammatical distraction puts an additional burden on the LLM, the authors muddled their original goal, which was to provide an easy riddle. I think it may have also muddled their data.
- patal 2y agoTheir AIW+ riddle is just ridiculous. It contains so many ambiguities, that there are several correct answers, even though the authors claim there be only one. Which is really unfortunate. Because now it only shows that LLMs have problems answering ill-framed riddles.
- fnordpiglet 2y agoSo interestingly if you are more explicit all of the models reliably answer correct. The confusion is clearly whether Alice is included as a sister in the count of sisters. By stipulating the prompt with “not including herself” it sufficiently disambiguates the problem and most LLMs get it right.