7 ms·
Gemini Ultra seems better on logic than GPT4. Still messing around testing but here's a prompt Ultra nailed but GPT4 completely botched: Tabitha likes cookies
by alphabetting 3y ago
Gemini Ultra seems better on logic than GPT4. Still messing around testing but here's a prompt Ultra nailed but GPT4 completely botched:
Tabitha likes cookies but not cake. She likes mutton but not lamb, and she likes okra but not squash. Following the same rule, will she like cherries or pears
https://i.imgur.com/KW6gQbc.jpeg https://i.imgur.com/KW6gQbc.jpeg
https://i.imgur.com/OSHSvLp.png https://i.imgur.com/OSHSvLp.png
- tomasff 3y agoIs image generation working for you?
- codekaze 3y agoIt's working for me (on Gemini Advanced)
- tomasff 3y agoMhm could be a region thing maybe? Not working for me on Gemini Advanced in the UK
- alphabetting 3y agoYeah. I don't have much personal use for image generation though.
- neel8986 3y agoWow. Is this a puzzle from internet. I could have never guessed the answer
- ipsum2 3y agoNote that Gemini pulled the answer off the Internet, while GPT-4 didn't. The answer can easily be found via Google search. Changing up the question a little, I reversed it and asked Ultra and it was unable to answer: Jake likes coke but not pepsi. He likes corn but not popcorn, and he likes pens but not pencils. Will Jake like salmon or cheese? https://i.imgur.com/lWU9HHS.png https://i.imgur.com/lWU9HHS.png edit: why was this downvoted? I don't understand Hacker News, and I've been here for over 12 years.
- c-fe 3y agoI dont think your reversed question makes sense. In the OP example, one item was always smaller/younger then the other item. In your example, I cannot, even as a human, identify the differences
- deleted 3y ago[deleted]
- flerchin 3y ago> even as a human Would all the humans please take one step forward. Not so fast c-fe.
- inhumantsar 3y agoit's based on syllable count
- alphabetting 3y agoHere's a logic question I just made up that GPT-4 failed and Gemini Advanced got right. https://i.imgur.com/3sNr3LW.png https://i.imgur.com/3sNr3LW.png https://i.imgur.com/EIj0nZg.png https://i.imgur.com/EIj0nZg.png
- rvnx 3y agoProof of Gemini cheating: https://i.imgur.com/eYJDFjS.png https://i.imgur.com/eYJDFjS.png Answer about cherries falling from the sky... (there is no question or context beforehand, this is the first question of the chat)
- rvnx 3y agoIn addition just realized it thinks Apple is only one syllable.
- alphabetting 3y agoThat's a bummer. I just made this one up which GPT-4 failed and Gemini got right but definitely need to do more testing to see what's being taken from online and what is actual reasoning. https://i.imgur.com/3sNr3LW.png https://i.imgur.com/3sNr3LW.png https://i.imgur.com/EIj0nZg.png https://i.imgur.com/EIj0nZg.png Edit: When I did your prompt it got it right on the first try fwiwi https://i.imgur.com/E3zYEca.png https://i.imgur.com/E3zYEca.png
- nycdatasci 3y agoGreat example! Thanks for sharing. I tested the same prompt using a variety of GPT-4 model versions and it consistently got the answer wrong. When I continued with "use algebra/formulas to confirm your answer" it corrected the initial mistake, but I'm surprised that it's not doing that out of the box.
- mewpmewp2 3y agoI tried through API with latest GPT-4-Turbo-Preview and this is what I got: ------ Let's break down the information given: The school is two blocks north of the pool. This means if you are at the pool and move two blocks north, you'll arrive at the school. The convenience store is one block south of the school. Therefore, if you start at the school and move one block south, you'll reach the convenience store. Based on the first two statements, the relative positions from north to south are: School, Convenience Store, Pool. Thus, the third statement saying "The pool is south of the convenience store" is indeed true, based on the positions provided by the first two statements. ----- But it starts weirdly messing up, when I asked it to explain it using coordinates. --- Let's use a simple coordinate system to represent the positions of the school, the pool, and the convenience store. Let's assume each block corresponds to one unit in this system. We'll start with the pool as our reference point at coordinates (0,0). Remember, moving north increases the y-coordinate, while moving south decreases it. The school is two blocks north of the pool. If the pool is at (0,0), moving two blocks north puts the school at (0, +2). The convenience store is one block south of the school. If the school is at (0, +2), moving one block south puts the convenience store at (0, +1). Given these positions: The pool is at (0, 0). The school is at (0, +2). The convenience store is at (0, +1). Therefore, instead of the pool being south of the convenience store, the pool is actually located to the north of the convenience store if you consider "north" to mean increasing y-coordinates. The convenience store, being at (0, +1), is southward relative to the school but not to the pool. The pool, being at the origin point (0,0), is south of both the school and the convenience store in terms of their y-coordinates. Thus, the third statement, "The pool is south of the convenience store," is false according to the given coordinates. --- It gets coordinates and everything right, but still somehow messes up North and South.
- DreamGen 3y agoI would not be so quick to jump to conclusions. GPT-4 beats it easily in this simple logic puzzle: https://www.reddit.com/r/singularity/comments/1altttv/bard_advanced_gemini_ultra_is_worse_than_gpt4_at/ https://www.reddit.com/r/singularity/comments/1altttv/bard_a... We need more data.
- ChatGTP 3y agoSo logical, it’s a really logical question right ? I mean who doesn’t like food based on the number of syllables in English name of foods.
- shantara 3y agoI would have never guessed the answer. With such little data available, one can invent any arbitrary rules to fit their favorite answer. It would be more impressive to practical use cases, if a LLM simply said that it's impossible to guess without inventing their own reasoning or looking up the answer online.
- OJFord 3y agoSame, I had to look to see what the intended answer was. In fairness though, GPT4 was objectively incorrect, it's not even internally consistent or coherent - it either thinks b & h are vowels, or that lamb and squash don't end in those letters, or has changed its mind about the rule mid-sentence, or something.
- deleted 3y ago[deleted]
- a1o 3y agoI guess I learned that syllables count very different in English than I assumed.
- deleted 3y ago[deleted]
- diggan 3y agoWith some additional guidance and prompting to GPT4 on ChatGPT I've gotten it to at least output the correct solution sometimes (7 correct answers out of 10 tries): Find the correct answer to this riddle: > Tabitha likes cookies but not cake. She likes mutton but not lamb, and she likes okra but not squash. Following the same rule, will she like cherries or pears? Employ the following strategy: - Suggest a list of 5 unique and novel patterns that potentially can find the answer - Check if the patterns applies without exceptions - Slowly double-check if the patterns was correctly applied, that you correctly assessed if it's accurate or not - Explain your reasoning for each step to ensure nothing vital was missed
- ipsum2 3y agoGreat prompt. GPT-4 was able to answer this, but Gemini Ultra was not: Jake likes coke but not pepsi. He likes corn but not popcorn, and he likes pens but not pencils. Will Jake like salmon or cheese?
- tomasff 3y agoGemini Ultra answered this correctly for me: " It's impossible to say for sure if Jake will like salmon or cheese based on the information given."
- ipsum2 3y agoWhile technically true, you could say that about all riddles. See GPT-4's explanation here, which intuits the rule and answers the riddle correctly. https://chat.openai.com/share/b1452950-b493-4e27-b097-e64f211153a4 https://chat.openai.com/share/b1452950-b493-4e27-b097-e64f21...
- tomasff 3y ago~~Well, worth nothing that the original prompt you provided didn't include the suggested strategy~~ Edit: My bad didn't read the parent comment properly
- jwithington 3y agoI’m a human and i dont know the answer to that quesiton
- LightMachine 3y agoIf we want to test these beasts in logic, we should probably start using actual formalized logic, rather than English. In just one test, Gemini flopped hard, while GPT-4-Turbo nailed it. Here is my prompt: Below is a well-typed CoC function: foo : ∀(P: Nat -> *) ∀(s: ∀{n} -> ∀(x: (P n)) -> (P (n + 1))) ∀(z: (P 0)) (P 3) = λP λs λz (s (s (s z))) Below is an incomplete CoC function: foo : ∀(P: Nat -> *) ∀(f: ∀{n} -> ∀(x: (P n)) -> (P (n * 3))) ∀(g: ∀{n} -> ∀(x: (P n)) -> (P (n * 2))) ∀(h: ∀{n} -> ∀(x: (P n)) -> (P (n + 5))) ∀(z: (P 1)) (P 17) = λP λf λg λh λz {{FILL_HERE}} Complete it with the correct replacement for {{FILL_HERE}}. Your answer must contain only the correct answer, and nothing else. - *GPT-4-Turbo answer:* `(f (g (h (g z))))` (correct) - *Gemini Advanced answer:* `h (h (g (f z)))` (wrong) Also, Gemini couldn't follow the "answer only with the solution" instruction and provided a bunch of hallucinated justifications. I think we have a winner... (screenshots: https://imgur.com/a/GotG0yF https://imgur.com/a/GotG0yF)
- ajross 3y ago> I think we have a winner... It makes me sad that the complete and total lack of an objective way to measure these products means that the coming decades will be filled with this kind of hyper-specific gotcha test made in inappropriately confident internet posts. Literally this could have been down to one extra book in someone's training corpus, or a tokenizer that failed to understand λ as a non-letter. But no matter, "we have a winner!". It's the computer science equivalent of declaring global warming a fraud because it snowed last night.
- dreamcompiler 3y agoDisagree. People are going to rely on these things, and when they make stupid but confident mistakes (i.e. they produce bullshit), they are dangerous. An AI system that produces right answers 90% of the time but 10% of the time drives your car into a lane divider, or says "there are 4 US states that start with 'K'" or "Napoleon was defeated at the Battle of Gettysburg" is worse than useless: It's dangerous. As long as we call it a bullshit parlor trick, no problem. But unfortunately people are making important decisions based on these things.
- planb 3y agoPlease anyone correct me if I'm wrong: LLMs cannot solve this kind of riddle. This has nothing to do with their capabilities for logical reasoning, but with the way words are represented as tokens. While they might know that "apples" has two syllables because that is mentioned somewhere in their training data, if you make up a fruit "bratush" a human will see that as two syllables, but this might be 1 to 7 tokens to a LLM without any information about the word itself.
- mborch 3y agoChatGPT knows how to answer this question, but how I don't know. Perhaps it's programmed to answer that question?
- huytersd 3y agoWell I tried it out in GPT4 with made up words- Tabitha likes bratush but not zot. She likes protel but not kig, and she likes motsic but not pez. Following the same rule, will she like tridos or kip Given the examples, one speculative pattern could be that Tabitha likes words with at least two syllables or a certain complexity in structure. Therefore, following this speculative rule, Tabitha might like “tridos” more than “kip.”
- bitmovements 3y agoZot is a word already being used in the world: verb. (slang) To zap, kill, or destroy. So is protel: https://en.m.wikipedia.org/wiki/Protel https://en.m.wikipedia.org/wiki/Protel So is kig: https://en.m.wiktionary.org/wiki/kig https://en.m.wiktionary.org/wiki/kig Pez is a well known brand name in America. Kip is commonly used name in America. Motsic is a fairly common last name from searching. Tridos is used all over the internet as a brand name so this all seems probable to be in the training data. These words are not new nor are they made up.
- ComputerGuru 3y agoGPT-4 has a really good tokenizer that is able to retain and use more information about input tokens than one might naively think.
- ramoz 3y agoFor me, GPT nailed it with this prompt: Tabitha likes cookies but not cake. She likes mutton but not lamb, and she likes okra but not squash. Following this rule, will she like cherries or pears? Take a deep breath in order to solve the problem, and do not rush to naïve solutions.
- ramoz 3y agohttps://chat.openai.com/share/c89aaac5-a6f7-4f50-9cac-ae477737cfc0 https://chat.openai.com/share/c89aaac5-a6f7-4f50-9cac-ae4777...
- lupire 3y agoThat looks like 3.5, which gets lucky sometimes because it has memorized the Internet. Watch it flail here: https://chat.openai.com/share/c2b14eb0-dc45-4eaf-a547-951ff068499a https://chat.openai.com/share/c2b14eb0-dc45-4eaf-a547-951ff0...
- andrewla 3y agoIn what sense is this a question involving "logic"?
- jiggawatts 3y agoYou say Gemini “nailed it”, but that’s just because it guessed what you were thinking, not because it knew the right answer. For example, it’s equally valid to say that Tabitha likes small foods since cookies are small and cakes are large, and lamb is the smaller younger version of sheep — also known as mutton. Hence she likes cherries because they’re smaller… or taste better… or her uncle abused her with a pear… or whatever. You haven’t actually asked a logic question where there is a clear and unambiguous answer that can be derived using formal methods starting from clearly stated axioms. If you gave this question to a bunch of humans, they would give you inconsistent guesses as well — not because they’re wrong but because the question has no single right answer.
- voidhorse 3y agoTotally tangential, but I absolutely despise logic teasers of this kind. First of all, they are so completely divorced from patterns of culturally conditioned human reasoning as to make them come off completely absurd (most people reason about their food preferences using a logic of tastes, not syllables in a word). The game is less about logic and more about ignoring message contents, moving up a level, and treating the text as data without any legitimate evidence that you are justified in doing so. This is not a logic problem, it's a "guess the register shift/meta language" problem. The problem is about noticing that the question is not about the message content but about the structure of the message itself, and requires a bold leap. In real life justifying the conclusion would actually require a very sophisticated inference that allowed you to rule out the much more common application of a logic of tastes or cultural codes completely.