10 ms·

Aren't humans also prone to this? Take the classic trick question, for example: "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.

• mecsred 2y ago
  I think you could reasonably describe that as "pattern matching at the wrong level". If you tell most people the first answer is wrong they will go up a level or two and work out the correct answer.
  • nwhitehead 2y ago
    You would think so... I asked this question in a college-level class with clickers. For the initial question I told them, "This is a trick question, your first answer might not be right". Still less than 10% of students got the right answer.
    • adamisom 2y ago
      Students are more arrogant than LLMs (but still more generally intelligent)
      • roenxi 2y ago
        This idea that students are "more generally intelligent" requires specific arguments to support. ChatGPT alone has a vast breadth of knowledge that is impossible for a human to keep up with as well as superior skills to a student in any number of fields. I can find evidence that it has an IQ of around 124 [0] which is going to stretch most students (although in fairness the same article also speculates an IQ of 0). Students can keep ahead of it with training in specific fields and it has a few weaknesses in specific skills but I think someone could make a reasonable claim that ChatGPT has superior general intelligence. [0] https://medium.com/@soltrinox/the-i-q-of-gpt4-is-124-approx-2a29b7e5821e https://medium.com/@soltrinox/the-i-q-of-gpt4-is-124-approx-...
        • magic_hamster 2y ago
          Claiming that Chatgpt is more intelligent than a student is the same as saying an encyclopedia or a library is more intelligent than a student. Sure they retain more information. But Chatgpt is not AGI and it has no idea what it is even talking about.
        • chmod775 2y ago
          > superior skills to a student in any number of fields It's abhorrent any time we measure pure distilled intelligence. When asked to come up with any non-basic novel algorithm and data structure, it creates nonsense. Especially when you ask it to create vector instruction friendly memory layouts and it can't code in its preferred way. I had some fun trying to make it spit out a brute-force-ish solver for a problem involving basic orbital mechanics and some forces. Wouldn't even want try something more complicated. It can do generalized solvers somewhat, since it can copy that homework, but none that can express the kinds of terms you'd be working with (despite those also having code available in some research papers). Speaking of which, it cannot even figure out some basic truths in orbital mechanics that can be somewhat easily derived from the formulas commonly given, nine times out of ten (you can get there if you're very patient and are able to filter its wrong answers). But at the end of the day it was still a valuable tool to me as I was learning these things myself, since despite being often wrong, it nevertheless spat out useful things I could plug into Google to find more trustworthy sources that would teach me. Really neat if you're going in blind into a new subject.
        • ben_w 2y ago
          > I can find evidence that it has an IQ of around 124 Despite me being someone who is generally impressed by the best LLMs, I think this says more about IQ tests than it does any AI. Which isn't to shame those tests — we made those tests for humans, we were the only intelligence we knew of that used complex abstract language and tools until about InstructGPT — but it does mean we should reconsider what we mean by "intelligence". My gut feeling is that a better measure is how fast we learn stuff. Computers have a speed advantage because transistors outpace synapses by the degree to which a pack of wolves outpaces continental drift (yes I did do the calculation), so what I mean here is how many examples rather than how many seconds. But as I say, gut feeling — this isn't a detailed proposal for a new measure, it likely needs a lot of work to even turn this into something that can be a good measure.
      • deafpolygon 2y ago
        Knowledge != Intelligence
  • deleted 2y ago
    [deleted]
• amy-petrik-214 2y ago
  10 cents! I'm 100% sure
  • SpaghettiCthulu 2y ago
    You are 100% wrong.
    • ignoramous 2y ago
      "A robot must obey orders given it by human beings..." https://g.co/gemini/share/94238c5ed174 https://g.co/gemini/share/94238c5ed174 / https://archive.is/ZTkjI https://archive.is/ZTkjI
      • card_zero 2y ago
        It might still say 5 cents if the problem was "the bat costs 90 cents more than the ball", since 5 cents is overwhelmingly the right answer to questions that look like this.
        • deleted 2y ago
          [deleted]
      • qingcharles 2y ago
        That's a great conversation. I anthropomorphically feel bad for Gemini, in that it actually got a hard question right, and then it was gaslit into believing it was wrong.
• dinobones 2y ago
  This question becomes easy if you think about it algebraically/mathematically, but comically hard thinking about it intuitively, as in, trying to reason with language. bat=ball+$1 (ball+$1)+ball=$1.10 2ball + $1 = $1.10 2ball = $0.10 ball = $0.05
• CooCooCaCha 2y ago
  “Don’t humans do this too?” Comes up in every thread about LLMs yet the conversation changes when you actually ask what you’d trust an LLM to do.
  • SpicyLemonZest 2y ago
    I'm not saying this is you, but in my experience, the conversation changes because people flatly refuse to believe any concrete examples. I would and many companies do trust an LLM with low to medium skilled text processing tasks: internal knowledge sharing, first line customer support, low stakes document review. Github has their famous Copilot product, which I don't personally find too useful but many of my coworkers use.
    • CooCooCaCha 2y ago
      In other words, you don't trust it with anything important.
      • SpicyLemonZest 2y ago
        Really depends on what you understand the word to mean. All of the things I listed are “important” in the sense that they’ve gotta get done, and in the sense that it matters how well they’re done. Would I trust an LLM with something business-critical? No, but there’s quite a lot of people I’d put in the same boat.
        • CooCooCaCha 2y ago
          The difference is some humans exist that can be trusted with business critical tasks but no AIs exist with that level of competence. This is actually a great example of what I'm talking about. All this talk about how useful AIs are, and how humans have flaws and yet the conclusion is always the same. AIs are only useful for tasks that are relatively easy and have a higher tolerance for failure.
• wavemode 2y ago
  When are we going to stop defending LLM intelligence on the grounds that "some people are stupid too"? Human intelligence is not benchmarked by its lowest common denominators (just like how we don't judge LLM's on the basis of tiny 100M parameter models).
  • jacobsimon 2y ago
    The paper linked here starts off by asking whether LLMs can “solve complex problems in ways that resemble human thinking.” Why would we try to answer that question without discussing how humans think?
    • mrjin 2y ago
      Coz we simply don't understand how we understand, period.
      • mewpmewp2 2y ago
        So we shouldn't discuss it or try to understand it at all?
        • mrjin 2y ago
          Where did you get that? You don't know you don't know. It's not going to sell if you literally say I don't know what we're doing...
          • ben_w 2y ago
            I'm a different person than you replied to, but I agree with them. Your previous reply was to: > Why would we try to answer that question without discussing how humans think? And by replying to that with > Coz we simply don't understand how we understand, period. Well, when we don't understand how we understand, that is exactly when we should be discussing how humans think. Or at least how we think we think. And the bat and a ball example relates to research about how we think we think. So yeah, your reply definitely comes across as saying the first half, "So we shouldn't discuss it", while finishing it with "period" also suggests "or try to understand it at all".
• bad_user 2y ago
  People pattern match, but people also apply logic and abstract thinking, far surpassing the abilities of LLMs, and this is a fundamental limitation that won't get fixed by more processing power and training data. The big difference in behavior is in how people or LLMs approach new problems. A LLM is incapable of solving a problem that's similar to one that it was trained on, but with slightly changed requirements. A LLM is also incapable of learning, once you point out their error, or of admitting that it doesn't know. Regarding people, I find it interesting that even lower IQ people are capable of tasks that are currently completely out of reach for AI. It's not just the obvious, such as self-reflection, but even tasks that should've been solved by AI already, such as driving to the grocery store.
  • viraptor 2y ago
    There's lots of assumptions here that we've got examples to disprove: > but people also apply logic and abstract thinking Which people? If we universally did that as a default, elections would look massively different. > A LLM is incapable of solving a problem that's similar to one that it was trained on, but with slightly changed requirements Getting good answers for coding questions on my private code/databases disproves this. The requirements have been changed significantly. I've been through ~20 turn chat with LLM investigating a previously unseen database, suggesting queries to get more information, acting on it to create hypotheses and follow up on them. > A LLM is also incapable of learning, once you point out their error, This is the standard coding agent loop - you feed back the error to get a better answer through in-context learning. It works. > or of admitting that it doesn't know. Response from gpt: «I apologize, but I'm not able to find any reliable information about a person named "Thrust Energetic Aneksy."»
    • photonthug 2y ago
      > This is the standard coding agent loop - you feed back the error to get a better answer through in-context learning. It works. This looks like it works sometimes, but only if "pointing out the error" is coincidentally the same as "clarifying problem spec". Admittedly for really simple cases those are the same or hard to tell apart. But it always seems clear that adding error correction context is similar to adding additional search terms to get to a better stack-overflow page. This feels very different than updating any kind of logical model for the problem representation.
• dTal 2y ago
  Indeed, domain-specific reasoning ability is a seminal psychology result: people are in general sucky at a specific type of logic puzzle, unless you frame it as enforcing a social rule - then they're excellent. https://en.wikipedia.org/wiki/Wason_selection_task https://en.wikipedia.org/wiki/Wason_selection_task
• gcanyon 2y ago
  Since I've known this puzzle since childhood, my "pattern match" answer is the correct answer. I wonder if someone took it up a level and posed a puzzle that looked like this puzzle but had some other subtle complication, whether it would fool me.
• Balgair 2y ago
  > Aren't humans also prone to this? Actually, no. We can do 'real' reasoning and come up with novel conclusions. In fact, we have such an example going all the way back to Plato in Meno, about 2300 years ago. It's the doubling of the square dialog of Socrates and the slave boy. https://classics.mit.edu/Plato/meno.html https://classics.mit.edu/Plato/meno.html (Crtl+F for 'square') Really, the whole dialog is relevant to this discussion with LLMs knowing things.