3 ms·
> remember when AI couldn’t count the number of Rs in “strawberry”? GPT-5 still gets this wrong occasionally. Source: I just asked it: How many r's are in "str
by csallen 1y ago
> remember when AI couldn’t count the number of Rs in “strawberry”?
GPT-5 still gets this wrong occasionally. Source: I just asked it: How many r's are in "strawberry"?
It said 2.
(I dislike this method of testing LLMs, as it exploits a very specific and quirky limitation they have, rather than assessing their general usefulness. But still, I couldn't resist.)
- Kerrick 1y agoMy favorite test is to ask it to invent a magic trick given a set of constraints and props. Because magic is generally published very secretly, surprisingly little of it is in most training sets. Pretty much just the most common method & gimmick exposures people tend to parrot online, but not the theory or exact routines behind those methods. The worse an LLM is, the more likely it is to suggest literally impossible actions in the method, like “turn the card over twice to show that it now has three sides. Your spectators can examine the three-sided card.” It can’t tell logic from fantasy, or method from effect.
- jfengel 1y agoI would love to see a trick where you keep turning the same card over with different faces. Bill it as a card trick with just one card. I can think of a half dozen good ways to do it (and you'd need more than one).
- AndyNemmity 1y agoThat's kind of magician thinking though. I'm not sure it's any better than the Elmsley's Dazzle.
- genghisjahn 1y agoThere is a very basic three card trick. You turn each card up one at a time, showing that all three cards are the same(say king spades). End with all three face down in your hand. Then you show all three cards again and all three are a different card(king hearts). At no time are there more or less than 3 cards in your hand.
- AndyNemmity 1y agoI'm a magician. There is a magician AI that is seeded with a ton of tricks that can perform this activity. But it's all context after the fact. There's very little an LLM is going to have about that context as you rightly pointed out.
- goda90 1y agoI wouldn't consider that a measure of it's usefulness or lack thereof, but it is an indicator that LLMs continue to not actually work like a human brain, which isn't surprising if you know the technology but might be to the lay user.
- currymj 1y agoif the router is working properly, it should send questions like this to thinking mode which can handle it well. unfortunately the router doesn't always seem to make the right call.
- akgoel 1y agoI've come to realize that asking an LLM to do something that a purpose-built software program can already do is a mistake. Instead, the LLM must be an agent for that computer program. Therefore, the correct prompt is "write a python program to count the number of letters in a word, and then use it to count the number of Rs in strawberry".
- akomtu 1y agoRemember those interview questions: "how many golf balls fit in a 747?" Turns out they are actually a good way to see if the candidate is a LLM-style imitator or is a someone who knows how to switch to a formal constraints-based thinking, come up with that "python program" to count golf balls and execute it step-by-step.
- r0fl 1y ago“There are 2 — the r’s are in strawberry (letters 3 and 8–9 form the double r).“ That took 4 seconds What a waste of resources
- deleted 1y ago[deleted]
- standardUser 1y agoIs this really a quirky limitation? LLMs seem to struggle with any sort of task requiring a "complete" list of anything. Which is a massive problem because those are super common tasks.
- hereme888 1y ago> how many r's are there in "strawberry"? >> 3 This was for GPT-5 regular