5 ms·

Does anyone here have access to Grok 4 yet? If so, could you please try asking it to solve this basic word search problem [0] and share the results? It's just a

Does anyone here have access to Grok 4 yet? If so, could you please try asking it to solve this basic word search problem [0] and share the results? It's just a simple grid of letters where you have to find the position of each word, the kind of problem that any young child can easily solve. [0] https://imgur.com/VxNP5jG https://imgur.com/VxNP5jG

kadushka 1y ago
These models are not trained on character level input. Why would anyone expect them to perform well on character level puzzles?
- Jensson 1y ago
  They are trained on many billions of tokens of text dealing with character level input, they would be rather dumb if they couldn't learn it anyway. Every human learns that, when you hear the sound "strawberry" you don't hear the double r there, yet you still know the answer.
  - brookst 1y ago
    These models operate on tokens, not characters. It’s true that training budgets could be spent on exhaustively enumerating how many of each letter are in every word in every language, but it’s just not useful enough to be worth it. It’s more like asking a human for the Fourier components of how they pronounce “strawberry”. I mean the audio waves are right there, why don’t you know?
    - yahoozoo 1y ago
      Although a vast majority of tokens are 4+ characters, you’re seriously saying that each individual character of the English alphabet didn’t make the cut? What about 0-9?
      - kadushka 1y ago
        Each character made the cut, but the word "strawberry" is a single token, and that single token is what the model gets as input. When humans read some text, they can see each individual character in the word "strawberry" everytime they see that word. LLMs don't see individual characters when they process input text containing the word "strawberry". They can only learn the spelling if some text explicitly maps "strawberry" to the sequence of characters s t r a w b e r r y. My guess is there are not enough of such mappings present in the training dataset for the model to learn it well.
        nl 1y ago
        > the word "strawberry" is a single token, and that single token is what the model gets as input. This is incorrect. strawberry is actually 4 tokens (at least for GPT but most LLM are similar). See https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
        kadushka 1y ago
        I got 3 tokens: st, raw, and berry. My point still stands: processing "berry" as a single token does not allow the model to learn its spelling directly, the way human readers do. It still has to rely on an explicit mapping of the word "berry" to b e r r y explained in some text in the training dataset. If that explanation is not present in the training data, it cannot learn the spelling - in principle.
        brookst 1y ago
        Exactly. If “st” is 123, “raw” is 456, “berry” is 789, and “r” is 17… it makes little sense to ask the models to count the [17]’s in [123,466,789]: it demands an awareness of the abstraction that does not exist. To the extent the knowledge is there it’s from data in the input corpus, not direct examination of the text or tokens in the prompt.
        asadotzler 1y ago
        So much for generalized intelligence, I guess.
- brrrrrm 1y ago
  emergent behavior. These things are surprisingly good at generalizing
modeless 1y ago
They said they're training a new base model for better multimodal performance soon. I wouldn't expect it to be able to read an image like that today. Maybe if you provided it in text format.
- Szpadel 1y ago
  description from openrouter: > Grok 4 is xAI's latest reasoning model with a 256k context window. It supports parallel tool calling, structured outputs, and both image and text inputs. Note that reasoning is not exposed, reasoning cannot be disabled, and the reasoning effort cannot be specified. unfortunately no requests are passing because of some rate limits
- TheAceOfHearts 1y ago
  As a point of interest and for comparison, Gemini 2.5 Pro is able to generate a Python program that outputs the complete correct solution when run, but it can't figure out how to one-shot the problem if asked directly. This is just a for-fun test to get a sense of how models are progressing; it highlights the jagged nature of their intelligence and capabilities. None of the big AI labs are testing for such a basic problem type, which makes it a bit of an interesting check. I think it's still interesting to see how Grok 4 performs, even if we don't use this test to draw any broader conclusions about what capabilities it offers.
vnchr 1y ago
Mix of hits and misses: https://x.com/i/grok/share/CWE4XhSUlqVe370CehF9At5Tc https://x.com/i/grok/share/CWE4XhSUlqVe370CehF9At5Tc
- drexlspivey 1y ago
  This is grok 3 not 4

kadushka 1y ago

These models are not trained on character level input. Why would anyone expect them to perform well on character level puzzles?

Jensson 1y ago
They are trained on many billions of tokens of text dealing with character level input, they would be rather dumb if they couldn't learn it anyway. Every human learns that, when you hear the sound "strawberry" you don't hear the double r there, yet you still know the answer.
- brookst 1y ago
  These models operate on tokens, not characters. It’s true that training budgets could be spent on exhaustively enumerating how many of each letter are in every word in every language, but it’s just not useful enough to be worth it. It’s more like asking a human for the Fourier components of how they pronounce “strawberry”. I mean the audio waves are right there, why don’t you know?
  - yahoozoo 1y ago
    Although a vast majority of tokens are 4+ characters, you’re seriously saying that each individual character of the English alphabet didn’t make the cut? What about 0-9?
    - kadushka 1y ago
      Each character made the cut, but the word "strawberry" is a single token, and that single token is what the model gets as input. When humans read some text, they can see each individual character in the word "strawberry" everytime they see that word. LLMs don't see individual characters when they process input text containing the word "strawberry". They can only learn the spelling if some text explicitly maps "strawberry" to the sequence of characters s t r a w b e r r y. My guess is there are not enough of such mappings present in the training dataset for the model to learn it well.
      - nl 1y ago
        > the word "strawberry" is a single token, and that single token is what the model gets as input. This is incorrect. strawberry is actually 4 tokens (at least for GPT but most LLM are similar). See https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
        kadushka 1y ago
        I got 3 tokens: st, raw, and berry. My point still stands: processing "berry" as a single token does not allow the model to learn its spelling directly, the way human readers do. It still has to rely on an explicit mapping of the word "berry" to b e r r y explained in some text in the training dataset. If that explanation is not present in the training data, it cannot learn the spelling - in principle.
        brookst 1y ago
        Exactly. If “st” is 123, “raw” is 456, “berry” is 789, and “r” is 17… it makes little sense to ask the models to count the [17]’s in [123,466,789]: it demands an awareness of the abstraction that does not exist. To the extent the knowledge is there it’s from data in the input corpus, not direct examination of the text or tokens in the prompt.
        asadotzler 1y ago
        So much for generalized intelligence, I guess.
brrrrrm 1y ago
emergent behavior. These things are surprisingly good at generalizing

modeless 1y ago

They said they're training a new base model for better multimodal performance soon. I wouldn't expect it to be able to read an image like that today. Maybe if you provided it in text format.

Szpadel 1y ago
description from openrouter: > Grok 4 is xAI's latest reasoning model with a 256k context window. It supports parallel tool calling, structured outputs, and both image and text inputs. Note that reasoning is not exposed, reasoning cannot be disabled, and the reasoning effort cannot be specified. unfortunately no requests are passing because of some rate limits
TheAceOfHearts 1y ago
As a point of interest and for comparison, Gemini 2.5 Pro is able to generate a Python program that outputs the complete correct solution when run, but it can't figure out how to one-shot the problem if asked directly. This is just a for-fun test to get a sense of how models are progressing; it highlights the jagged nature of their intelligence and capabilities. None of the big AI labs are testing for such a basic problem type, which makes it a bit of an interesting check. I think it's still interesting to see how Grok 4 performs, even if we don't use this test to draw any broader conclusions about what capabilities it offers.

vnchr 1y ago

Mix of hits and misses: https://x.com/i/grok/share/CWE4XhSUlqVe370CehF9At5Tc https://x.com/i/grok/share/CWE4XhSUlqVe370CehF9At5Tc

drexlspivey 1y ago
This is grok 3 not 4