3 ms·

Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make t

Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that? If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.

f311a 1mo ago
The whole point of his model is to optimize for a very specific benchmark. BUT, he does not use labels when training, so the model does not know the answers.
- embedding-shape 1mo ago
  > The whole point of his model is to optimize for a very specific benchmark. But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
  - f311a 1mo ago
    Why not? There is $700k reward for the next iteration of this benchmark https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2/overview https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2... I would not call this overfitting, it's finetuning for specific task where you have a benchmark.
  - K0balt 1mo ago
    There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations. Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
  - Dylan16807 1mo ago
    Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.
bkaae 1mo ago
I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days. I found it to be a very interesting angle.
- embedding-shape 1mo ago
  > He is showing computers can solve these issues, without using an LLM architecture. Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life. > A lot of people have sort of forgot that machine learning is more than just LLMs these days. Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.
  - dakolli 1mo ago
    Nowhere does he say he built an llm. Hes using a transformer, not an llm.
    - embedding-shape 1mo ago
      > Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?
      - evilmathkid 1mo ago
        Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles
artyomsv 1mo ago
[dead]

f311a 1mo ago

The whole point of his model is to optimize for a very specific benchmark. BUT, he does not use labels when training, so the model does not know the answers.

embedding-shape 1mo ago
> The whole point of his model is to optimize for a very specific benchmark. But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
- f311a 1mo ago
  Why not? There is $700k reward for the next iteration of this benchmark https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2/overview https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2... I would not call this overfitting, it's finetuning for specific task where you have a benchmark.
- K0balt 1mo ago
  There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations. Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
- Dylan16807 1mo ago
  Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.

bkaae 1mo ago

I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days. I found it to be a very interesting angle.

embedding-shape 1mo ago
> He is showing computers can solve these issues, without using an LLM architecture. Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life. > A lot of people have sort of forgot that machine learning is more than just LLMs these days. Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.
- dakolli 1mo ago
  Nowhere does he say he built an llm. Hes using a transformer, not an llm.
  - embedding-shape 1mo ago
    > Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?
    - evilmathkid 1mo ago
      Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles

artyomsv 1mo ago

[dead]