5 ms·

I trained a small transformer in 1.5hrs and it beats many LLMs

671 points by porridgeraisin 2mo ago

embedding-shape 2mo ago
Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that? If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.
- f311a 2mo ago
  The whole point of his model is to optimize for a very specific benchmark. BUT, he does not use labels when training, so the model does not know the answers.
  - embedding-shape 2mo ago
    > The whole point of his model is to optimize for a very specific benchmark. But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
    - f311a 2mo ago
      Why not? There is $700k reward for the next iteration of this benchmark https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2/overview https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2... I would not call this overfitting, it's finetuning for specific task where you have a benchmark.
    - K0balt 2mo ago
      There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations. Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
    - Dylan16807 2mo ago
      Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.
- bkaae 2mo ago
  I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days. I found it to be a very interesting angle.
  - embedding-shape 2mo ago
    > He is showing computers can solve these issues, without using an LLM architecture. Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life. > A lot of people have sort of forgot that machine learning is more than just LLMs these days. Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.
    - dakolli 2mo ago
      Nowhere does he say he built an llm. Hes using a transformer, not an llm.
      - embedding-shape 2mo ago
        > Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?
        evilmathkid 2mo ago
        Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles
- artyomsv 2mo ago
  [dead]
xeonax 2mo ago
Even cooler is his about me mention of saving his own life https://mvakde.github.io/ https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
- qlm 2mo ago
  Crazy, considering rhabdo isn't that rare.
  - p-e-w 2mo ago
    “What do you call a medical student who graduated at the bottom of their class?” “Doctor.”
    - draw_down 2mo ago
      Well, I guess it would be nice if the graduation cutoff were above the level of “knows what rhabdo is”
    - echelon 2mo ago
      I studied biochem in undergrad and my classes were full of premed students. I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds. I've never looked at doctors the same way again after college. I looked up to them as a child, yet after seeing how the sausages were made, I started to doubt everything. I frequently ask doctors, who spend all of ten minutes with me while the nurses do all the work, about the molecular specifics of what they're talking about. They talk down to me as if they're explaining to a child, yet they're frequently quite wrong. I'm not trying to sound superior to them, but I'm shocked they seem to care so little about the subject. It doesn't give me much hope about what they know and their abilities or competency. I suspect surgeons and specialists are a different breed and aren't like this at all. And to be clear, this isn't everyone. But it does seem to be the majority I've interacted with throughout my life. When they act disgruntled at patient interaction, I detest that their profession tries to cap the number of med students per year. We should be letting in as many med students as we can take. We should let doctors from overseas immigrate and easily become practicing doctors here in the US. We should provide easy paths for nurses to become doctors. The premed students in my university were chiefly concerned about money and prestige. They drove BMWs gifted to them by their parents and laughed at what I drove and how hard I studied. I had to put up with their bullying for years. I know not everyone who studies to become a doctor is like that, but it permanently skewed my view of their profession.
eis 2mo ago
> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training. That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests. The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?
- bkaae 2mo ago
  Then nothing - that's awesome. People think that LLMs are the know-all do-all solution to every problem now. Putting solutions in terms of cents is a great way to potentially win over some ai boosters imo. There are other ways to solve hard problems.
- evilmathkid 2mo ago
  > The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt) And more compute increases the perf (after dealing with other scaling problems)
- boomlinde 2mo ago
  > The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. Nothing about saying that it cost 67 cents implies that it will. Knowing only that it costs 67 cents you also have no reasonable basis for extrapolation. It doesn't indicate a trend whatsoever.
larodi 2mo ago
"I don’t understand why others didn’t figure this out" - how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.
- woah 2mo ago
  Could be that many presumed ML experts don't even know how to train on the evaluation set
pwmglenn 2mo ago
Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?
kvn95ss 2mo ago
> Also, I’m not sure whether “general reasoning” even exists in the first place? Maybe humans are specialised too I have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.
evilmathkid 2mo ago
Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples. - Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient) - Can the perf be improved? Yes but the competition is ongoing so can't talk about it - Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho - Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML
- evilhackerdude 2mo ago
  nothing like legendary shrugging and keeping the mind open
- kuczmama 2mo ago
  Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?
  - evilmathkid 2mo ago
    Yeah I've reached much higher perf but - it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me - more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)
westurner 2mo ago
How does it perform on ARC-AGI-3? There was this a few weeks ago: "Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163 https://news.ycombinator.com/item?id=48938163 >> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol What does that do with 5.6 Luna instead of the expensive models? What of 'schema' would improve the performance of mdlARC? mdlARC: https://github.com/mvakde/mdlARC https://github.com/mvakde/mdlARC There's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week: "A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s
- evilmathkid 2mo ago
  its not gonna do well on ARC-3 without some significant changes and effort The new arch in that video is kinda misleading. Didn't really compare against proper baselines
  - westurner 2mo ago
    Yeah, but still, how does that agent perform if paired with this inexpensively trained transformer instead of the more expensive foundation models?
GovPulse 2mo ago
[flagged]
kriro 2mo ago
Thanks for motivating me to work a bit on non-LLM things again :)
hn45e7pbij 2mo ago
Nice to see arc-agi-1 framed this way — I'd been circling the same idea without the right words.
dhruv3006 2mo ago
> The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training. I saw this on the community note for the last blog you wrote - anything to do here.
- evilmathkid 2mo ago
  Not true. This is allowed in a metalearning context. Its called transductive learning and has existed since the 90s: https://en.wikipedia.org/wiki/Transduction_(machine_learning) https://en.wikipedia.org/wiki/Transduction_(machine_learning... I address this in more detail in the blog
bee_rider 2mo ago
I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop? > Training on the eval puzzles is cheating / “training on test” > No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. > Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles. > Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid. > The ARC, the label is only the test pair’s output grid in an eval puzzle. > These labels were not trained on. They are hidden. You can delete it beforehand if you wish I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?
- porridgeraisin 2mo ago
  Basically, you have a bunch of Q,A pairs in the training dataset. Here, it was trained to next-word predict the question itself, as well as next-word predict the answer given the question as prompt. This is bog-standard, no one's complaining. In the test dataset's Q,A pairs, it was only trained to next-word predict the question itself, and it was not given the answer at all. It was then evaluated by seeing if it is able to output A_test given the Q_test as prompt. What would be cheating is training it to produce A_test (given Q_test as prompt) as well, since then you can always make a model that scores 100% by just memorising Q_test, A_test pairs. The complaints online mostly stem from not reading that properly and assuming they trained on Q_test,A_test instead of just Q_test. This is further because these days large LLMs are inadvertently trained on many benchmark solutions even unintentionally due to the massive scale of data and the infeasibility of auditing it all. But none of that is the case here. The reason you want to train on Q_test is because in these AR transformer models, they learn useful composable encodings of Q by simply learning to next-word predict Q. So you enable the model to learn composable encodings of the test questions, so that it can hopefully "connect it" to an earlier train problem it had seen, and adapt the solution it had seen for that, much like humans do in school exams. Without this step, you are making it difficult for the model to "connect" the test question to a train question it had seen earlier, and then it still has to adapt the solution. This way, you precompute that "this test question is like this train question" and then during the exam you only have to do the adapting the solution part after a simpler "retrieval" process. You can just think of next-word training Q_test as a "retrieval" process. This practice often used in continual learning or "test time training" is not yet useful in general real world ML tasks due to the differences in memory and compute requirements, and more so the general fragility of training large neural networks in a streaming realtime way (as opposed to large data, batched), versus inferencing from a static neural network. It is due to that fragility that I believe (correct me if I am wrong) this guy had to train on a batch of Q_tests. If you enforced that you will not provide Q_test_2 before they answer Q_test_1, the performance will drop. While the increased compute and memory is difficult to solve inherently, there are various efforts being made to fix the fragility, especially in reinforcement learning where this is called "streaming RL", there is revival of interest as seen in RLC 2026. [Note] Arc-AGI-1 doesn't have any actual english words or such, but it's simpler to pretend it was a basic Q&A benchmark to explain the above
maz1b 2mo ago
Just wanted to say that the rhabdo part mentioned on your site was really impressive! Speaking as a medical doctor and full stack engineer myself.
- evilmathkid 2mo ago
  thanks! was incredibly scary when it happened
docheinestages 2mo ago
Isn't this cheating? Or rather, are frontier agents only looking at one question at a time? If I understand correctly, you're looking at all the examples of the exam questions. If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore.
- nullbio 2mo ago
  Why would that be cheating? That's what humans do when they learn, they look for the signals and patterns that reduce the possible set of answers so they can converge on the solution and narrow the search space.
  - docheinestages 2mo ago
    I mean, I'm interested to know if the frontier models also get to see all questions at once. Then it's more fair game than if they just see one question at a time.
    - JBits 2mo ago
      They look at each problem individually. As the ARC AGI guidelines [1] state, 'a core design principle of ARC-AGI is that the test taker must not know what the test will be.' [1] https://arcprize.org/policy#dataset-security https://arcprize.org/policy#dataset-security
deleted 2mo ago
[deleted]
6_7 2mo ago
I'm so sorry, in how many?
lackoftactics 2mo ago
Sounds like a good day to be you, top 5 on Kaggle with a publication like this. It seems like you will be on a plane to SF shortly
rappatic 2mo ago
I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.
- xdavidliu 2mo ago
  the caveat is that we are not learning those small number of samples from scratch, since we're coming in with a large amount of training already, much of it from before we were even born
  - nomel 2mo ago
    I've seen a few videos about people having their girlfriend/dad/etc play very simple video games for the very first time. It's a really great example of this. The most basic, simple, mechanics are difficult for them, and take surprising repetition to remember, because they have zero framework around any of it. It's really similar to watching a genetic algorithm learn to play Mario, but on fast forward.
- 10xDev 2mo ago
  No, the unsolved problem of AI is continuous learning. We never stop learning, we don't have a "training phase". You are always updating your world model even when you sleep. Also more quality training data does lead to greater learning efficiency as you have more priors to work with.
  - suddenlybananas 2mo ago
    Why not both?
  - matteoraso 2mo ago
    >We never stop learning, we don't have a "training phase". We kind of do, if you count critical periods in childhood.
foota 2mo ago
> I agree that its rare to see to face problem sets in real life where every problem is given at once. Even if it is (like an exam), humans can usually only attempt one at a time Just one small snippet that I thought was interesting. I would always read through ~the entire exam before starting. Both so that I could find the problems most approachable to me, but also because sometimes it helps me figure out the rest of the questions :-)
- evilmathkid 2mo ago
  same! I'd often learn during the exam by solving an easier problem and then that would let me tackle a hard problem
foota 2mo ago
> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is possible Guarantees no synthetic data can be used It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created) I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?
- nicman23 2mo ago
  how would you as a human know the answer to the arc-agi questions?
  - foota 2mo ago
    I spent years learning logic and doing puzzles. I don't think a baby or even average kid could solve these.
    - anon291 2mo ago
      Predicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions. > I don't think a baby or even average kid could solve these The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).
- evilmathkid 2mo ago
  train from scratch only during the 12 hours allowed on Kaggle Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun
  - foota 2mo ago
    [dead]
TZubiri 2mo ago
>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. I disagree, but i agree that training with answers is worse. In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was the same exam sometimes they were 'slightly' different. In no occasions were the answers shared, and being math exams it wouldn't have made a difference, since solving the exercises is the actual training that allows replication of results and adaptation in testing. It's a grey area for sure, but it's a quantitative matter, studying 20 different exams for 2 months is quite different than trying out 1 exam 1 week prior to an exam to verify all is well. It's called teaching to the test, not teaching to the test and answers. In essence OP holds a naive version of what cheating is, and thus they think they are absolved, when actual cheating is much more nuanced. Many such cases. Fwiw, the second uni I dropped out of was worse in that some students just used their phone during tests and talked with each other or googled. OP sounds like a student from Uni 1 claiming they don't do what Uni 2 students do. And for reference, the exams I did pass I did by just reading the whole bibliography on my own, and doing exercises from the book if needed, I was passing with like 80-90%, never did I have to get a copy of a previous exam and study that, I think it's a ridiculous concept that has been normalized to meet an increasing societal pressure on everyone being an elite graduate (we can't ALL be elite), and if this repo is successful, it's because this attitude is so normalized that it's seeping into machine learning by diffusing the lines between training and testing set, and increasing the ratio between one and the other. Hell, I'm not surprised that the software that the mass of test -studiers develop is software that studies tests. In the same manner that the software that cheaters develop is software that cheats Guardrails and breaks ToSes
- evilmathkid 2mo ago
  > In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam" What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs). (There's also access to experts who know the answer, which kaggle bans by banning the internet) Lucas describes it well here (and his original tweet up the thread): https://x.com/giffmana/status/2002128356901597509 https://x.com/giffmana/status/2002128356901597509 -- Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2
deleted 2mo ago
[deleted]
porridgeraisin 2mo ago
@mods I am not sure the editorialized title is better! (It is a bit clickbaity, since this is a transformer that does not compete with LLMs at all outside the arc-agi-1 benchmark)
ACV001 2mo ago
Nice. It took me an hour just to understand what this competition is about :)
RoadieRoller 2mo ago
I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him? https://news.ycombinator.com/item?id=49412396 https://news.ycombinator.com/item?id=49412396
- evilmathkid 2mo ago
  Yeah but I posted this work back in Dec-Feb lol
  - RoadieRoller 2mo ago
    My above comment wasn't aimed at you. Thanks for the response though! What you did is phenomenal, and I wasn't taking a dig at you in any way. What I intended in my comment was that, may be starting from scratch (like you did many months before, and what PaulG suggested recently) is the way to go for future job prospects and startups. Most of the comments in the thread I posted was negative for PaulG saying that.
    - evilmathkid 2mo ago
      Yeah agree with pg completely. Multiple people I know did this
AmazingEveryDay 2mo ago
So this achieves 44%, I'm wondering: To consider yourself a reasonably competent human, what % to achieve on ARC-AGI-1? Thanks in advance to anyone that knows, can answer!
5555watch 2mo ago
It means the questions including their answers are dependent. Ie, theres a data generating process for them, that the model uncovers. Like a KNN is known to have near Bayes accuracy as k/n to 0, n to infty, k to infty. The data reveals the dgp. I suspect if you feed unrelated or even garbage questions into the eval set, it would reduce the performance.
usernametaken29 2mo ago
I think you’re asking the right questions, sample inefficiency is horrible in modern LLMs. Despite this, I saw your analysis: > The biggest increases in scores were due to Modern architecture (SwiGlu instead of GELU, RMSnorm not layernorm, etc.) More data diversity, better shuffling of data scaling up: 8 layers instead of 4 This is commonly called squeezing the lemon and is usually a bit of a last resort. You should be able to achieve near SoTa with your new method, before you squeeze any lemons. This is, because the old SoTa is typically not using new optimisers and thus your results will be distorted by a large margin. In terms of sample efficiency I want to add two things: Runtime per-puzzle fine tuning is a very good target that provides a LOT of information. People have not looked at evolutionary methods to harness induction since the 90ies - if I was to work on ARC ever again I’m fairly certain this is where I’d look. Best of luck, padawan
juggle73 2mo ago
[flagged]

671 points by porridgeraisin 2mo ago

Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that? If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.

f311a 2mo ago
The whole point of his model is to optimize for a very specific benchmark. BUT, he does not use labels when training, so the model does not know the answers.
- embedding-shape 2mo ago
  > The whole point of his model is to optimize for a very specific benchmark. But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
  - f311a 2mo ago
    Why not? There is $700k reward for the next iteration of this benchmark https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2/overview https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2... I would not call this overfitting, it's finetuning for specific task where you have a benchmark.
  - K0balt 2mo ago
    There are plenty of applications where a machine learning system needs to optimize for a very limited data set that is still intractable by linear logic systems of reasonable scale and complexity. It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach. Obviously a lot of other ways to achieve similar goals, but it’s cool to see someone back porting the modern tools towards older style optimizations. Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
  - Dylan16807 2mo ago
    Overfitting, as well as the specific instances I've seen of the word benchmaxxing, involve knowing the answers and training to those answers. That did not happen here. The model is limited in scope, which means it's not being scored on generic intelligence, but neither is it defective and terrible at solving new problems inside its scope, like you get with overfitting.
bkaae 2mo ago
I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days. I found it to be a very interesting angle.
- embedding-shape 2mo ago
  > He is showing computers can solve these issues, without using an LLM architecture. Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life. > A lot of people have sort of forgot that machine learning is more than just LLMs these days. Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.
  - dakolli 2mo ago
    Nowhere does he say he built an llm. Hes using a transformer, not an llm.
    - embedding-shape 2mo ago
      > Nowhere does he say he built an llm. Hes using a transformer, not an llm. Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results". Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?
      - evilmathkid 2mo ago
        Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles
artyomsv 2mo ago
[dead]

xeonax 2mo ago

Even cooler is his about me mention of saving his own life https://mvakde.github.io/ https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)

qlm 2mo ago
Crazy, considering rhabdo isn't that rare.
- p-e-w 2mo ago
  “What do you call a medical student who graduated at the bottom of their class?” “Doctor.”
  - draw_down 2mo ago
    Well, I guess it would be nice if the graduation cutoff were above the level of “knows what rhabdo is”
  - echelon 2mo ago
    I studied biochem in undergrad and my classes were full of premed students. I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds. I've never looked at doctors the same way again after college. I looked up to them as a child, yet after seeing how the sausages were made, I started to doubt everything. I frequently ask doctors, who spend all of ten minutes with me while the nurses do all the work, about the molecular specifics of what they're talking about. They talk down to me as if they're explaining to a child, yet they're frequently quite wrong. I'm not trying to sound superior to them, but I'm shocked they seem to care so little about the subject. It doesn't give me much hope about what they know and their abilities or competency. I suspect surgeons and specialists are a different breed and aren't like this at all. And to be clear, this isn't everyone. But it does seem to be the majority I've interacted with throughout my life. When they act disgruntled at patient interaction, I detest that their profession tries to cap the number of med students per year. We should be letting in as many med students as we can take. We should let doctors from overseas immigrate and easily become practicing doctors here in the US. We should provide easy paths for nurses to become doctors. The premed students in my university were chiefly concerned about money and prestige. They drove BMWs gifted to them by their parents and laughed at what I drove and how hard I studied. I had to put up with their bullying for years. I know not everyone who studies to become a doctor is like that, but it permanently skewed my view of their profession.

eis 2mo ago

> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training. That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests. The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?

bkaae 2mo ago
Then nothing - that's awesome. People think that LLMs are the know-all do-all solution to every problem now. Putting solutions in terms of cents is a great way to potentially win over some ai boosters imo. There are other ways to solve hard problems.
evilmathkid 2mo ago
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt) And more compute increases the perf (after dealing with other scaling problems)
boomlinde 2mo ago
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. Nothing about saying that it cost 67 cents implies that it will. Knowing only that it costs 67 cents you also have no reasonable basis for extrapolation. It doesn't indicate a trend whatsoever.

larodi 2mo ago

"I don’t understand why others didn’t figure this out" - how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.

woah 2mo ago
Could be that many presumed ML experts don't even know how to train on the evaluation set

pwmglenn 2mo ago

Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?

kvn95ss 2mo ago

> Also, I’m not sure whether “general reasoning” even exists in the first place? Maybe humans are specialised too I have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.

evilmathkid 2mo ago

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples. - Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient) - Can the perf be improved? Yes but the competition is ongoing so can't talk about it - Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho - Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML

evilhackerdude 2mo ago
nothing like legendary shrugging and keeping the mind open
kuczmama 2mo ago
Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?
- evilmathkid 2mo ago
  Yeah I've reached much higher perf but - it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me - more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)

westurner 2mo ago

How does it perform on ARC-AGI-3? There was this a few weeks ago: "Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163 https://news.ycombinator.com/item?id=48938163 >> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol What does that do with 5.6 Luna instead of the expensive models? What of 'schema' would improve the performance of mdlARC? mdlARC: https://github.com/mvakde/mdlARC https://github.com/mvakde/mdlARC There's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week: "A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s

evilmathkid 2mo ago
its not gonna do well on ARC-3 without some significant changes and effort The new arch in that video is kinda misleading. Didn't really compare against proper baselines
- westurner 2mo ago
  Yeah, but still, how does that agent perform if paired with this inexpensively trained transformer instead of the more expensive foundation models?

GovPulse 2mo ago

[flagged]

kriro 2mo ago

Thanks for motivating me to work a bit on non-LLM things again :)

hn45e7pbij 2mo ago

Nice to see arc-agi-1 framed this way — I'd been circling the same idea without the right words.

dhruv3006 2mo ago

> The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training. I saw this on the community note for the last blog you wrote - anything to do here.

evilmathkid 2mo ago
Not true. This is allowed in a metalearning context. Its called transductive learning and has existed since the 90s: https://en.wikipedia.org/wiki/Transduction_(machine_learning) https://en.wikipedia.org/wiki/Transduction_(machine_learning... I address this in more detail in the blog

bee_rider 2mo ago

I think(?) you’ve already probably done a good job of explaining this criticism for semi-informed people. But can you dumb it down even more for those of us who are almost entirely out-of-the-loop? > Training on the eval puzzles is cheating / “training on test” > No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. > Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles. > Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid. > The ARC, the label is only the test pair’s output grid in an eval puzzle. > These labels were not trained on. They are hidden. You can delete it beforehand if you wish I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?

porridgeraisin 2mo ago
Basically, you have a bunch of Q,A pairs in the training dataset. Here, it was trained to next-word predict the question itself, as well as next-word predict the answer given the question as prompt. This is bog-standard, no one's complaining. In the test dataset's Q,A pairs, it was only trained to next-word predict the question itself, and it was not given the answer at all. It was then evaluated by seeing if it is able to output A_test given the Q_test as prompt. What would be cheating is training it to produce A_test (given Q_test as prompt) as well, since then you can always make a model that scores 100% by just memorising Q_test, A_test pairs. The complaints online mostly stem from not reading that properly and assuming they trained on Q_test,A_test instead of just Q_test. This is further because these days large LLMs are inadvertently trained on many benchmark solutions even unintentionally due to the massive scale of data and the infeasibility of auditing it all. But none of that is the case here. The reason you want to train on Q_test is because in these AR transformer models, they learn useful composable encodings of Q by simply learning to next-word predict Q. So you enable the model to learn composable encodings of the test questions, so that it can hopefully "connect it" to an earlier train problem it had seen, and adapt the solution it had seen for that, much like humans do in school exams. Without this step, you are making it difficult for the model to "connect" the test question to a train question it had seen earlier, and then it still has to adapt the solution. This way, you precompute that "this test question is like this train question" and then during the exam you only have to do the adapting the solution part after a simpler "retrieval" process. You can just think of next-word training Q_test as a "retrieval" process. This practice often used in continual learning or "test time training" is not yet useful in general real world ML tasks due to the differences in memory and compute requirements, and more so the general fragility of training large neural networks in a streaming realtime way (as opposed to large data, batched), versus inferencing from a static neural network. It is due to that fragility that I believe (correct me if I am wrong) this guy had to train on a batch of Q_tests. If you enforced that you will not provide Q_test_2 before they answer Q_test_1, the performance will drop. While the increased compute and memory is difficult to solve inherently, there are various efforts being made to fix the fragility, especially in reinforcement learning where this is called "streaming RL", there is revival of interest as seen in RLC 2026. [Note] Arc-AGI-1 doesn't have any actual english words or such, but it's simpler to pretend it was a basic Q&A benchmark to explain the above

maz1b 2mo ago

Just wanted to say that the rhabdo part mentioned on your site was really impressive! Speaking as a medical doctor and full stack engineer myself.

evilmathkid 2mo ago
thanks! was incredibly scary when it happened

docheinestages 2mo ago

Isn't this cheating? Or rather, are frontier agents only looking at one question at a time? If I understand correctly, you're looking at all the examples of the exam questions. If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore.

nullbio 2mo ago
Why would that be cheating? That's what humans do when they learn, they look for the signals and patterns that reduce the possible set of answers so they can converge on the solution and narrow the search space.
- docheinestages 2mo ago
  I mean, I'm interested to know if the frontier models also get to see all questions at once. Then it's more fair game than if they just see one question at a time.
  - JBits 2mo ago
    They look at each problem individually. As the ARC AGI guidelines [1] state, 'a core design principle of ARC-AGI is that the test taker must not know what the test will be.' [1] https://arcprize.org/policy#dataset-security https://arcprize.org/policy#dataset-security

deleted 2mo ago

[deleted]

6_7 2mo ago

I'm so sorry, in how many?

lackoftactics 2mo ago

Sounds like a good day to be you, top 5 on Kaggle with a publication like this. It seems like you will be on a plane to SF shortly

rappatic 2mo ago

I don’t have any kind of ML background but I have always thought of sample efficiency as the great unsolved problem of AI. We humans have unbelievably good sample efficiency; often we can durably learn something on just a single example or two. This is the main area in which LLMs are vastly, vastly behind us.

xdavidliu 2mo ago
the caveat is that we are not learning those small number of samples from scratch, since we're coming in with a large amount of training already, much of it from before we were even born
- nomel 2mo ago
  I've seen a few videos about people having their girlfriend/dad/etc play very simple video games for the very first time. It's a really great example of this. The most basic, simple, mechanics are difficult for them, and take surprising repetition to remember, because they have zero framework around any of it. It's really similar to watching a genetic algorithm learn to play Mario, but on fast forward.
10xDev 2mo ago
No, the unsolved problem of AI is continuous learning. We never stop learning, we don't have a "training phase". You are always updating your world model even when you sleep. Also more quality training data does lead to greater learning efficiency as you have more priors to work with.
- suddenlybananas 2mo ago
  Why not both?
- matteoraso 2mo ago
  >We never stop learning, we don't have a "training phase". We kind of do, if you count critical periods in childhood.

foota 2mo ago

> I agree that its rare to see to face problem sets in real life where every problem is given at once. Even if it is (like an exam), humans can usually only attempt one at a time Just one small snippet that I thought was interesting. I would always read through ~the entire exam before starting. Both so that I could find the problems most approachable to me, but also because sometimes it helps me figure out the rest of the questions :-)

evilmathkid 2mo ago
same! I'd often learn during the exam by solving an easier problem and then that would let me tackle a hard problem

foota 2mo ago

> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is possible Guarantees no synthetic data can be used It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created) I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?

nicman23 2mo ago
how would you as a human know the answer to the arc-agi questions?
- foota 2mo ago
  I spent years learning logic and doing puzzles. I don't think a baby or even average kid could solve these.
  - anon291 2mo ago
    Predicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions. > I don't think a baby or even average kid could solve these The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).
evilmathkid 2mo ago
train from scratch only during the 12 hours allowed on Kaggle Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun
- foota 2mo ago
  [dead]

TZubiri 2mo ago

>Training on the eval puzzles is cheating / “training on test” No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on. I disagree, but i agree that training with answers is worse. In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those questions. Sometimes it was the same exam sometimes they were 'slightly' different. In no occasions were the answers shared, and being math exams it wouldn't have made a difference, since solving the exercises is the actual training that allows replication of results and adaptation in testing. It's a grey area for sure, but it's a quantitative matter, studying 20 different exams for 2 months is quite different than trying out 1 exam 1 week prior to an exam to verify all is well. It's called teaching to the test, not teaching to the test and answers. In essence OP holds a naive version of what cheating is, and thus they think they are absolved, when actual cheating is much more nuanced. Many such cases. Fwiw, the second uni I dropped out of was worse in that some students just used their phone during tests and talked with each other or googled. OP sounds like a student from Uni 1 claiming they don't do what Uni 2 students do. And for reference, the exams I did pass I did by just reading the whole bibliography on my own, and doing exercises from the book if needed, I was passing with like 80-90%, never did I have to get a copy of a previous exam and study that, I think it's a ridiculous concept that has been normalized to meet an increasing societal pressure on everyone being an elite graduate (we can't ALL be elite), and if this repo is successful, it's because this attitude is so normalized that it's seeping into machine learning by diffusing the lines between training and testing set, and increasing the ratio between one and the other. Hell, I'm not surprised that the software that the mass of test -studiers develop is software that studies tests. In the same manner that the software that cheaters develop is software that cheats Guardrails and breaks ToSes

evilmathkid 2mo ago
> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam" What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs). (There's also access to experts who know the answer, which kaggle bans by banning the internet) Lucas describes it well here (and his original tweet up the thread): https://x.com/giffmana/status/2002128356901597509 https://x.com/giffmana/status/2002128356901597509 -- Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2

deleted 2mo ago

[deleted]

porridgeraisin 2mo ago

@mods I am not sure the editorialized title is better! (It is a bit clickbaity, since this is a transformer that does not compete with LLMs at all outside the arc-agi-1 benchmark)

ACV001 2mo ago

Nice. It took me an hour just to understand what this competition is about :)

RoadieRoller 2mo ago

I know that this is Transformer, and not LLM per se. But isn't this the same idea that PaulG said the other day, and many of the comments criticized him? https://news.ycombinator.com/item?id=49412396 https://news.ycombinator.com/item?id=49412396

evilmathkid 2mo ago
Yeah but I posted this work back in Dec-Feb lol
- RoadieRoller 2mo ago
  My above comment wasn't aimed at you. Thanks for the response though! What you did is phenomenal, and I wasn't taking a dig at you in any way. What I intended in my comment was that, may be starting from scratch (like you did many months before, and what PaulG suggested recently) is the way to go for future job prospects and startups. Most of the comments in the thread I posted was negative for PaulG saying that.
  - evilmathkid 2mo ago
    Yeah agree with pg completely. Multiple people I know did this

AmazingEveryDay 2mo ago

So this achieves 44%, I'm wondering: To consider yourself a reasonably competent human, what % to achieve on ARC-AGI-1? Thanks in advance to anyone that knows, can answer!

5555watch 2mo ago

It means the questions including their answers are dependent. Ie, theres a data generating process for them, that the model uncovers. Like a KNN is known to have near Bayes accuracy as k/n to 0, n to infty, k to infty. The data reveals the dgp. I suspect if you feed unrelated or even garbage questions into the eval set, it would reduce the performance.

usernametaken29 2mo ago

I think you’re asking the right questions, sample inefficiency is horrible in modern LLMs. Despite this, I saw your analysis: > The biggest increases in scores were due to Modern architecture (SwiGlu instead of GELU, RMSnorm not layernorm, etc.) More data diversity, better shuffling of data scaling up: 8 layers instead of 4 This is commonly called squeezing the lemon and is usually a bit of a last resort. You should be able to achieve near SoTa with your new method, before you squeeze any lemons. This is, because the old SoTa is typically not using new optimisers and thus your results will be distorted by a large margin. In terms of sample efficiency I want to add two things: Runtime per-puzzle fine tuning is a very good target that provides a LOT of information. People have not looked at evolutionary methods to harness induction since the 90ies - if I was to work on ARC ever again I’m fairly certain this is where I’d look. Best of luck, padawan

juggle73 2mo ago

[flagged]