5 ms·

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer tra

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs - Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples. - Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient) - Can the perf be improved? Yes but the competition is ongoing so can't talk about it - Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho - Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML

evilhackerdude 1mo ago
nothing like legendary shrugging and keeping the mind open
kuczmama 1mo ago
Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?
- evilmathkid 1mo ago
  Yeah I've reached much higher perf but - it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me - more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)
asabla 1mo ago
Thank you for answering these questions. Looking forward for the next write up about this.
pj_mukh 1mo ago
>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?
- dnautics 1mo ago
  It's an slm
  - bkaae 1mo ago
    There is no language in the training of this, so there is no l.
    - _ache_ 1mo ago
      In computer science, that is technically a language. A formal language if you want to look it up on Wikipedia.
      - FridgeSeal 1mo ago
        Technically correct, but not in a functionally useful way. The “L” in LLM’s generally refers to human-language specifically. You’d expect to feed it…human text. Nitpicking that the human text also constitutes a mathematical language is like, correct, but so general as to be unhelpful.
- dfdydx 1mo ago
  Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
  - perfmode 1mo ago
    In this case, what are the tokens?
    - evilmathkid 1mo ago
gandreani 1mo ago
Fyi, the link to rhabdomyolysis is broken on the homepage! The URL is repeated
lai7th_0 1mo ago
[dead]
bbor 1mo ago
First: this is really great technical writing, especially when you get into the rebuttals. Firm & clear without polemics -- props, and thanks for open-sourcing! That said; I don't have the time, energy, or anywhere near the expertise to challenge you on the DL specifics, but I feel compelled to add another voice to the chorus of doubters nonetheless. Using other ARC examples at runtime (effectively, yes?) for "transduction" may not violate what some officer behind ARC said on Twitter --and is certainly a fantastic tool for certain problem spaces-- but it just seems like a glaring and unavoidable philosophical problem in this one. My issue isn't with using the eval set per-se (though that obviously sets off well-justified alarm bells), but rather building an AGI system whose performance relies on the arbitrary size and shape of this particular dataset. There's a lot of ways to frame this, but given the transduction citations the most appropriate is probably the AI winter's infamous 'Frame Problem': You say upfront that this works in the first place because ARC has "very few samples... in a high dimensional space"; to me, that seems like an extremely strong indicator that the datasets are not intended to capture anywhere near the full semantic space that we would consider relevant for AGI. If true, your approach would indeed be ""cheating"" by using an arbitrary & unavoidable feature of the dataset (that they didn't have the time or money to craft 100 million high quality cases by hand instead of 1000) to solve the frame problem upfront for you. This would explain why you don't even need a full LLM here -- that's the unsolvable problem that LLMs solve for us. That is... even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason. In plainer English: surely you'd agree that your model would do worse if we swapped it out with Opus behind the scenes than the next-highest-scoring ARC model would do in the same position, yes? For coding, research, dumb questions, SVG pelicans -- the lot? If so, that seems like hard proof that this scores high on a benchmark at the cost of the benchmark itself. Like, if this transductive approach leads to ARC1 being claimed (which I thought it was ages ago but :shrug:), they'll either have to abandon the whole benchmark or ban this approach retroactively. If not... well, I guess I encourage you to try it! It seems like you'd need 1000 truly stellar hand-picked examples to transductively cover that whole space, for one thing.
- sigbottle 1mo ago
  Kind of hijacking, would you say that LLM's have solved the frame problem? To me, the frame problem is: Can you function in an open vs closed world, and to me the answer is yes, LLM's can definitely function in an open world where the rules are fuzzy, changing, undefined, etc. At the very least, much better than all GOFAI approaches by far. The issue is now grounding - It can "function", but what would it take to "ground" them? A personality, maybe? Actual consequences? Making them interact only with constrained tools that are formally verified? Right now it's a combination of harness engineering, and ml philosophers arguing about compression leading to the "objectively correct intelligence", whatever that means. I think LLM's are "A[x]I" right now in the sense of "they have the capability to integrate with everything" - but obviously you can argue how much this actually reflects "A[x]I" (if you gave someone integration with everything, is that really your success or people handing you it)? But they are still missing some oomph factors that need to be clarified IMO. Maybe it's something as "mundane" as just having actual persistent memory, or maybe it's some deep philosophical thing like qualia. Who knows.
deleted 1mo ago
[deleted]
_superposition_ 1mo ago
I've been interested in training a transformer from scratch for the same learning reasons. The GPU cost/availability seemed prohibitive to do anything useful but you seem to have flipped that on its head. I love your outside the box approach.
nullbio 1mo ago
"- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho" Also, in curating the training data in a deliberate manner, with attention to detail. Most people just use existing datasets and call it a day. It's a lot of work, which is why there are gains on the table.
TZubiri 1mo ago
>Personally I think today's frontier models can be beat by training from scratch You mean for a specific usecase? Also aren't frontier models trained 'from scratch'?
- evilmathkid 1mo ago
  > You mean for a specific usecase? Yepp > Also aren't frontier models trained 'from scratch'? The full sentence was supposed to be "training from scratch only on ARC data" the point was you dont need large scale pretraining
tasuki 1mo ago
As an aside, the anagram is so good!
- evilmathkid 1mo ago
  haha thanks!
alibero 1mo ago
Thank you for this excellent post series. It reminds me a lot of the pre-LLM days, though I was mostly using LSTMs back then. When the original GPT paper came out, I thought the future would be using LLMs to generate tons of synthetic labeled data and then training specialized LSTM or transformer models per-task. Had a couple of questions: 1) You note that ARC-AGI is a meta-learning task, have you tried any meta-learning algorithms such as MAML? 2) Do you think this approach could extend to ARC-AGI 3? Or do you think the interactive environments require a higher level of complexity than what can be achieved with a small model?
- evilmathkid 1mo ago
  Glad to know you like it! 1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time! 2) Possibly, but it would require significant changes and effort. But much larger models would be required imo (must have capacity greater than the complexity of the problem)
- bunderbunder 1mo ago
  I spent some time working with that approach of using LLMs to generate synthetic labeled data for use in training more specialized models. It mostly didn't work. The problem was that getting the LLM to generate training data that sufficiently resembled real-world data was labor intensive and expensive. More labor intensive and expensive, it turns out, than just using real data. What worked better was using the LLM to label the training data. But even there we had to be careful about introducing weird biases.
ozim 1mo ago
*Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho* On specific tasks sure on "generic performance" whatever that means for you not really.
- evilmathkid 1mo ago
  As of today yes I agree with you in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)
deleted 1mo ago
[deleted]

evilhackerdude 1mo ago

nothing like legendary shrugging and keeping the mind open

kuczmama 1mo ago

Hey thanks for sharing this. Was curious did you find the more you trained the model the more perf improved, or did it start plateauing. For example, let's say you didn't spend 67 cents, but you spent 67 dollars do you think you would get major benefits from that?

evilmathkid 1mo ago
Yeah I've reached much higher perf but - it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me - more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)

asabla 1mo ago

Thank you for answering these questions. Looking forward for the next write up about this.

pj_mukh 1mo ago

>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?

dnautics 1mo ago
It's an slm
- bkaae 1mo ago
  There is no language in the training of this, so there is no l.
  - _ache_ 1mo ago
    In computer science, that is technically a language. A formal language if you want to look it up on Wikipedia.
    - FridgeSeal 1mo ago
      Technically correct, but not in a functionally useful way. The “L” in LLM’s generally refers to human-language specifically. You’d expect to feed it…human text. Nitpicking that the human text also constitutes a mathematical language is like, correct, but so general as to be unhelpful.
dfdydx 1mo ago
Not all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
- perfmode 1mo ago
  In this case, what are the tokens?
  - evilmathkid 1mo ago

gandreani 1mo ago

Fyi, the link to rhabdomyolysis is broken on the homepage! The URL is repeated

lai7th_0 1mo ago

[dead]

bbor 1mo ago

First: this is really great technical writing, especially when you get into the rebuttals. Firm & clear without polemics -- props, and thanks for open-sourcing! That said; I don't have the time, energy, or anywhere near the expertise to challenge you on the DL specifics, but I feel compelled to add another voice to the chorus of doubters nonetheless. Using other ARC examples at runtime (effectively, yes?) for "transduction" may not violate what some officer behind ARC said on Twitter --and is certainly a fantastic tool for certain problem spaces-- but it just seems like a glaring and unavoidable philosophical problem in this one. My issue isn't with using the eval set per-se (though that obviously sets off well-justified alarm bells), but rather building an AGI system whose performance relies on the arbitrary size and shape of this particular dataset. There's a lot of ways to frame this, but given the transduction citations the most appropriate is probably the AI winter's infamous 'Frame Problem': You say upfront that this works in the first place because ARC has "very few samples... in a high dimensional space"; to me, that seems like an extremely strong indicator that the datasets are not intended to capture anywhere near the full semantic space that we would consider relevant for AGI. If true, your approach would indeed be ""cheating"" by using an arbitrary & unavoidable feature of the dataset (that they didn't have the time or money to craft 100 million high quality cases by hand instead of 1000) to solve the frame problem upfront for you. This would explain why you don't even need a full LLM here -- that's the unsolvable problem that LLMs solve for us. That is... even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason. In plainer English: surely you'd agree that your model would do worse if we swapped it out with Opus behind the scenes than the next-highest-scoring ARC model would do in the same position, yes? For coding, research, dumb questions, SVG pelicans -- the lot? If so, that seems like hard proof that this scores high on a benchmark at the cost of the benchmark itself. Like, if this transductive approach leads to ARC1 being claimed (which I thought it was ages ago but :shrug:), they'll either have to abandon the whole benchmark or ban this approach retroactively. If not... well, I guess I encourage you to try it! It seems like you'd need 1000 truly stellar hand-picked examples to transductively cover that whole space, for one thing.

sigbottle 1mo ago
Kind of hijacking, would you say that LLM's have solved the frame problem? To me, the frame problem is: Can you function in an open vs closed world, and to me the answer is yes, LLM's can definitely function in an open world where the rules are fuzzy, changing, undefined, etc. At the very least, much better than all GOFAI approaches by far. The issue is now grounding - It can "function", but what would it take to "ground" them? A personality, maybe? Actual consequences? Making them interact only with constrained tools that are formally verified? Right now it's a combination of harness engineering, and ml philosophers arguing about compression leading to the "objectively correct intelligence", whatever that means. I think LLM's are "A[x]I" right now in the sense of "they have the capability to integrate with everything" - but obviously you can argue how much this actually reflects "A[x]I" (if you gave someone integration with everything, is that really your success or people handing you it)? But they are still missing some oomph factors that need to be clarified IMO. Maybe it's something as "mundane" as just having actual persistent memory, or maybe it's some deep philosophical thing like qualia. Who knows.

deleted 1mo ago

[deleted]

_superposition_ 1mo ago

I've been interested in training a transformer from scratch for the same learning reasons. The GPU cost/availability seemed prohibitive to do anything useful but you seem to have flipped that on its head. I love your outside the box approach.

nullbio 1mo ago

"- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho" Also, in curating the training data in a deliberate manner, with attention to detail. Most people just use existing datasets and call it a day. It's a lot of work, which is why there are gains on the table.

TZubiri 1mo ago

>Personally I think today's frontier models can be beat by training from scratch You mean for a specific usecase? Also aren't frontier models trained 'from scratch'?

evilmathkid 1mo ago
> You mean for a specific usecase? Yepp > Also aren't frontier models trained 'from scratch'? The full sentence was supposed to be "training from scratch only on ARC data" the point was you dont need large scale pretraining

tasuki 1mo ago

As an aside, the anagram is so good!

evilmathkid 1mo ago
haha thanks!

alibero 1mo ago

Thank you for this excellent post series. It reminds me a lot of the pre-LLM days, though I was mostly using LSTMs back then. When the original GPT paper came out, I thought the future would be using LLMs to generate tons of synthetic labeled data and then training specialized LSTM or transformer models per-task. Had a couple of questions: 1) You note that ARC-AGI is a meta-learning task, have you tried any meta-learning algorithms such as MAML? 2) Do you think this approach could extend to ARC-AGI 3? Or do you think the interactive environments require a higher level of complexity than what can be achieved with a small model?

evilmathkid 1mo ago
Glad to know you like it! 1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time! 2) Possibly, but it would require significant changes and effort. But much larger models would be required imo (must have capacity greater than the complexity of the problem)
bunderbunder 1mo ago
I spent some time working with that approach of using LLMs to generate synthetic labeled data for use in training more specialized models. It mostly didn't work. The problem was that getting the LLM to generate training data that sufficiently resembled real-world data was labor intensive and expensive. More labor intensive and expensive, it turns out, than just using real data. What worked better was using the LLM to label the training data. But even there we had to be careful about introducing weird biases.

ozim 1mo ago

*Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho* On specific tasks sure on "generic performance" whatever that means for you not really.

evilmathkid 1mo ago
As of today yes I agree with you in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)

deleted 1mo ago

[deleted]