4 ms·
Modded-NanoGPT: NanoGPT (124M) quality in 3.25B tokens
- gavindean90 2y agoSeems like this is a modded NanoGPT not the original.
- munchler 2y agoYes. It’s literally called “Modded-NanoGPT”.
- Scene_Cast2 2y agoI wonder how much improvement is owed to which changes. I've also never heard of "Muon - Momentum Orthogonalized by Newton-schulz" being used. EDIT: there's a bit more info on his twitter - https://x.com/kellerjordan0 https://x.com/kellerjordan0 It looks like he created this optimizer. Works on 2D matrices only.
- whiplash451 2y agoCool work. No license?
- molticrystal 2y agoJust needs a Zero To Hero series episode offering line by line commentary to follow along on why each choice was made over alternatives.
- m3kw9 2y agoSo it compresses info better.
- pyinstallwoes 2y agoThat is literally intelligence.
- parineum 2y agoIt's not.
- nikonyrh 2y agoI suppose you aren't a fan of the https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize . > The goal of the Hutter Prize is to encourage research in artificial intelligence (AI). The organizers believe that text compression and AI are equivalent problems.
- parineum 2y agoI believe that they believe that and that it _could_ be true. That's far from declaratively stating that they are the same thing, as if there was some sort of evidence and consensus of such a claim.
- byyoung3 2y agodo you have a baseline of the regular implementation with 3x learning rate?