9 ms·
Microgpt
- tithos 8mo agoWhat is the prime use case
- aaronblohowiak 8mo ago“Art project”
- pixelatedindex 8mo agoIf writing is art, then I’ve been amazed at the source code written by this legend
- geerlingguy 8mo agoLooks like to learn how a GPT operates, with a real example.
- foodevl 8mo agoYeah, everyone learns differently, but for me this is a perfect way to better understand how GPTs work.
- antonvs 8mo agoTo confuse people who only think in terms of use cases. Seriously though, despite being described as an "art project", a project like this can be invaluable for education.
- bourjwahwah 8mo agoEven art serves a use case of giving the artist a means of expression. Use case does not need to be technical.
- hrmtst93837 8mo ago[flagged]
- keyle 8mo agoit's a great learning tool and it shows it can be done concisely.
- jackblemming 8mo agoCase study to whenever a new copy of Programming Pearls is released.
- deleted 8mo ago[deleted]
- inerte 8mo agoKaparthy to tell you things you thought were hard in fact fit in a screen.
- colonCapitalDee 8mo agoBeautiful work
- ViktorRay 8mo agoWhich license is being used for this?
- fulafel 8mo agoThis could make an interesting language shootout benchmark.
- hrmtst93837 8mo ago[flagged]
- profsummergig 8mo agoIf anyone knows of a way to use this code on a consumer grade laptop to train on a small corpus (in less than a week), and then demonstrate inference (hallucinations are okay), please share how.
- simsla 8mo agoThe blog post literally explains how to do so.
- hrmtst93837 8mo ago[flagged]
- hrmtst93837 8mo ago[flagged]
- profsummergig 8mo agoThis. I literally am asking for a step-by-step guide outlining every step (including an existing corpus that can be used on a consumer-grade laptop to train the model in under a week).
- hrmtst93837 8mo ago[flagged]
- ThrowawayTestr 8mo agoThis is like those websites that implement an entire retro console in the browser.
- Paddyz 8mo ago[flagged]
- misiti3780 8mo agoagreed - no one else is saying this.
- janis1234 8mo agoI found reading Linux source more useful than learning about xv6 because I run Linux and reading through source felt immediately useful. I.e, tracing exactly how a real process I work with everyday gets created. Can you explain this O(n2) vs O(n) significance better?
- Paddyz 8mo ago[dead]
- fc417fc802 8mo ago> Each step is O(n) instead of recomputing everything, and total work across all steps drops to O(n^2) In terms of computation isn't each step O(1) in the cached case, with the entire thing being O(n)? As opposed to the previous O(n) and O(n^2).
- wasabi991011 8mo agoI still don't quite get your insight. Maybe it would help me better if you could explain it while talking like a pirate?
- fc417fc802 8mo agoIt's weird because while the second comment felt like slop to me due to the reasoning pattern being expressed (not really sure how to describe it, it's like how an automaton that doesn't think might attempt to model a person thinking) skimming the account I don't immediately get the same vibe from the other comments. Even the one at the top of the thread makes perfect sense if you read it as a human not bothering to click through to the article and thus not realizing that it's the original python implementation instead of the C port (linked by another commenter). Perhaps I'm finally starting to fail as a turing test proctor.
- lynxbot2026 8mo ago[flagged]
- sdwr 8mo agoIf you know your exact use case, have prior work to build on, think deeply and extensively about the problem domain, and don't need competitive results, you can save a lot of lines of code!
- ZeWaka 8mo agoWhy write many word when few do trick?
- GuB-42 8mo agoThe answer is in the article: "Everything else is just efficiency" Another example is a raytracer. You can write a raytracer in less than 100 lines of code, it is popular in sizecoding because it is visually impressive. So why are commercial 3D engines so complex? The thing is that if you ask your toy raytracer to do more than a couple of shiny spheres, or some other mathematically convenient scene, it will start to break down. Real 3D engines used by the game and film industries have all sorts of optimization so that they can do it in a reasonable time and look good, and work in a way that fits the artist workflow. This is where the million of lines come from.
- wasabi991011 8mo agoSpecifically, why do you think the parent comment mentioned 1000 lines of C?
- awwaiid 8mo agoWhere is this 1000 lines of C coming from? This is python.
- pnexk 8mo agoFirsthand evidence of AI bot accounts posting slop on here/astroturfing I suppose.
- dhruv3006 8mo agoKarapthy with another gem !
- charcircuit 8mo ago[flagged]
- rramadass 8mo agoC++ version - https://github.com/Charbel199/microgpt.cpp?tab=readme-ov-file https://github.com/Charbel199/microgpt.cpp?tab=readme-ov-fil... Rust version - https://github.com/mplekh/rust-microgpt https://github.com/mplekh/rust-microgpt
- almaight 8mo agohttps://github.com/novvoo/gogpt https://github.com/novvoo/gogpt
- deleted 8mo ago[deleted]
- jimbokun 8mo agoIt’s pretty staggering that a core algorithm simple enough to be expressed in 200 lines of Python can apparently be scaled up to achieve AGI. Yes with some extra tricks and tweaks. But the core ideas are all here.
- wasabi991011 8mo ago1000 lines?? What is going on in this thread
- ViktorRay 8mo agoIt’s pretty sad. The only way we know these comments are from AI bots for now is due to the obvious hallucinations. What happens when the AI improves even more…will HN be filled with bots talking to other bots?
- the_af 8mo agoWhat's bizarre is this particular account is from 2007. Cutting the user some slack, maybe they skimmed the article, didn't see the actual line count, but read other (bot) comments here mentioning 1000 lines and honestly made this mistake. You know what, I want to believe that's the case.
- birole 8mo agoWhy would anyone runs bots on this website? What is the benefit for them? Is someone happens to know about it?
- tomrod 8mo agoMaintaining or injecting commentary to guide towards targeted outcomes. Guerrilla marketing of a sort.
- ashdksnndck 8mo agoIt already is in some threads. Sometimes you get the bots writing back and forth really long diatribes at inhuman frequency. Sometimes even anti-LLM content!
- red_hare 8mo agoThis is beautiful and highly readable but, still, I yearn for a detailed line-by-line explainer like the backbone.js source: https://backbonejs.org/docs/backbone.html https://backbonejs.org/docs/backbone.html
- altcognito 8mo agoask a high end LLM to do it
- ashish01 8mo agoThat is really beautiful literate program. Seeing it after a long time. Here is a opus generate version of this code - https://ashish01.github.io/microgpt.html https://ashish01.github.io/microgpt.html
- subset 8mo agoAndrej Karpathy has a walkthrough blog post here: https://karpathy.github.io/2026/02/12/microgpt/ https://karpathy.github.io/2026/02/12/microgpt/
- tomjakubowski 8mo agoI believe that Backbone's annotated source is generated with Docco, another project from the creator of CoffeeScript. https://ashkenas.com/docco/ https://ashkenas.com/docco/ It's really neat. I wish I published more of my code this way.
- subset 8mo agoI had good fun transliterating it to Rust as a learning experience (https://github.com/stochastical/microgpt-rs https://github.com/stochastical/microgpt-rs). The trickiest part was working out how to represent the autograd graph data structure with Rust types. I'm finalising some small tweaks to make it run in the browser via WebAssmebly and then compile it up for my blog :) Andrej's code is really quite poetic, I love how much it packs into such a concise program
- hei-lima 8mo agoGreat work! Might do it too in some other language...
- pmarreck 8mo agoZig, here. Anything but Python
- O5vYtytb 8mo agoAt least python can do this exercise without pulling 3rd party dependencies :)
- justinhj 8mo agoWhat's missing from Zig and its std lib for this?
- moderation 8mo agoZig version [0] doesn't need any external dependencies. 0. https://tangled.org/m17e.co/microgpt https://tangled.org/m17e.co/microgpt
- pmarreck 8mo agoSweet! Presumably this one is even faster than the JS?
- kelvinjps10 8mo agoWhy there is multiple comments talking about 1000 c lines, bots?
- the_af 8mo agoOr even 1000 python lines, also wrong. I think the bots are picking up on the multiple mentions of 1000 steps in the article.
- thatxliner 8mo agobtw my friend is asking if your username is a "Klara and the Sun" reference
- 0xbadcafebee 8mo agoSince this post is about art, I'll embed here my favorite LLM art: the IOCCC 2024 prize winner in bot talk, from Adrian Cable (https://www.ioccc.org/2024/cable1/index.html https://www.ioccc.org/2024/cable1/index.html), minus the stdlib headers: #define a(_)typedef _##t #define _(_)_##printf #define x f(i, #define N f(k, #define u _Pragma("omp parallel for")f(h, #define f(u,n)for(I u=0;u<(n);u++) #define g(u,s)x s%11%5)N s/6&33)k[u[i]]=(t){(C*)A,A+s*D/4},A+=1088*s; a(int8_)C;a(in)I;a(floa)F;a(struc){C*c;F*f;}t;enum{Z=32,W=64,E=2*W,D=Z*E,H=86*E,V='}\0'};C*P[V],X[H],Y[D],y[H];a(F _)[V];I*_=U" 炾ોİ䃃璱ᝓ၎瓓甧染ɐఛ瓁",U,s,p,f,R,z,$,B[D],open();F*A,*G[2],*T,w,b,c;a()Q[D];_t r,L,J,O[Z],l,a,K,v,k;Q m,e[4],d[3],n;I j(I e,F*o,I p,F*v,t*X){w=1e-5;x c=e^V?D:0)w+=r[i]*r[i]/D;x c)o[i]=r[i]/sqrt(w)*i[A+e*D];N $){x W)l[k]=w=fmax(fabs(o[i])/~-E,i?w:0);x W)y[i+k*W]=*o++/w;}u p)x $){I _=0,t=h*$+i;N W)_+=X->c[t*W+k]*y[i*W+k];v[h]= _*X->f[t]*l[i]+!!i*v[h];}x D-c)i[r]+=v[i];}I main(){A=mmap(0,8e9,1,2,f=open(M,f),0);x 2)~f?i[G]=malloc(3e9):exit( puts(M" not found"));x V)i[P]=(C*)A+4,A+=(I)*A;g(&m,V)g(&n,V)g(e,D)g(d,H)for(C*o;;s>=D?$=s=0:p<U||_()("%s",$[P]))if(! (*_?$=*++_:0)){if($<3&&p>=U)for(_()("\n\n> "),0<scanf("%[^\n]%*c",Y)?U=*B=1:exit(0),p=_(s)(o=X,"[INST] %s%s [/INST]",s? "":"<<SYS>>\n"S"\n<</SYS>>\n\n",Y);z=p-=z;U++[o+=z,B]=f)for(f=0;!f;z-=!f)for(f=V;--f&&f[P][z]|memcmp(f[P],o,z););p<U? $=B[p++]:fflush(0);x D)R=$*D+i,r[i]=m->c[R]*m->f[R/W];R=s++;N Z){f=k*D*D,$=W;x 3)j(k,L,D,i?G[~-i]+f+R*D:v,e[i]+k);N 2)x D)b=sin(w=R/exp(i%E/14.)),c=1[w=cos(w),T=i+++(k?v:*G+f+R*D)],T[1]=b**T+c*w,*T=w**T-c*b;u Z){F*T=O[h],w=0;I A=h*E;x s){N E)i[k[L+A]=0,T]+=k[v+A]*k[i*D+*G+A+f]/11;w+=T[i]=exp(T[i]);}x s)N E)k[L+A]+=(T[i]/=k?1:w)*k[i*D+G[1]+A+f];}j(V,L ,D,J,e[3]+k);x 2)j(k+Z,L,H,i?K:a,d[i]+k);x H)a[i]*=K[i]/(exp(-a[i])+1);j(V,a,D,L,d[$=H/$,2]+k);}w=j($=W,r,V,k,n);x V)w=k[i]>w?k[$=i]:w;}}
- thatxliner 8mo agowiat what does this do?
- aix1 8mo agoAs the contest entry page explains: > ChatIOCCC is the world’s smallest LLM (large language model) inference engine - a “generative AI chatbot” in plain-speak. ChatIOCCC runs a modern open-source model (Meta’s LLaMA 2 with 7 billion parameters) and has a good knowledge of the world, can understand and speak multiple languages, write code, and many other things. Aside from the model weights, it has no external dependencies and will run on any 64-bit platform with enough RAM. (Model weights need to be downloaded using an enclosed shell script.) https://www.ioccc.org/2024/cable1/index.html https://www.ioccc.org/2024/cable1/index.html
- coolThingsFirst 8mo agoIncredibly fascinating. One thing is that it seems still very conceptual. What id be curious about how good of a micro llm we can train say with 12 hours of training on macbook.
- hackersk 8mo ago[flagged]
- lukan 8mo ago"The math makes so much more sense when you implement it yourself vs reading papers." Something I found to be universal true when dealing with math. My brain pretty much refuses to learn abstract math concepts in theory, but applying them with a practical problem is a very different experience for me (I wish school math would have had a bigger focus on practical applications).
- Sammi 8mo agoIt's like you learn math best with your hands. The mind catches up to your hands afterwards.
- byang364 8mo agoImagine the people on here spraying their AI takes everywhere while being this oblivious, the code is more or less a standard assignment in all Deep Learning courses. The "reasoning" is two matrix transformations based on how often words appear next to each other.
- harvey9 8mo agoQuite a few people on here are neither math nor CS grads and some of us don't work in tech for our day jobs either.
- 0xEF 8mo agoRight. But HN, among other platforms, is full of users who will confidently run their mouths about something they don't fully understand while believing they do. I think the previous commenter was being too shy in pointing out that even exceptionally smart people sometimes forget where the limits of their own knowledge are, not to mention consider themselves immune to any propaganda that surrounds the subject at hand.
- deleted 8mo ago[deleted]
- freakynit 8mo agoIs there something similar for diffusion models? By the way, this is incredibly useful for learning in depth the core of LLM's.
- verma7 8mo agoI wrote a C++ translation of it: https://github.com/verma7/microgpt/blob/main/microgpt.cc https://github.com/verma7/microgpt/blob/main/microgpt.cc 2x the number of lines of code (~400L), 10x the speed The hard part was figuring out how to represent the Value class in C++ (ended up using shared_ptrs).
- WithinReason 8mo agoI made an explicit reverse pass (no autodiff), it was 8x faster in Python
- love2read 8mo agoCan you share a link?
- WithinReason 8mo agohttps://www.ideone.com/VAz4Nn https://www.ideone.com/VAz4Nn Doesn't run inside IDEone due to the external download link, but you can copy&paste the code over
- hu3 8mo agoI made an explicit double-reverse pass (no code!), it was 80x faster in my head!
- spopejoy 8mo ago"I've got an ipod -- In My Mind" https://theonion.com/i-have-an-ipod-in-my-mind-1819584018/ https://theonion.com/i-have-an-ipod-in-my-mind-1819584018/
- WithinReason 8mo agocode here, it's just not interesting to look at: https://news.ycombinator.com/item?id=47220542 https://news.ycombinator.com/item?id=47220542
- 8mo ago
- MattyRad 8mo agoHoenikker had been experimenting with melting and re-freezing ice-nine in the kitchen of his Cape Cod home. Beautiful, perhaps like ice-nine is beautiful.
- teleforce 8mo agoSomeone has modified microgpt to build a tiny GPT that generates Korean first names, and created a web page that visualizes the entire process [1]. Users can interactively explore the microgpt pipeline end to end, from tokenization until inference. [1] English GPT lab: https://ko-microgpt.vercel.app/ https://ko-microgpt.vercel.app/
- camkego 8mo agoI have no affiliation with the website, but the website is pretty neat if you are learning LLM internals. It explains: Tokenization, Embedding, Attention, Loss & Gradient, Training, Inference and comparison to "Real GPT" Pretty nifty. Even if you are not interested in the Korean language
- love2read 8mo agoBy "modified" this person of course means that they swapped out the list of X0,000 names from English to Korean names. That is seemingly the only change. The attached website is a fully ai-generated "visualization" based on the original blog post with little added.
- big-chungus4 8mo agoIt's a good website and probably AI generated with some insane expensive model that us mere mortals are too poor to afford, thus it has a value
- sprobertson 8mo agoThis kind of thing is pretty easy to do with a much leaner model https://docs.pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html https://docs.pytorch.org/tutorials/intermediate/char_rnn_gen...
- arijun 8mo agoI assume the goal isn't to generate Korean names but to learn GPTs.
- znnajdla 8mo agoSuper useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a specialized harness for one task only: writing code for that framework (perhaps in tandem with a commodity frontier model). Could a single programmer or a small team on a household budget afford to train a model that works better/faster than OpenAI/Anthropic/DeepSeek for specialized tasks? My gut tells me this is possible; and I have a feeling that this will become mainstream, and then custom model training becomes the new “software development”.
- the_arun 8mo agoIf we can run them on commodity hardware with cpus, nothing like it
- teleforce 8mo agoThis is possible but not for training but fine-tuning the existing open source models. This can be mainstream, and then custom model fine-tuning becomes the new “software development”. Please check out this new fine-tuning method for LLM by MIT and ETH Zurich teams that used a single NVIDIA H200 GPU [1], [2], [3]. Full fine-tuning of the entire model’s parameters were performed based on the Hugging Face TRL library. [1] MIT's new fine-tuning method lets LLMs learn new skills without losing old ones (news): https://venturebeat.com/orchestration/mits-new-fine-tuning-method-lets-llms-learn-new-skills-without-losing-old https://venturebeat.com/orchestration/mits-new-fine-tuning-m... [2] Self-Distillation Enables Continual Learning (paper): https://arxiv.org/abs/2601.19897 https://arxiv.org/abs/2601.19897 [3] Self-Distillation Enables Continual Learning (code): https://self-distillation.github.io/SDFT.html https://self-distillation.github.io/SDFT.html
- jeremyjh 8mo agoFine tuning does not make a model any smaller. It can make a smaller model more effective at a specific task, but a larger model with the same architecture fine-tuned on the same dataset will always be more capable in a domain as general as programming or software design. Of course, as architecture and related tooling improves the smallest model that is "good enough" will continue to get smaller.
- growingswe 8mo agoGreat stuff! I wrote an interactive blogpost that walks through the code and visualizes it: https://growingswe.com/blog/microgpt https://growingswe.com/blog/microgpt
- spinningslate 8mo agoThat’s beautifully done, thanks for posting. As helpful again to an ML novice like me as Karpathy’s original.
- hei-lima 8mo agoGreat!
- evntdrvn 8mo agoreally nice, thanks
- evntdrvn 8mo agoYou should totally submit that to HN as an article, if you haven't already.
- dang 8mo agoWe've put https://news.ycombinator.com/item?id=47205208 https://news.ycombinator.com/item?id=47205208 in the second-chance pool (https://news.ycombinator.com/pool https://news.ycombinator.com/pool, explained at https://news.ycombinator.com/item?id=26998308 https://news.ycombinator.com/item?id=26998308), so it will get a random placement on HN's front page.
- joenot443 8mo agoThis is awesome! Normally I'm pretty critical of LLM-assisted-blogging, but this one's a real winner.
- O4epegb 8mo ago> By the end of training, the model produces names like "kamon", "karai", "anna", and "anton". None of them are copies from the dataset. All 4 are in the dataset, btw
- with 8mo ago"everything else is just efficiency" is a nice line but the efficiency is the hard part. the core of a search engine is also trivial, rank documents by relevance. google's moat was making it work at scale. same applies here.
- lukan 8mo agoSure, but understanding the core concepts are essential to make things efficient and as far as I understand, this has mainly educational purposes ( it does not even run on a GPU).
- with 8mo agoyep, agreed. wasn’t knocking the project at all, it’s great for exactly that purpose
- geon 8mo agoI think the hard part is improving on the basic concept. The current top of the line models are extremely overfitted and produce so much nonsense they are useless for anything but the most simple tasks. This architecture was an interesting experiment, but is not the future.
- abhitriloki 8mo ago[flagged]
- xuki 8mo agoHuman internet is dead. I don't know how we can come back from this.
- growingswe 8mo agoIt's tragic. So many bots!
- dang 8mo agoIt's going to take a while for HN (the community, the mods, and the software systems) to adapt. Hopefully we can find a new equilibrium, but there is going to be quite some turbulence for a while. In the meantime, it's super helpful for people to let us know at hn@ycombinator.com when they see accounts like these which are posting nothing but what appear to be generated comments, so we can ban them. Edit: (perhaps I shouldn't bury the lede): Generated comments aren't allowed on HN - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20%22generated%20comments%22&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que.... They never have been, and of course this rule is becoming more relevant these days.
- kuberwastaken 8mo agoI'm half shocked this wasn't on HN before? Haha I built PicoGPT as a minified fork with <35 lines of JS and another in python And it's small enough to run from a QR code :) https://kuber.studio/picogpt/ https://kuber.studio/picogpt/ You can quite literally train a micro LLM from your phone's browser
- cootsnuck 8mo agoIt was: https://news.ycombinator.com/item?id=47000263 https://news.ycombinator.com/item?id=47000263
- iberator 8mo ago[flagged]
- kuberwastaken 8mo agolol there is source code as a gist
- lelandfe 8mo agohttps://github.com/Kuberwastaken/picogpt/blob/main/picogpt.js https://github.com/Kuberwastaken/picogpt/blob/main/picogpt.j...
- dang 8mo agoPlease don't be a jerk on HN, and especially not when responding to someone's work. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html.
- dang 8mo agoWow I agree - surprising that it took 2 weeks to make HN's frontpage. We do generally like HN to be a bit uncorrelated with the rest of the internet, but it feels like a miss to me that neither https://news.ycombinator.com/item?id=47000263 https://news.ycombinator.com/item?id=47000263 nor https://news.ycombinator.com/item?id=47018557 https://news.ycombinator.com/item?id=47018557 made the frontpage.
- ruszki 8mo ago> [p for mat in state_dict.values() for row in mat for p in row] I'm so happy without seeing Python list comprehensions nowadays. I don't know why they couldn't go with something like this: [state_dict.values() for mat for row for p] or in more difficult cases [state_dict.values() for mat to mat*2 for row for p to p/2] I know, I know, different times, but still.
- deleted 8mo ago[deleted]
- WithinReason 8mo agoI would have gone for: [for p in row in mat in state_dict.values()]
- ruszki 8mo agoThat’s also an option. The left to right flow is better for the sake of autocomplete and comprehension: when you start to read your right to left version, you don’t know what is p, then row, then mat. With left to right, this problem doesn’t exist. One for sure, both are superior to the garbled mess of Python’s. Of course if the programming language would be in a right to left natural language, then these are reversed.
- shevy-java 8mo agoMicroslop is alive!
- genie3io 8mo ago[dead]
- retube 8mo agoCan you train this on say Wikipedia and have it generate semi-sensible responses?
- OJFord 8mo agoIf you increase all the numbers (including, as a result, the time to train).
- geon 8mo agoThat’s exactly what chatgpt etc are.
- krisoft 8mo agoNo. But there are a few layers to that. First no is that the model as is has too few parameters for that. You could train it on the wikipedia but it wouldn’t do much of any good. But what if you increase the number of parameters? Then you get to the second layer of “no”. The code as is is too naive to train a realistic size LLM for that task in realistic timeframes. As is it would be too slow. But what if you increase the number of parameters and improve the performance of the code? I would argue that would by that point not be “this” but something entirely different. But even then the answer is still no. If you run that new code with increased parameters and improved efficiencly and train it on wikipedia you would still not get a model which “generate semi-sensible responses”. For the simple reason that the code as is only does the pre-training. Without the RLHF step the model would not be “responding”. It would just be completing the document. So for example if you ask it “How long is a bus?” it wouldn’t know it is supposed to answer your question. What exactly happens is kinda up to randomness. It might output a wikipedia like text about transportation, or it might output a list of questions similar to yours, or it might output broken markup garbage. Quite simply without this finishing step the base model doesn’t know that it is supposed to answer your question and it is supposed to follow your instructions. That is why this last step is called “instruction tuning” sometimes. Because it teaches the model to follow instructions. But if you would increase the parameter count, improve the efficiency, train it on wikipedia, then do the instruction tuning (wich involves curating a database of instruction - response pairs) then yes. After that it would generate semi-sensible responses. But as you can see it would take quite a lot more work and would stretch the definition of “this”. It is a bit like asking if my car could compete in formula-1. The answer is yes, but first we need to replace all parts of it with different parts, and also add a few new parts. To the point where you might question if it is the same car at all.
- WithinReason 8mo agoPreviously: https://news.ycombinator.com/item?id=47000263 https://news.ycombinator.com/item?id=47000263
- geokon 8mo ago> What’s the deal with “hallucinations”? The model generates tokens by sampling from a probability distribution. It has no concept of truth, it only knows what sequences are statistically plausible given the training data. Extremely naiive question.. but could LLM output be tagged with some kind of confidence score? Like if I'm asking an LLM some question does it have an internal metric for how confident it is in its output? LLM outputs seem inherently rarely of the form "I'm not really sure, but maybe this XXX" - but I always felt this is baked in the model somehow
- DavidSJ 8mo agoYes, the actual LLM returns a probability distribution, which gets sampled to produce output tokens. [Edit: but to be clear, for a pretrained model this probability means "what's my estimate of the conditional probability of this token occurring in the pretraining dataset?", not "how likely is this statement to be true?" And for a post-trained model, the probability really has no simple interpretation other than "this is the probability that I will output this token in this situation".]
- podnami 8mo agoWhat happens before the probability distribution? I’m assuming say alignment or other factors would influence it?
- DavidSJ 8mo agoIn microgpt, there's no alignment. It's all pretraining (learning to predict the next token). But for production systems, models go through post-training, often with some sort of reinforcement learning which modifies the model so that it produces a different probability distribution over output tokens. But the model "shape" and computation graph itself doesn't change as a result of post-training. All that changes is the weights in the matrices.
- mr_toad 8mo agoIt’s often very difficult (intractable) to come up with a probability distribution of an estimator, even when the probability distribution of the data is known. Basically, you’d need a lot more computing power to come up with a distribution of the output of an LLM than to come up with a single answer.
- Jaxon_Varr 8mo ago[dead]
- OussamaAfnakkar 8mo ago[dead]
- la_fayette 8mo agoThis guy is so amazing! With his video and the code base I really have the feeling I understand gradient descent, back propagation, chain rule etc. Reading math only just confuses me, together with the code it makes it so clear! It feels like a lifetime achievement for me :-)
- mentos 8mo agoCurious if you could try to explain it. It’s my goal to sit down with it and attempt to understand it intuitively. Karpathy says if you want to truly understand something then you also have to attempt to teach it to someone else ha
- la_fayette 8mo agoYes, that’s true! That could be my next step… though I have to admit, writing this in a HN comment feels like a bit of a challenge.
- bytesandbits 8mo agosensei karpathy has done it again
- stuckkeys 8mo agoThat web interface that someone commented in your github was flawless.
- sieste 8mo agoThe typos are interesting ("vocavulary", "inmput") - One of the godfathers of LLMs clearly does not use an LLM to improve his writing, and he doesn't even bother to use a simple spell checker.
- meltyness 8mo agovocabulary* *In the code above, we collect all unique characters across the dataset
- shepherdjerred 8mo ago> Write me an AI blog post $ Sure, here's a blog post called "Microgpt"! > "add in a few spelling/grammar mistakes so they think I wrote it" $ Okay, made two errors for you!
- mold_aid 8mo ago"art" project?
- agenthustler 8mo ago[flagged]
- borplk 8mo agoCan anyone mention how you can "save the state" so it doesn't have to train from scratch on every run?
- geon 8mo agoIs there a similarly simple implementation with tensorflow? I tried building a tiny model last weekend, but it was very difficult to find any articles that weren’t broken ai slop.
- joefourier 8mo agoTensorflow is largely dead, it’s been years since I’ve seen a new repo use it. Go with Jax if you want a PyTorch alternative that can have better performance for certain scenarios.
- geon 8mo agoAny recommendations for Typescript?
- nickpsecurity 8mo agoAlso, TPU support. Hardware diversity.
- etothet 8mo agoEven if you have some basic understanding of how LLMs work, I highly recommend Karpathy’s intro to LLMs videos on YouTube. - https://m.youtube.com/watch?v=7xTGNNLPyMI https://m.youtube.com/watch?v=7xTGNNLPyMI - https://m.youtube.com/watch?v=EWvNQjAaOHw https://m.youtube.com/watch?v=EWvNQjAaOHw
- arvid-lind 8mo agothanks for the recommendations. it seems like i keep coming back to the basics of how i interact with LLMs and how they work to learn the new stuff. every time i think i understand, someone else explaining their approach usually makes me think again about how it all works. trying my best to keep up with what and how to learn and threads like this are dense with good info. feel like I need an AI helper to schedule time for my youtube queue at this point!
- grey-area 8mo agoThanks, this is very very long but very good background on how production LLMs work.
- raphaelmolly8 8mo ago[dead]
- vadimf 8mo agoI’m 100% sure the future consists of many models running on device. LLMs will be the mobile apps of the future (or a different architecture, but still intelligence).
- ajnin 8mo agoThe future right now looks more like everything in remote datacenters, no autonomous capabilities and no control by the user. But I like yours better.
- latchkey 8mo agoI don't mind the remote datacenters, I just don't like the lack of control.
- pizzafeelsright 8mo agoThis is the path forward, with some overhead. 1. Generic model that calls other highly specific, smaller, faster models. 2. Models loaded on demand, some black box and some open. 3. There will be a Rust model specifically for Rust (or whatever language) tasks. In about 5-8 years we will have personalized models based upon all our previous social/medical/financial data that will respond as we would, a clone, capable of making decisions similar with direction of desired outcomes. The big remaining blocker is that generic model that can be imprinted with specifics and rebuilt nightly. Excluding the training material but the decision making, recall, and evaluation model. I am curious if someone is working on that extracted portion that can be just a 'thinking' interface.
- coldtea 8mo agoIf anything, memory ain't getting cheaper, disks aren't either, and as for graphics cards, forget it. People wont be competing with even a current 2026 SOTA from their home LLM nowhere soon. Even actual SOTA LLM providers are not competing either - they're losing money on energy and costs, hopping to make it up on market capture and win the IPO races.
- OtherShrezzing 8mo ago
- astroanax 8mo agoI feel its wrong to call it microgpt, since its smaller than nanogpt, so maybe picogpt would have been a better name? nice project tho
- huqedato 8mo agoLooking for alternative in Julia.
- jonjacky 8mo agoI wonder if such a small GPT exhibits plagiarism. Are some of the generated names the same as names in the input data?
- hkbuilds 8mo ago[flagged]
- chenster 8mo agoThe best ML learning for dummies.
- smj-edison 8mo agoSomewhat unrelated, but the generated names are surprisingly good! They're certainly more sane then appending -eigh to make a unique name.
- hoppp 8mo agoWas that code generated by claude?
- lynx97 8mo agoQuestion: Can this be modified to score a "document"? I'd basically like to pass it a name, and get a score (0..1) on how realistic themodel "thinks" the document is? This would be extremely helpful for a project of mine.
- spopejoy 8mo agoSorry to RFELI5 but but ... I thought a "token" was a word? The example is of names and the output is new improvised names, implying that a character is a token? Or do all LLMs operate at character level? Also is there some minima of training data? E.g. if you just trained on "True" "False" I assume it would be .5 Bernoulli? What is the minimum to see "interesting" results I guess.
- easygenes 8mo agoInspiring. Definitely got nerd sniped by this. Now you can train it in under a second on one CPU core with no dependencies: https://github.com/Entrpi/eemicrogpt https://github.com/Entrpi/eemicrogpt Detailed optimizing journey in the readme too.
- hmcamp 8mo agoCool
- huambda 8mo ago[dead]
- meta-level 8mo agoIn case you intend to play around with microgpt: use Python 3.14! I found it to be 3x faster compared to 3.13 or 3.12..