3 ms·
Ask HN: Build Your Own LLM?
The best way to really understand how something works is to build it yourself. So I am wondering if there are any good tutorials on building your own LLM from scratch. I.e. implementing tokenisation, embeddings, attention and so on. I am not suggesting one could replicate chatGPT, but more a toy model that implements the core features but based on a much smaller corpus and training data.
- 2ro 1y agoHow about this? https://mathstodon.xyz/@empty/115088095028020763 https://mathstodon.xyz/@empty/115088095028020763
- retube 1y agothanks
- pm2222 1y agohttps://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167 https://www.amazon.com/Build-Large-Language-Model-Scratch/dp...
- retube 1y agothanks. looks potential
- ryanchants 1y agoI'd get it straight from Manning and save a few bucks and take out the middle man: https://www.manning.com/books/build-a-large-language-model-from-scratch https://www.manning.com/books/build-a-large-language-model-f...
- sfmz 1y agoAndrej Karpathy: Let's build GPT: from scratch, in code, spelled out. https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY
- beardyw 1y agoAndrej Karpathy's Nano GPT is reasonably accessible and easy to run. https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT
- runjake 1y agoSince you're posting here, you're looking for the shortcut. The shortcut is Karpathy's "Let's Build GPT: from scratch, in code, spelled out" video: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY Then there is a good video that dives into LLMs and how they work that is quite approachable: https://www.youtube.com/watch?v=7xTGNNLPyMI https://www.youtube.com/watch?v=7xTGNNLPyMI From there, flesh out knowledge with his other videos, where he goes both extremely light and extremely deep: https://www.youtube.com/@AndrejKarpathy/videos https://www.youtube.com/@AndrejKarpathy/videos Anyway, I really like's Karpathy's video because he's very good at explaining LLMs at every level.
- khamidou 1y agoSorry to self-promote but I did exactly that a few months back: https://khamidou.com/gpt2/ https://khamidou.com/gpt2/ Generally, I think the Karpathy tutorials are a good starting point but they're very mathy (despite people telling you you only need high school math to understand llms, a lot of the abstractions and concepts he uses are a bit foreign to programmers). I found out rebuilding inference of a known model taught me a lot more than passively sitting through the videos and maybe retyping his code. You should try it with something simple, like a model from a few years back!
- liqilin1567 1y agoThere is a new repo of karpathy: https://github.com/karpathy/nanochat https://github.com/karpathy/nanochat. It's a full-stack implementation of an LLM like ChatGPT in a single, clean, minimal, hackable, dependency-lite codebase.