3 ms·
Aside from Karpathy's videos, I'd say: Start by any book on traditional neural networks, so you get a decent understanding of what neural networks are, what is
by kolinko 3y ago
Aside from Karpathy's videos, I'd say:
Start by any book on traditional neural networks, so you get a decent understanding of what neural networks are, what is an activation function and what the backpropagation is.
From then on, my path was:
- numpy user manual, if you don't know it already
- a book about pytorch
- a book about transformers
- a book from Wolfram about LLMs
(forgot the names of books, but you can find plenty on Amazon)
That's as far as reading / theoretical understanding works.
As for understanding in practice (when you've read the theory), I'd say - implement a basic picoGPT/Llama/Mistral model from scratch in python & numpy.
I don't know how good your college is in math, but matrix-vector multiplications, dot products and linear algebra in general is a must.
- gandutraveler 3y agoLet's say one builds a basic LLM in python. What's next? Aren't we limited by compute and GPU power to do anything further ?
- HPsquared 3y agoThat's the algorithm side, you also need training data. Can experiment with different data.
- FezzikTheGiant 3y agoIs this all the math that is required? If I know how to do basic linalg is that about what I need to know to be able to understand how LLMs work? My goal right now is to build stuff on top of LLMs