5 ms·

How does this compare to the karpathy video [0]? I'm trying to get into LLMs and am trying to figure out what the best resource to get that level of understandi

How does this compare to the karpathy video [0]? I'm trying to get into LLMs and am trying to figure out what the best resource to get that level of understanding would be. [0] https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY

_giorgio_ 3y ago
You can't understand it unless you already know most of the stuff. I've watched it many times to understand well most of it. And obviously you must already know pytorch really well, including the matrix multiplication, backpropagation etc. He speaks very fast too...
- tayo42 3y ago
  He has like 4 or 5 videos that can be watched before that one where all of that is covered. He goes over stuff like writing back prop from scratch and implementing layers without torch.
  - _giorgio_ 3y ago
    I know... That material isn't for beginners.
    - mikeiavelli 3y ago
      ...but then, what material did you expect as a beginner?
      - _giorgio_ 3y ago
        How can Karpathy videos defined for beginners when you have to know: programming, python, pytorch, matrix multiplication, derivatives...
- hadjian 3y ago
  Did you really watch all videos in the playlist? I am at video 4 and had no background in PyTorch or numpy. In my opinion he covers everything needed to understand his lectures. Even broadcasting and multidimensional indexing with numpy. Also in the first lecture you will implement your own python class for building expressions including backprop with an API modeled after PyTorch. IMHO it is the second lecture I can recommend without hesitation. The other is Gilbert Strang on linear algebra.
  - _giorgio_ 3y ago
    I was talking about the last video. It's difficult unless you don't know most of the material, or if you havent watched the other videos in the series. Anyway those videos are quite advanced. Surely not for beginners.
  - fbdab103 3y ago
    To echo this sentiment, I thought he does a really reasonable job of working up to the topic. Sure, it is fast paced, but it is a video you can rewind, plus play with the notebooks. There is a lot to learn, but I think he touches on all of the highlights which would give the viewer the tools to have a better understanding if they want to explore the topic in depth. Plus, I like that the videos are not overly polished. He occasionally makes a minor blunder, which really humanizes the experience.
rasbt 3y ago
Haven't fully watched this but from a brief skimming, here are some differences that the book has: - it implements a real word-level LLM instead of a character-level LLM - after pretraining also shows how to load pretrained weights - instruction-finetune that LLM after pretraining - code the alignment process for the instruction-finetuned LLM - also show how to finetune the LLM for classification tasks - the book it overall has a lots of figures. For Chapter 3, there are 26 figures alone :) The video looks awesome though. I think it's probably a great complementary resource to get a good solid intro because it's just 2 hours. I think reading the book will probably be more like 10 times that time investment.
- malermeister 3y ago
  Thank you for the answer! What is the knowledge that your book requires? If I have a lot of software dev experience and sorta kinda remember algebra from uni, would it be a good fit?
  - rasbt 3y ago
    Good question. I think a Python background is strongly recommended. PyTorch knowledge would be a nice to have (although I've written a comprehensive 40 page intro for the Appendix, which is also already available). From a math perspective, I think it should be gentle. I'm introducing dot products in Chapter 3, but I also explain how you could do the same with for-loops. Same with matrix multiplication. I'm bad at estimating requirements, but I hope this should be sufficient.

_giorgio_ 3y ago

You can't understand it unless you already know most of the stuff. I've watched it many times to understand well most of it. And obviously you must already know pytorch really well, including the matrix multiplication, backpropagation etc. He speaks very fast too...

tayo42 3y ago
He has like 4 or 5 videos that can be watched before that one where all of that is covered. He goes over stuff like writing back prop from scratch and implementing layers without torch.
- _giorgio_ 3y ago
  I know... That material isn't for beginners.
  - mikeiavelli 3y ago
    ...but then, what material did you expect as a beginner?
    - _giorgio_ 3y ago
      How can Karpathy videos defined for beginners when you have to know: programming, python, pytorch, matrix multiplication, derivatives...
hadjian 3y ago
Did you really watch all videos in the playlist? I am at video 4 and had no background in PyTorch or numpy. In my opinion he covers everything needed to understand his lectures. Even broadcasting and multidimensional indexing with numpy. Also in the first lecture you will implement your own python class for building expressions including backprop with an API modeled after PyTorch. IMHO it is the second lecture I can recommend without hesitation. The other is Gilbert Strang on linear algebra.
- _giorgio_ 3y ago
  I was talking about the last video. It's difficult unless you don't know most of the material, or if you havent watched the other videos in the series. Anyway those videos are quite advanced. Surely not for beginners.
- fbdab103 3y ago
  To echo this sentiment, I thought he does a really reasonable job of working up to the topic. Sure, it is fast paced, but it is a video you can rewind, plus play with the notebooks. There is a lot to learn, but I think he touches on all of the highlights which would give the viewer the tools to have a better understanding if they want to explore the topic in depth. Plus, I like that the videos are not overly polished. He occasionally makes a minor blunder, which really humanizes the experience.

rasbt 3y ago

Haven't fully watched this but from a brief skimming, here are some differences that the book has: - it implements a real word-level LLM instead of a character-level LLM - after pretraining also shows how to load pretrained weights - instruction-finetune that LLM after pretraining - code the alignment process for the instruction-finetuned LLM - also show how to finetune the LLM for classification tasks - the book it overall has a lots of figures. For Chapter 3, there are 26 figures alone :) The video looks awesome though. I think it's probably a great complementary resource to get a good solid intro because it's just 2 hours. I think reading the book will probably be more like 10 times that time investment.

malermeister 3y ago
Thank you for the answer! What is the knowledge that your book requires? If I have a lot of software dev experience and sorta kinda remember algebra from uni, would it be a good fit?
- rasbt 3y ago
  Good question. I think a Python background is strongly recommended. PyTorch knowledge would be a nice to have (although I've written a comprehensive 40 page intro for the Appendix, which is also already available). From a math perspective, I think it should be gentle. I'm introducing dot products in Chapter 3, but I also explain how you could do the same with for-loops. Same with matrix multiplication. I'm bad at estimating requirements, but I hope this should be sufficient.