4 ms·

I jumped to Github thinking this is would be a free resource (with all due respect to the author work). What free resources are available and recommended in th

by wslh 3y ago

I jumped to Github thinking this is would be a free resource (with all due respect to the author work). What free resources are available and recommended in the "from scratch vein"?

larme 3y ago
https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/ for a gpt2 inference engine in numpy then https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-kvcache/ https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-k... for adding a kv cache implementation
- larme 3y ago
  I'd like to add that most of these text only talking about inference part. This book (I also purchased the draft version) has training and finetuning in the TOC. I assume it will include materials about how to do training and finetuning from scratch.
rasbt 3y ago
I added notes to the Jupyter notebooks, I hope they are also readable as standalone from the repo.
natrys 3y ago
Neural Networks: Zero to Hero[1] by Andrej Karpathy [1] https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html
- villedespommes 3y ago
  +1, Andrey is an amazing educator! I'd also recommend his https://youtu.be/kCc8FmEb1nY?si=mP0cQlQ4rcceL2uP https://youtu.be/kCc8FmEb1nY?si=mP0cQlQ4rcceL2uP and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high.
PheonixPharts 3y ago
I honestly cannot fathom why anyone working in the AI space would find $50 too much to spend to gain a deeper insight into the subject. Creating educational materials requires an insane amount of work, and I can promise, no matter how successful this book is, if rasbt were do the math on income generated over hours spent creating it wouldn't make sense as an hourly rate. Plenty of other people have this understanding of these topics, and you know what they chose to do with that knowledge? Keep it to themselves and go work at OpenAI to make far more money keeping that knowledge private. If you want to live in a world where this knowledge is open, at the very least refrain from publicly complaining about a book that cost roughly the same as a decent dinner.
- rasbt 3y ago
  Yeah, I don't think creating educational materials makes sense from an economical perspective, but it's one of my hobbies that gives me joy for some reason :). Hah, and 'insane amount of work' is probably right -- lots of sacrifices to carve out that necessary time.
- wslh 3y ago
  Not talking about affordability but about following links thinking that I would find another kind of resource. Beyond this case, this happens all the time with click-baity content. Again, if the link was to Amazon or the editors it will be clear associated with a product while Github is associated with open source content. Not being pedantic, just an observation browsing the web.
  - wslh 3y ago
    I would add that I can find 5k of useful resources for USD 50 each. It is not the individual item but the all the context. Personally, I am not focused on a specific topic such as LLMs but work on an spectrum of topics more akind an analyst job + broad research skills.
- _giorgio_ 3y ago
  Not to be pedantic, but in this case it's probably 30 usd for print and ebook (there are always coupons on the manning website).
politelemon 3y ago
I'd go with https://course.fast.ai/ https://course.fast.ai/ It's much more accessible to regular developers, and doesn't make assumptions about any kind of mathematics background. It's a good starting poing after which other similar resources start to make more sense.

by wslh 3y ago

I jumped to Github thinking this is would be a free resource (with all due respect to the author work). What free resources are available and recommended in the "from scratch vein"?

larme 3y ago

https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/ for a gpt2 inference engine in numpy then https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-kvcache/ https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-k... for adding a kv cache implementation

larme 3y ago
I'd like to add that most of these text only talking about inference part. This book (I also purchased the draft version) has training and finetuning in the TOC. I assume it will include materials about how to do training and finetuning from scratch.

rasbt 3y ago

I added notes to the Jupyter notebooks, I hope they are also readable as standalone from the repo.

natrys 3y ago

Neural Networks: Zero to Hero[1] by Andrej Karpathy [1] https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html

villedespommes 3y ago
+1, Andrey is an amazing educator! I'd also recommend his https://youtu.be/kCc8FmEb1nY?si=mP0cQlQ4rcceL2uP https://youtu.be/kCc8FmEb1nY?si=mP0cQlQ4rcceL2uP and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high.

PheonixPharts 3y ago

I honestly cannot fathom why anyone working in the AI space would find $50 too much to spend to gain a deeper insight into the subject. Creating educational materials requires an insane amount of work, and I can promise, no matter how successful this book is, if rasbt were do the math on income generated over hours spent creating it wouldn't make sense as an hourly rate. Plenty of other people have this understanding of these topics, and you know what they chose to do with that knowledge? Keep it to themselves and go work at OpenAI to make far more money keeping that knowledge private. If you want to live in a world where this knowledge is open, at the very least refrain from publicly complaining about a book that cost roughly the same as a decent dinner.

rasbt 3y ago
Yeah, I don't think creating educational materials makes sense from an economical perspective, but it's one of my hobbies that gives me joy for some reason :). Hah, and 'insane amount of work' is probably right -- lots of sacrifices to carve out that necessary time.
wslh 3y ago
Not talking about affordability but about following links thinking that I would find another kind of resource. Beyond this case, this happens all the time with click-baity content. Again, if the link was to Amazon or the editors it will be clear associated with a product while Github is associated with open source content. Not being pedantic, just an observation browsing the web.
- wslh 3y ago
  I would add that I can find 5k of useful resources for USD 50 each. It is not the individual item but the all the context. Personally, I am not focused on a specific topic such as LLMs but work on an spectrum of topics more akind an analyst job + broad research skills.
_giorgio_ 3y ago
Not to be pedantic, but in this case it's probably 30 usd for print and ebook (there are always coupons on the manning website).

politelemon 3y ago

I'd go with https://course.fast.ai/ https://course.fast.ai/ It's much more accessible to regular developers, and doesn't make assumptions about any kind of mathematics background. It's a good starting poing after which other similar resources start to make more sense.