5 ms·

One sees from scratch, and then one also sees from fancy_module import magic_functions I'm semi-serious here, of course. To me, for something to be called '

One sees from scratch, and then one also sees from fancy_module import magic_functions I'm semi-serious here, of course. To me, for something to be called 'from scratch', requisite knowledge should be built ground up. To wit, I'd want to write the tokenizer myself but don't want to derive laws of quantum physics that makes the computation happen.

jatins 2y ago
what's the fancy_module you are talking about here?
- OnACoffeeBreak 2y ago
  PyTorch The book's code repo: https://github.com/rasbt/LLMs-from-scratch https://github.com/rasbt/LLMs-from-scratch
skipants 2y ago
I went through parts 1 and 8 and did not see an import statement in any of the posts
- gpjt 2y ago
  OP here: my posts are kind of reading notes for the book, so I don't normally copy the code from there -- so you wouldn't have seen the packages used. So far there's been tiktoken for tokenization (Raschka shows how to write a simple tokenizer and explains the workings of the byte-pair tokenization that he recommends, though) and PyTorch for CUDA-acceleratable matrix maths, automated differentiation for gradient descent, and so on.
shpongled 2y ago
I feel very similarly. When I want to understand from scratch, that means no pytorch, no numpy, etc.
slicktux 2y ago
I totally agree; I was contemplating giving the book a try…I still might but knowing myself I’m going to go down the rabbit hole of how those mathematical functions work before I can continue…
- gpjt 2y ago
  I have 100% set myself a "no side quests" rule while going through the book so that I don't do that. I've had... patchy success with that, but I think I'm doing pretty well apart from the week I spent getting LaTeX rendering working on my blog so that I could do the pretty maths in that post. What I'm doing is building up a list of things to dig into in depth once I've finished the book. Kind of like a treat to encourage me to push forward when I'm working through a bit that's tough to understand.
gpjt 2y ago
I do see your point! But at the end of the day, it depends on where you want to spend your time. "Build an LLM from scratch" is over 300 pages -- and they are very dense pages. My blog post covers fewer than 10 of them (though TBF they are the hardest pages). Adding on tokenizers in depth from scratch would add on 100 or so more. Adding on efficient-enough matrix multiplication to do anything would add on a few hundred more, and doing it in CUDA would probably be a couple of thousand. Now add on automated differentiation to work out the gradients for training -- a few thousand more? Optimizers for the training -- even more than that, perhaps. You have to draw the line somewhere, as otherwise (as you suggest) the "from scratch" book has to start "go out and get some really clean sand" so that you can start fabbing your own chips. I think that tiktoken and PyTorch are a solid choice for that line, as it means that the book is manageable in size and gives you enough of an overview of the underlying stuff to be able to work out what you want to dig into next.

jatins 2y ago

what's the fancy_module you are talking about here?

OnACoffeeBreak 2y ago
PyTorch The book's code repo: https://github.com/rasbt/LLMs-from-scratch https://github.com/rasbt/LLMs-from-scratch

skipants 2y ago

I went through parts 1 and 8 and did not see an import statement in any of the posts

gpjt 2y ago
OP here: my posts are kind of reading notes for the book, so I don't normally copy the code from there -- so you wouldn't have seen the packages used. So far there's been tiktoken for tokenization (Raschka shows how to write a simple tokenizer and explains the workings of the byte-pair tokenization that he recommends, though) and PyTorch for CUDA-acceleratable matrix maths, automated differentiation for gradient descent, and so on.

shpongled 2y ago

I feel very similarly. When I want to understand from scratch, that means no pytorch, no numpy, etc.

slicktux 2y ago

I totally agree; I was contemplating giving the book a try…I still might but knowing myself I’m going to go down the rabbit hole of how those mathematical functions work before I can continue…

gpjt 2y ago
I have 100% set myself a "no side quests" rule while going through the book so that I don't do that. I've had... patchy success with that, but I think I'm doing pretty well apart from the week I spent getting LaTeX rendering working on my blog so that I could do the pretty maths in that post. What I'm doing is building up a list of things to dig into in depth once I've finished the book. Kind of like a treat to encourage me to push forward when I'm working through a bit that's tough to understand.

gpjt 2y ago

I do see your point! But at the end of the day, it depends on where you want to spend your time. "Build an LLM from scratch" is over 300 pages -- and they are very dense pages. My blog post covers fewer than 10 of them (though TBF they are the hardest pages). Adding on tokenizers in depth from scratch would add on 100 or so more. Adding on efficient-enough matrix multiplication to do anything would add on a few hundred more, and doing it in CUDA would probably be a couple of thousand. Now add on automated differentiation to work out the gradients for training -- a few thousand more? Optimizers for the training -- even more than that, perhaps. You have to draw the line somewhere, as otherwise (as you suggest) the "from scratch" book has to start "go out and get some really clean sand" so that you can start fabbing your own chips. I think that tiktoken and PyTorch are a solid choice for that line, as it means that the book is manageable in size and gives you enough of an overview of the underlying stuff to be able to work out what you want to dig into next.