9 ms·
Building LLMs from the Ground Up: A 3-Hour Coding Workshop
- leopoldj 2y agoThis is the exact level of details I was looking for. I'm fairly experienced with deep learning and pytorch and don't want to see them built from scratch. I found Andrej's materials too low level and I tend to get lost in the weeds. This is not a criticism but just a comment for someone in a similar situation as I'm.
- abusaidm 2y agoNice write up Sebastian, looking forward to the book. There are lots of details on the LLM and how it’s composed, would be great if you can expand on how Llama and OpenAI could be cleaning and structuring their training data given it seems this is where the battle is heading in the long run.
- rakahn 2y agoYes. Would love to read that.
- rahimnathwani 2y agohow Llama and OpenAI could be cleaning and structuring their training data If you're interested in this, there are several sections in the Llama paper you will likely enjoy: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/ https://ai.meta.com/research/publications/the-llama-3-herd-o...
- kbrkbr 2y agoBut isn't it the beauty of llm's that they need comparably little preparation (unstructured text as input) and pick the features on their own so to say? edit: grammar
- aDyslecticCrow 2y agoYes, if you want an LLM that doesn't listen to instructions and just endlessly babbles about anything and everything. What turned GPT into chatGPT was a lot of structured training with human feedback.
- rahimnathwani 2y agoExactly. Section 4.3.7 briefly explains how they trained the model to better follow instructions ('steerability').
- atum47 2y agoExcuse my ignorance, is this different from Andrej Karpathy https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY Anyway I will watch it tonight before bed. Thank you for sharing.
- BaculumMeumEst 2y agoAndrej's series is excellent, Sebastian's book + this video are excellent. There's a lot of overlap but they go into more detail on different topics or focus on different things. Andrej's entire series is absolutely worth watching, his upcoming Eureka Labs stuff is looking extremely good too. Sebastian's blog and book are definitely worth the time and money IMO.
- bilsbie 2y agoWhat are the main differences?
- samstave 2y ago[flagged]
- BaculumMeumEst 2y agofair point bud
- brcmthrowaway 2y agowhat book
- StefanBatory 2y agoMost likely this one. https://www.manning.com/books/build-a-large-language-model-from-scratch?utm_source=raschka&utm_medium=affiliate&utm_campaign=book_raschka_build_12_12_23&a_aid=raschka&a_bid=4c2437a0&chan=mm_website https://www.manning.com/books/build-a-large-language-model-f... (I've taken it from the footnotes on the article)
- eclectic29 2y agoThis is excellent. Thanks for sharing. It's always good to go back to the fundamentals. There's another resource that is also quite good: https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/
- _giorgio_ 2y agoNot true. Your resource is really bad. "We'll then load the trained GPT-2 model weights released by OpenAI into our implementation and generate some text."
- skinner_ 2y ago> Your resource is really bad. What a bad take. That resource is awesome. Sure, it is about inference, not training, but why is that a bad thing?
- szundi 2y agoThis is not “building from the ground up”
- abustamam 2y agoWhy is that bad?
- deleted 2y ago[deleted]
- skinner_ 2y agoNeither the author of the GPT from scratch post, nor eclectic29 who recommended it above did ever promise that the post is about building LLMs from the ground up. That was the original post. The GPT from scratch post explains, from the ground up, ground being numpy, what calculations take place inside a GPT model.
- 2y ago
- TAE_WKingDaki 2y ago[flagged]
- adultSwim 2y agoThis page is just a container for a youtube video. I suggest updating this HN link to point to the video directly, which contains the same links as the page in its description.
- yebyen 2y agoWhy not support the author's own website? It looks like a nice website
- _giorgio_ 2y agoHe shares a ton of videos and code. His material is really valuable. Just support him?
- mdp2021 2y agoOn the contrary, I saved you that extra step of looking for Sebastian Raschka's repository of writings.
- bschmidt1 2y agoLove stuff like this. Tangentially I'm working on useful language models without taking the LLM approach: Next-token prediction: https://github.com/bennyschmidt/next-token-prediction https://github.com/bennyschmidt/next-token-prediction Good for auto-complete, spellcheck, etc. AI chatbot: https://github.com/bennyschmidt/llimo https://github.com/bennyschmidt/llimo Good for domain-specific conversational chat with instant responses that doesn't hallucinate.
- p1esk 2y agoWhy do you call your language model “transformer”?
- bschmidt1 2y agoLanguage is the language model that extends Transformer. Transformer is a base model for any kind of token (words, pixels, etc.). However, currently there is some language-specific stuff in Transformer that should be moved to Language :) I'm focusing first on language models, and getting into image generation next.
- p1esk 2y agoNo, I mean, a transformer is a very specific model architecture, and your simple language model has nothing to do with that architecture. Unless I’m missing something.
- richrichie 2y agoFor a century, transformer meant a very different thing. Power systems people are justifiably amused.
- p1esk 2y agoAnd it means something else in Hollywood. But we are discussing language models here, aren’t we?
- karmakaze 2y agoThis is great. Just yesterday I was wondering how exactly transformers/attention and LLMs work. I'd worked through how back-propagation works in a deep RNN a long while ago and thought it would be interesting to see the rest.
- aDyslecticCrow 2y agoFor some intuition, 3b1b has some videos that explain it nicely. But he doesn't go that far into the nitty gritty.
- ein0p 2y agoI’m not sure why you’d want to build an LLM these days - you won’t be able to train it anyway. It’d make a lot of sense to teach people how to build stuff with LLMs, not LLMs themselves.
- ckok 2y agoThis has been said about pretty much every subject. Writing your own Browsers, compilers, cryptography, etc. But at least for me even if nothing comes of it just knowing how it really works, What steps are involved are part of using things properly. Some people are perfectly happy using a black box, but without kowning how its made, how do we know the limits? How will the next generation of llms happen if nobody can get excited about the internal workings?
- ein0p 2y agoYou don’t need to write your own LLM to know how it works. And unlike, say, a browser it doesn’t really do anything even remotely impressive unless you have at least a few tens of thousands of dollars to spend on training. Source: my day job is to do precisely what I’m telling you not to bother doing, but I do have access to a large pool of GPUs. If I didn’t, I’d be doing what I suggest above.
- richrichie 2y agoGood points. For learning purpose, just understanding what a neural network is and how it works covers it all.
- BaculumMeumEst 2y agoBut I mean people can always rent GPUs too. And they're getting pretty ubiquitous as we ramp up from the AI hype craze, I am just an IT monkey at the moment and even I have on-demand access to a server with something like 4x192GB GPUs at work.
- ein0p 2y ago
- alok-g 2y agoThis is great! Hope it works on a Windows 11 machine too (I often find that when Windows isn't explicitly mentioned, the code isn't tested on it and usually fails to work due to random issues).
- sidkshatriya 2y agoWhen it does not work on Windows 11 -- what about trying it out on WSL (Windows Subsystem for Linux ) ?
- politelemon 2y agoThis should work perfectly fine in WSL2 as it has access to a GPU. Do remember to install the Cuda toolkit, NVidia has one for WSL2 specifically. https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local https://developer.nvidia.com/cuda-downloads?target_os=Linux&...
- 1zael 2y agoSebastian, you are a god among mortals. Thank you.
- paradite 2y agoI wrote a practical guide on how to train nanoGPT from scratch on Azure a while ago. It's pretty hands-on and easy to follow: https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guide.html https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guid...
- firesteelrain 2y agoDid it really only cost $200? What sort of things could you do with it? How do you train it on current events?
- paradite 2y agoYes. I checked the Azure usage after training. Beyond learning how it all works and demo, there is not much practical usage. You can train it on current events if you feed that corpus during training instead of just OpenWebText. Shouldn't be hard.
- alecco 2y agoUsing PyTorch is not "LLMs from the ground up". It's a fine PyTorch tutorial but let's not pretend it's something low level.
- chuckSu 2y ago[dead]
- menzoic 2y agoIs this a joke? Can’t tell. OpenAI uses PyTorch to build LLMs
- jnhl 2y agoYou could always go deeper and from some points of view, it's not "from the ground up" enough unless you build your own autograd and tensors from plain numpy arrays.
- 0cf8612b2e1e 2y agoNumpy sounds like cheating on the backs of others. Going to need your own hand crafted linear algebra routines.
- TZubiri 2y agoSource please?
- leobg 2y agoPeople think of the Karpathy tutorials which do indeed build LLMs from the ground up, starting with Python dictionaries.
- krmboya 2y agoFrom scratch is relative. To a python programmer, from scratch may mean starting with dictionaries but a non-programmer will have to learn what python dicts are first. To someone who already knows excel, from scratch with excel sheets instead of python may work with them.
- theanonymousone 2y agoIt may be unreasonable, but I have a default negativity toward anything that uses the word "coding" instead of programming or development.
- xanderlewis 2y agoProbably now an unpopular view (as is any opinion perceived as 'judgemental' or 'gatekeeping'), but I agree.
- smartmic 2y agoI fully agree. We had a discussion about this one year ago: https://news.ycombinator.com/item?id=36924239 https://news.ycombinator.com/item?id=36924239
- ljlolel 2y agoThis is more a European thing
- atoav 2y agoI am from Europe and I am not completely sure about that to be honest. I also prefer programming. I also dislike software development as it reminds me of developing a photograhic negative – like "oh let's check out how the software we developed came out". It should be software engineering and it should be held to a similar standard as other engineering fields if it isn't done in a non-professional context.
- mdp2021 2y ago> software development Wrong angle. There is a problem, your consideration of the problem, the refinement of your solution to the problem: the solution gradually unfolds - it is developed.
- reichstein 2y agoThe word "development" can mean several things. I don't think "software development" sounds bad when grouped with a phrase like "urban development". It describes growing and tuning software for, well, working better, solving more needs, and with fewer failure modes. I do agree that a "coder" creates code, and a programmer creates programs. I expect more of a complete program than of a bunch of code. If a text says "coder", it does set an expectation about the professionalism of the text. And I expect even more from a software solution created by a software engineer. At least a specification! Still, I, a professional software engineer and programmer, also write "code" for throwaway scripts, or just for myself, or that never gets completed. Or for fun. I will read articles by and for coders too. The word is a signal. It's neither good nor bad, but If that's not the signal the author wants to send, they should work on their communication.
- activesciencep 2y ago[flagged]
- cpill 2y agoyeah really valuable stuff. so we know how the ginormous model that we can't train or host works (putting practice there are so many hacks and optimizations that none of them work like this). great.