5 ms·
NanoGPT: The simplest, fastest repository for training medium-sized GPTs
- asicsp 2y agoPrevious discussion: https://news.ycombinator.com/item?id=34336386 https://news.ycombinator.com/item?id=34336386 (1532 points | Jan 11, 2023 | 320 comments)
- imurray 2y agoSee also (from the same author) https://github.com/karpathy/llm.c https://github.com/karpathy/llm.c — "LLMs in simple, pure C/CUDA with no need for 245MB of PyTorch or 107MB of cPython."
- szundi 2y agoThis guy is awesome
- paradite 2y agoIt takes a bit of effort to setup and train a proper GPT-2 model, especially if you are not familiar with GPU drivers and python environment. Also don't try to train GPT-2 on your own machine as it takes days, even with a good gaming GPU. If you are interested in trying it out, but don't have the right GPU or OS, you can check out the guide I wrote on how I did it on Azure with T4 GPU instance: https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guide.html https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guid...
- alok-g 2y agoThanks a lot for sharing this. How much does the net cost come out to for a run? (Am not a startup.) Also, any smaller datasets you would recommend that still demonstrate some useful capability? Thanks.
- paradite 2y agoUpdate: added result section to my post with more details: https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guide.html#results https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guid... I don't think it can go any smaller and demonstrate its capabilities. Even GPT-2 is mostly generating nonsensical sentences that resembles English. You might be looking at ngram or markov model for simpler NLP capabilities.
- alok-g 2y agoThanks!
- mhavelka77 2y agoThe fact that Andrej Karpathy is part of OpenAI is giving me at least a bit of hope that the company is not going to ruin society as we know it…
- captainregex 2y agohe…isn’t?
- ninjin 2y agoIndeed, I think he left (for the second time) early this year.
- animex 2y agoHe left in Feb 2024. https://x.com/karpathy/status/1757600075281547344?t=gpYK73N5finFRoOsdPuzkw&s=19 https://x.com/karpathy/status/1757600075281547344?t=gpYK73N5...
- VagabundoP 2y agoWould any GPU - even an old one - work for training these models? I have a bunch of little home apps/ideas for some specific training I'd like to do, but don't currently have any recent GPU of note. Just a few old ones in my attic somewhere. EDIT: ah someone has posted a blog post in the comments with some info EDIT2: to be clearer I was talking about finetuning. I assume thats quicker/less intensive. I really need to checkout some online courses I think.
- mromanuk 2y agoYes, fine tuning is fast, cheap and "easy", you can do it in a no so expensive GPU without issues.
- CapsAdmin 2y agoSomething from previous discussions about the whole "Rather than paying $50k up front for 8 x A100's, you can just rent some GPU's for $1.2k to train the whole thing in 4 days" That feel off to me is that it completely ignores the compute time spent exploring new ideas, failing, tweaking the training data, etc.
- mikeqq2024 2y agoLink for the previous discussion? Which model, dataset, training strategy?
- deleted 2y ago[deleted]
- mromanuk 2y agoHave in mind that instead of training a GPT-2 model from scratch, you can opt for fine-tuning. Fine-tuning allows you to take a pre-trained model and adapt it to your specific task with much less computational resources and time. It leverages the existing knowledge of the model, resulting in better performance and faster development cycles.
- serverlord 2y agoWe need a simpler way to make GPT get mass adoption and I don't think GPT Store from OpenAI will do that. We need a unique interface with a standardized process to help people make use-case-specific GPTs. Let's see what the future holds.
- paradite 2y agoI actually built a purpose-made UI (desktop app) for coding using ChatGPT, because I found that chat interface is not ideal for daily coding tasks. Curious what you think about it: https://prompt.16x.engineer/ https://prompt.16x.engineer/
- fragmede 2y agoSo like aider, but with more gui.
- paradite 2y agoExactly.
- serverlord 2y agoThis looks super interesting. Thanks for sharing! I will share my thoughts in a couple of days.