10 ms·

Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

1110 points by georgehill 4y ago

notjonheyman 4y ago
I had the please of working under Andrej a few years ago. It was very interesting the problems we were solving until this other guy that is a billionaire would throw a wrench (and fits) into the whole thing.
- andruby 4y ago
  I’d be interested to hear more. It was surprising to hear of Andrej leaving.
- deleted 4y ago
  [deleted]
- bestbuyer__ 4y ago
  You mean he got hired by Elon Musk to work at Tesla? :P
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
hambes 4y ago
I really hate the phrase "from scratch" in the title.
yonz 4y ago
Just finished this, need a part 2 for the PPO reward functionality
diego898 4y ago
Andrej’s entire series is by far one of the most useful resources I’ve found. Even as an instructor of these topics myself I learn something new about the material and about how to teach it!
tailfra 4y ago
Fantastic material! By the way, one of the simplest explanation of the difference between BetchNorm and LayerNorm
vorticalbox 4y ago
He has a website[0] for these videos [0] https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html
xt00 4y ago
I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training?
- karpathy 4y ago
  rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf https://github.com/facebookresearch/metaseq/blob/main/projec... 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.
  - mycall 4y ago
    GPUs are much less efficient than cores of the type Cerebras has made.
  - vvrm 4y ago
    Thanks for laying out the plan. I was trying to understand the cost of each of these steps below and started wondering about the following: > rough steps: > 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ? > 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. It seems like the standard instructGPT model itself is based on a 1 billion param GPT model. Wouldn't that fit on a 24GB RTX 3090 ? Might take longer, maybe not enough opportunity for hyper-parameter search, but still possible right ? Or is hyper-parameter search on a thousand machines in parallel the real magic sauce here ? > 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec https://github.com/facebookresearch/metaseq/blob/main/projec... Sounds like a good opportunity to learn. No pain, no gain :-) > 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 https://arxiv.org/abs/2211.01786 to get a sense of the work involved here. Maybe somebody would open source the equivalent datasets for this soon ? Otherwise the data collection seems prohibitively expensive for somebody trying to do this for fun: contract expert annotators, train them, annotate/reannotate for months ?
keepquestioning 4y ago
The char-rnn guy does it again.
pottspotts 4y ago
This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
- javajosh 4y ago
  Same, but one that doesn't assume python or an operating system.
  - tysam_and 4y ago
    Same, but I would honestly rather we get away from silicon or anything resembling what we call these "computers" these days. Things are too complicated and we need to truly get back to the basics: rocks on a hill.
    - bitL 4y ago
      Abacus-driven deep learning with trained monkeys flipping gradients.
      - qup 4y ago
        Outside tools are not required. We'll selectively breed humans until one can just compute it in...meat[0]. https://www.youtube.com/watch?v=7tScAyNaRdQ https://www.youtube.com/watch?v=7tScAyNaRdQ
        tysam_and 4y ago
        in mitro computing
      - captn3m0 4y ago
        You don't need to flip gradients, just need to train the monkeys to operate matchboxes: https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_Crosses_Engine https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_...
      - 4y ago
nlowell 4y ago
Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.
- mensetmanusman 4y ago
  Tesla kick started the global EV revolution, so probably favorably regardless of the few negatives that inherently come along the way.
- coldtea 4y ago
  As if history will care?
  - dd36 4y ago
    The history AI will have an opinion that mirrors the politics of the user that asks it what it thinks of Karpathy’s time at Tesla.
    - burntalmonds 4y ago
      Only in the default bubble mode.
- gorango 4y ago
  What do you mean by "however"? He built the entire FSD architecture - in less time and with fewer people than Google, Apple, and god knows who else. He's likely saved more lives than most doctors at this point...
  - jackblemming 4y ago
    No he didn't, a (large from what I hear) team did. However Karpathy is still a smart, well accomplished, guy and he has a talent many academics don't: being able to write actual production grade, high quality software. And he's a good communicator.
  - amelius 4y ago
davewasthere 4y ago
Now be meta and build GPT with GPT.
ultrasounder 4y ago
Just started watching and Andrej is an excellent "thingexplainer". His in-depth knowledge of the underlying atoms/bits comes through. As an extra benefit to watching his lecture using the https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html is the link to his "discord chat". This is a very active community and Andrej is very active there. So feel free to watch lectures, cry through the assignments and come to the discord with questions. And they will be answered.
albert_e 4y ago
I am a simple man. I see a video post by karpathy, I upvote and watch. <end reddit-speak> I discovered Andrej very recently and I am a huge fan. Kudos to this whole effort! Two ideas -- 1. While these explainers are outstanding -- I can think of supplementary material/presentation that can nicely complement these explanations if they are presented visually. Especially the concepts of multidimensional tensors. Something like what 3B1B (or his followers that create 'Summer of Math Exposition' videos) does -- which is not a skill I have. I am thinking of creating some visual slides (my forte) but would there be interest in making this a larger collaboration that creates explainers for "visual learners"? 2. There should really be a discussion forum for people who follow along these "make more" tutorials -- to have discussions about each specific video, infact each specific timestamped chapter of these videos in that context. Is there a framework or tool that lets us integrate YouTube videos and timestamped chapters into a "discussion forum" -- whether a simple website or a discord/slack. Once again this is slightly outside my skillset but if it appeals to people, maybe some ideas and effort can come together to make this happen? EDIT: #facepalm -- I see there is already a discord [0] on the webpage [1] (but not likely tied to very specific chapters as I imagined -- but should be a excellent start anyway) [0] - https://discord.gg/3zy8kqD9Cp https://discord.gg/3zy8kqD9Cp [1] - https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html
- indigochill 4y ago
  In case you're not aware, 3B1B has a Github repo for the engine he uses for the math animations so that others can use it to make similar things: https://github.com/3b1b/manim https://github.com/3b1b/manim There's also a loose group of people already doing the visual learners "explainers" thing over here: https://explorabl.es/ https://explorabl.es/ (you can scroll down for links to tools they use to make their explainers). But yes, I also feel this is an important development and that this should be an ongoing way of teaching people things. Formal education has IMO stalled out around the printing press but there are massive opportunities on computers (and especially on globally networked computers) to take that a step further and leverage the capabilities of computers to make education even more engaging and information-dense.
- deleted 4y ago
  [deleted]
cdzm 4y ago
Karpathy also has a great Recipe for Training Neural Networks: http://karpathy.github.io/2019/04/25/recipe/ http://karpathy.github.io/2019/04/25/recipe/
thecleaner 4y ago
His cs231n course is how I learnt about neutral networks. And now on to learning GPT and Transformers.
lvl102 4y ago
A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.
- bamboozled 4y ago
  How can we really stop this though? I feel like Microsoft is doing what Microsoft does yet again. They piggybacked on the whole "open source" angle but this time, rather than be "software vendor lock-in closed source assholes", they're now being "IP theft open source" assholes.
  - sharemywin 4y ago
    It's not just Microsoft. Google probably has models at least on par with OpenAI. Facebook probably too.
- fithisux 4y ago
  Totally agree. I also doing some Gcloud training. This should also be public good. When I move my machines to Google/AWS/ .. my business changes ownership and I become slave chained by the "competition" and "pricing". To fix this we need to change mentality and governments. But people are satisfied with their: "Mirror, Mirror on the Wall, Who's the Coolest of Them All? ... You, because you are indistinguishable from the others!!!!"
- pjmorris 4y ago
  > I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake. I don't know if these ideas are worse technically or politically, or both, but what comes to mind are these alternatives: 1. Have someone start a non-profit that curates public data goods, maybe gaining access to data through voluntary donations and through buying all the data provider feeds, and funding through subscriptions by people and organizations who want to understand the data provider infrastructure. 2. Get legislation passed that identifies public data goods and requires that they be made available to all.
abraxas 4y ago
For me he is the best educator in this space bar none. When Karpathy explains stuff it just clicks in my head.

1110 points by georgehill 4y ago

I had the please of working under Andrej a few years ago. It was very interesting the problems we were solving until this other guy that is a billionaire would throw a wrench (and fits) into the whole thing.

andruby 4y ago
I’d be interested to hear more. It was surprising to hear of Andrej leaving.
deleted 4y ago
[deleted]
bestbuyer__ 4y ago
You mean he got hired by Elon Musk to work at Tesla? :P

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

hambes 4y ago

I really hate the phrase "from scratch" in the title.

yonz 4y ago

Just finished this, need a part 2 for the PPO reward functionality

diego898 4y ago

Andrej’s entire series is by far one of the most useful resources I’ve found. Even as an instructor of these topics myself I learn something new about the material and about how to teach it!

tailfra 4y ago

Fantastic material! By the way, one of the simplest explanation of the difference between BetchNorm and LayerNorm

vorticalbox 4y ago

He has a website[0] for these videos [0] https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html

xt00 4y ago

I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training?

karpathy 4y ago
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf https://github.com/facebookresearch/metaseq/blob/main/projec... 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.
- mycall 4y ago
  GPUs are much less efficient than cores of the type Cerebras has made.
- vvrm 4y ago
  Thanks for laying out the plan. I was trying to understand the cost of each of these steps below and started wondering about the following: > rough steps: > 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ? > 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. It seems like the standard instructGPT model itself is based on a 1 billion param GPT model. Wouldn't that fit on a 24GB RTX 3090 ? Might take longer, maybe not enough opportunity for hyper-parameter search, but still possible right ? Or is hyper-parameter search on a thousand machines in parallel the real magic sauce here ? > 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec https://github.com/facebookresearch/metaseq/blob/main/projec... Sounds like a good opportunity to learn. No pain, no gain :-) > 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 https://arxiv.org/abs/2211.01786 to get a sense of the work involved here. Maybe somebody would open source the equivalent datasets for this soon ? Otherwise the data collection seems prohibitively expensive for somebody trying to do this for fun: contract expert annotators, train them, annotate/reannotate for months ?

keepquestioning 4y ago

The char-rnn guy does it again.

pottspotts 4y ago

This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.

javajosh 4y ago
Same, but one that doesn't assume python or an operating system.
- tysam_and 4y ago
  Same, but I would honestly rather we get away from silicon or anything resembling what we call these "computers" these days. Things are too complicated and we need to truly get back to the basics: rocks on a hill.
  - bitL 4y ago
    Abacus-driven deep learning with trained monkeys flipping gradients.
    - qup 4y ago
      Outside tools are not required. We'll selectively breed humans until one can just compute it in...meat[0]. https://www.youtube.com/watch?v=7tScAyNaRdQ https://www.youtube.com/watch?v=7tScAyNaRdQ
      - tysam_and 4y ago
        in mitro computing
    - captn3m0 4y ago
      You don't need to flip gradients, just need to train the monkeys to operate matchboxes: https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_Crosses_Engine https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_...
    - 4y ago

nlowell 4y ago

Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.

mensetmanusman 4y ago
Tesla kick started the global EV revolution, so probably favorably regardless of the few negatives that inherently come along the way.
coldtea 4y ago
As if history will care?
- dd36 4y ago
  The history AI will have an opinion that mirrors the politics of the user that asks it what it thinks of Karpathy’s time at Tesla.
  - burntalmonds 4y ago
    Only in the default bubble mode.
gorango 4y ago
What do you mean by "however"? He built the entire FSD architecture - in less time and with fewer people than Google, Apple, and god knows who else. He's likely saved more lives than most doctors at this point...
- jackblemming 4y ago
  No he didn't, a (large from what I hear) team did. However Karpathy is still a smart, well accomplished, guy and he has a talent many academics don't: being able to write actual production grade, high quality software. And he's a good communicator.
- amelius 4y ago

davewasthere 4y ago

Now be meta and build GPT with GPT.

ultrasounder 4y ago

Just started watching and Andrej is an excellent "thingexplainer". His in-depth knowledge of the underlying atoms/bits comes through. As an extra benefit to watching his lecture using the https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html is the link to his "discord chat". This is a very active community and Andrej is very active there. So feel free to watch lectures, cry through the assignments and come to the discord with questions. And they will be answered.

albert_e 4y ago

I am a simple man. I see a video post by karpathy, I upvote and watch. <end reddit-speak> I discovered Andrej very recently and I am a huge fan. Kudos to this whole effort! Two ideas -- 1. While these explainers are outstanding -- I can think of supplementary material/presentation that can nicely complement these explanations if they are presented visually. Especially the concepts of multidimensional tensors. Something like what 3B1B (or his followers that create 'Summer of Math Exposition' videos) does -- which is not a skill I have. I am thinking of creating some visual slides (my forte) but would there be interest in making this a larger collaboration that creates explainers for "visual learners"? 2. There should really be a discussion forum for people who follow along these "make more" tutorials -- to have discussions about each specific video, infact each specific timestamped chapter of these videos in that context. Is there a framework or tool that lets us integrate YouTube videos and timestamped chapters into a "discussion forum" -- whether a simple website or a discord/slack. Once again this is slightly outside my skillset but if it appeals to people, maybe some ideas and effort can come together to make this happen? EDIT: #facepalm -- I see there is already a discord [0] on the webpage [1] (but not likely tied to very specific chapters as I imagined -- but should be a excellent start anyway) [0] - https://discord.gg/3zy8kqD9Cp https://discord.gg/3zy8kqD9Cp [1] - https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html

indigochill 4y ago
In case you're not aware, 3B1B has a Github repo for the engine he uses for the math animations so that others can use it to make similar things: https://github.com/3b1b/manim https://github.com/3b1b/manim There's also a loose group of people already doing the visual learners "explainers" thing over here: https://explorabl.es/ https://explorabl.es/ (you can scroll down for links to tools they use to make their explainers). But yes, I also feel this is an important development and that this should be an ongoing way of teaching people things. Formal education has IMO stalled out around the printing press but there are massive opportunities on computers (and especially on globally networked computers) to take that a step further and leverage the capabilities of computers to make education even more engaging and information-dense.
deleted 4y ago
[deleted]

cdzm 4y ago

Karpathy also has a great Recipe for Training Neural Networks: http://karpathy.github.io/2019/04/25/recipe/ http://karpathy.github.io/2019/04/25/recipe/

thecleaner 4y ago

His cs231n course is how I learnt about neutral networks. And now on to learning GPT and Transformers.

lvl102 4y ago

A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.

bamboozled 4y ago
How can we really stop this though? I feel like Microsoft is doing what Microsoft does yet again. They piggybacked on the whole "open source" angle but this time, rather than be "software vendor lock-in closed source assholes", they're now being "IP theft open source" assholes.
- sharemywin 4y ago
  It's not just Microsoft. Google probably has models at least on par with OpenAI. Facebook probably too.
fithisux 4y ago
Totally agree. I also doing some Gcloud training. This should also be public good. When I move my machines to Google/AWS/ .. my business changes ownership and I become slave chained by the "competition" and "pricing". To fix this we need to change mentality and governments. But people are satisfied with their: "Mirror, Mirror on the Wall, Who's the Coolest of Them All? ... You, because you are indistinguishable from the others!!!!"
pjmorris 4y ago
> I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake. I don't know if these ideas are worse technically or politically, or both, but what comes to mind are these alternatives: 1. Have someone start a non-profit that curates public data goods, maybe gaining access to data through voluntary donations and through buying all the data provider feeds, and funding through subscriptions by people and organizations who want to understand the data provider infrastructure. 2. Get legislation passed that identifies public data goods and requires that they be made available to all.

abraxas 4y ago

For me he is the best educator in this space bar none. When Karpathy explains stuff it just clicks in my head.