4 ms·

Ask HN: How to Learn LLM Fundamentals

30 points by AETackaberry 4y ago

As a Software Engineer that is interested in learning LLM fundamentals, what is your recommended course of action? Are there any courses or curricula you would recommend?

hislaziness 4y ago
I really liked this series. https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html
- AETackaberry 4y ago
  I am definitely planning to start with this series, seems like a no-brainer! The part I'm unsure of is what next? Seems like diving into papers might be a decent leap. Who knows, maybe GPT-4 can help me bridge the gap
  - hislaziness 4y ago
    Papers would be good next step. The concept and the algorithms do not seem very complex. The training and fine tuning is what needs effort and infrastructure.
- quickthrower2 4y ago
  Thanks that series looks quite fun to do
seydor 4y ago
There s this guy who has made in depth videos about transformers https://www.youtube.com/watch?v=O3xbVmpdJwU https://www.youtube.com/watch?v=O3xbVmpdJwU
is_true 4y ago
I think this blog posts is nice overview to help you go deeper in the topics mentioned: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
jytechdevops 4y ago
What a coincidence, I actually ran across this github repo for LLM fundamentals today: https://gist.github.com/rain-1/eebd5e5eb2784feecf450324e3341c8d https://gist.github.com/rain-1/eebd5e5eb2784feecf450324e3341... gl on your journey
TroyZ 4y ago
For a high level overview of transformers/GPT, you can check out this article: https://medium.com/design-bootcamp/how-chatgpt-really-works-explained-for-non-technical-people-71efb078a5c9 https://medium.com/design-bootcamp/how-chatgpt-really-works-... The article provides references to the original papers and other articles explaining the subject, which can be great sources to dive deeper.
felipelalli 4y ago
http://www.iasylum.net/writings/2023-03-29-if-you-are-curious-about-LLMs.html http://www.iasylum.net/writings/2023-03-29-if-you-are-curiou...
bfg_damien 4y ago
Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: https://twitter.com/hippopedoid/status/1641432291149848576?s=20 https://twitter.com/hippopedoid/status/1641432291149848576?s... 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: https://arxiv.org/pdf/2207.09238.pdf https://arxiv.org/pdf/2207.09238.pdf 3. Another thread specifically on large language models: https://twitter.com/cwolferesearch/status/1640446111348555776?s=20 https://twitter.com/cwolferesearch/status/164044611134855577... Once again these are not courses per se, but do provide intuitive explanations for how transformers work. There is also the nanoGPT series of videos by Karpathy on youtube. First video here: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY

30 points by AETackaberry 4y ago

As a Software Engineer that is interested in learning LLM fundamentals, what is your recommended course of action? Are there any courses or curricula you would recommend?

hislaziness 4y ago

I really liked this series. https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html

AETackaberry 4y ago
I am definitely planning to start with this series, seems like a no-brainer! The part I'm unsure of is what next? Seems like diving into papers might be a decent leap. Who knows, maybe GPT-4 can help me bridge the gap
- hislaziness 4y ago
  Papers would be good next step. The concept and the algorithms do not seem very complex. The training and fine tuning is what needs effort and infrastructure.
quickthrower2 4y ago
Thanks that series looks quite fun to do

seydor 4y ago

There s this guy who has made in depth videos about transformers https://www.youtube.com/watch?v=O3xbVmpdJwU https://www.youtube.com/watch?v=O3xbVmpdJwU

is_true 4y ago

I think this blog posts is nice overview to help you go deeper in the topics mentioned: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

jytechdevops 4y ago

What a coincidence, I actually ran across this github repo for LLM fundamentals today: https://gist.github.com/rain-1/eebd5e5eb2784feecf450324e3341c8d https://gist.github.com/rain-1/eebd5e5eb2784feecf450324e3341... gl on your journey

TroyZ 4y ago

For a high level overview of transformers/GPT, you can check out this article: https://medium.com/design-bootcamp/how-chatgpt-really-works-explained-for-non-technical-people-71efb078a5c9 https://medium.com/design-bootcamp/how-chatgpt-really-works-... The article provides references to the original papers and other articles explaining the subject, which can be great sources to dive deeper.

felipelalli 4y ago

http://www.iasylum.net/writings/2023-03-29-if-you-are-curious-about-LLMs.html http://www.iasylum.net/writings/2023-03-29-if-you-are-curiou...

bfg_damien 4y ago

Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: https://twitter.com/hippopedoid/status/1641432291149848576?s=20 https://twitter.com/hippopedoid/status/1641432291149848576?s... 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: https://arxiv.org/pdf/2207.09238.pdf https://arxiv.org/pdf/2207.09238.pdf 3. Another thread specifically on large language models: https://twitter.com/cwolferesearch/status/1640446111348555776?s=20 https://twitter.com/cwolferesearch/status/164044611134855577... Once again these are not courses per se, but do provide intuitive explanations for how transformers work. There is also the nanoGPT series of videos by Karpathy on youtube. First video here: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY