5 ms·

Building an LLM from Scratch: Automatic Differentiation (2023)

355 points by netwrt 3y ago

asgraham 3y ago
As a chronic premature optimizer my first reaction was, "Is this even possible in vanilla python???" Obviously it's possible, but can you train an LLM before the heat death of the universe? A perceptron, sure, of course. A deep learning model, plausible if it's not too deep. But a large language model? I.e. the kind of LLM necessary for "from vanilla python to functional coding assistant." But obviously the author already thought of that. The source repo has a great motto: "It don't go fast but it do be goin'" [1] I love the idea of the project and I'm curious to see what the endgame runtime will be. [1] https://github.com/bclarkson-code/Tricycle https://github.com/bclarkson-code/Tricycle
- gkbrk 3y ago
  Why wouldn't it be possible? You can generate machine code with Python and call into it with ctypes. All your deep learning code is still in Python, but in the runtime it gets JIT compiled into something faster.
cafaxo 3y ago
I did a similar thing for Julia: Llama2.jl contains vanilla Julia code [1] for training small Llama2-style models on the CPU. [1] https://github.com/cafaxo/Llama2.jl/tree/master/src/training https://github.com/cafaxo/Llama2.jl/tree/master/src/training
- andxor_ 3y ago
  Great stuff. Thanks for sharing.
- 3abiton 3y ago
  How hard was it to find open source data nowadays? I saw that books3 are already made illegal to train on.
itissid 3y ago
Every one should go through this rite of passage work and get to the "Attention is all you need" implementation. It's a world where engineering and the academic papers are very close and reproducible and a must for you to progress in the field. (see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)
- bschne 3y ago
  +1 for Karpathy, the series is really good
- calebkaiser 3y ago
  I would also recommend going through Callum McDougall/Neel Nanda's fantastic Transformer from Scratch tutorial. It takes a different approach to conceptualizing the model (or at least, it implements it in a way which emphasizes different characteristics of Transformers and self-attention), which I found deeply satisfying when I first explored them. https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Transformer_from_Scratch https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Trans...
  - joshua11 3y ago
    Thanks for sharing. This is a nice resource
- wredue 3y ago
  Is this YouTube series also “from scratch (but not really)” Edit - it is. Not to talk down on the series. I’m sure it’s good, but it is actually “LLM with PyTorch”. Edit - I looked again and I was actually not correct. He does ultimately use frameworks, but gives some early talk about how those function under the hood.
  - deadfast 3y ago
    I appreciate you coming back and giving more details, it encourages me to look into it now. Maybe my expectations on the internet are just low, but I thought it was a virtuous act worth the effort, I wish more people would continue with skepticism but be willing to follow through and let their opinions change given solid evidence.
nqzero 3y ago
is there an existing SLM that resembles an LLM in architecture that includes the code for training it ? i realize the cost and time to train may be prohibitive and that quality on general english might be very limited, but is the code itself available ?
- sva_ 3y ago
  Not sure what you mean with SLM, but https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT
andxor_ 3y ago
Very well written. AD is like magic and this is a good exposition on the basic building block. I quite like Jeremy's approach: https://nbviewer.org/github/fastai/fastbook/blob/master/17_foundations.ipynb https://nbviewer.org/github/fastai/fastbook/blob/master/17_f... It shows a very simple "Pythonic" approach to assemble gradient of a composition of functions from the gradients of the components.
revskill 3y ago
The only problem is it's implemented in Python. One reason is i hate to install python on my machine, and i don't know how to manage dependencies. The MacOSX required to upgrade to install native stuffs. Such a hell.
ESOLprof 3y ago
Amazing! Thank you

355 points by netwrt 3y ago

As a chronic premature optimizer my first reaction was, "Is this even possible in vanilla python???" Obviously it's possible, but can you train an LLM before the heat death of the universe? A perceptron, sure, of course. A deep learning model, plausible if it's not too deep. But a large language model? I.e. the kind of LLM necessary for "from vanilla python to functional coding assistant." But obviously the author already thought of that. The source repo has a great motto: "It don't go fast but it do be goin'" [1] I love the idea of the project and I'm curious to see what the endgame runtime will be. [1] https://github.com/bclarkson-code/Tricycle https://github.com/bclarkson-code/Tricycle

gkbrk 3y ago
Why wouldn't it be possible? You can generate machine code with Python and call into it with ctypes. All your deep learning code is still in Python, but in the runtime it gets JIT compiled into something faster.

cafaxo 3y ago

I did a similar thing for Julia: Llama2.jl contains vanilla Julia code [1] for training small Llama2-style models on the CPU. [1] https://github.com/cafaxo/Llama2.jl/tree/master/src/training https://github.com/cafaxo/Llama2.jl/tree/master/src/training

andxor_ 3y ago
Great stuff. Thanks for sharing.
3abiton 3y ago
How hard was it to find open source data nowadays? I saw that books3 are already made illegal to train on.

itissid 3y ago

Every one should go through this rite of passage work and get to the "Attention is all you need" implementation. It's a world where engineering and the academic papers are very close and reproducible and a must for you to progress in the field. (see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)

bschne 3y ago
+1 for Karpathy, the series is really good
calebkaiser 3y ago
I would also recommend going through Callum McDougall/Neel Nanda's fantastic Transformer from Scratch tutorial. It takes a different approach to conceptualizing the model (or at least, it implements it in a way which emphasizes different characteristics of Transformers and self-attention), which I found deeply satisfying when I first explored them. https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Transformer_from_Scratch https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Trans...
- joshua11 3y ago
  Thanks for sharing. This is a nice resource
wredue 3y ago
Is this YouTube series also “from scratch (but not really)” Edit - it is. Not to talk down on the series. I’m sure it’s good, but it is actually “LLM with PyTorch”. Edit - I looked again and I was actually not correct. He does ultimately use frameworks, but gives some early talk about how those function under the hood.
- deadfast 3y ago
  I appreciate you coming back and giving more details, it encourages me to look into it now. Maybe my expectations on the internet are just low, but I thought it was a virtuous act worth the effort, I wish more people would continue with skepticism but be willing to follow through and let their opinions change given solid evidence.

nqzero 3y ago

is there an existing SLM that resembles an LLM in architecture that includes the code for training it ? i realize the cost and time to train may be prohibitive and that quality on general english might be very limited, but is the code itself available ?

sva_ 3y ago
Not sure what you mean with SLM, but https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT

andxor_ 3y ago

Very well written. AD is like magic and this is a good exposition on the basic building block. I quite like Jeremy's approach: https://nbviewer.org/github/fastai/fastbook/blob/master/17_foundations.ipynb https://nbviewer.org/github/fastai/fastbook/blob/master/17_f... It shows a very simple "Pythonic" approach to assemble gradient of a composition of functions from the gradients of the components.

revskill 3y ago

The only problem is it's implemented in Python. One reason is i hate to install python on my machine, and i don't know how to manage dependencies. The MacOSX required to upgrade to install native stuffs. Such a hell.

ESOLprof 3y ago

Amazing! Thank you