4 ms·

This metric from the article is a bit misleading. It's not 500 lines in the same sense as mergesort is 20 lines. The whole of the deep learning stack including

This metric from the article is a bit misleading. It's not 500 lines in the same sense as mergesort is 20 lines. The whole of the deep learning stack including nvidia firmware blobs is used by those 500 lines. You may create a small NN from scratch in 500 lines for training a toy dataset, but nothing to actually train real LLMs unless you use the existing stack. Even Karpathy's no-torch C only 500 lines Llama2 code is inference only. So it is 500 extra lines compared to what we already had before, and in this sense it indeed is a breakthrough. Otherwise you could also say it's 10 lines of fastAI/lightning style code where almost everything is hidden by the APIs.

miki123211 3y ago
FOr inference, less than 1KLOC of pure, dependency-free C is enough (if you include the tokenizer and command line parsing)[1]. This was a non-obvious fact for me, in principle, you could run a modern LLM 20 years ago with just 1000 lines of code, assuming you're fine with things potentially taking days to run of course. Training wouldn't be that much harder, Micrograd[2] is 200LOC of pure Python, 1000 lines would probably be enough for training an (extremely slow) LLM. By "extremely slow", I mean that a training run that normally takes hours could probably take dozens of years, but the results would, in principle, be the same. If you were writing in C instead of Python and used something like Llama CPP's optimization tricks, you could probably get somewhat acceptable training performance in 2 or 3 KLOC. You'd still be off by one or two orders of magnitude when compared to a GPU cluster, but a lot better than naive, loopy Python. [1] https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c [2] https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd
IanCal 3y ago
I don't think it's misleading, and the responses of exactly how much code you can get away with is IMO missing the point. The point is not about the total stack size or anything of the sort, it's that they didn't require much new stuff to be built. They're tools that can answer questions, role play, translate, write code and more and the architecture isn't a huge new system. It's not "First we do this, then we encode like this, then we perform a graph search, then we rank, then we decide which subsystem to start, then we start the custom planner, then a custom iteration over..." It's not a large, unique thing only one group knows how to make. > Otherwise you could also say it's 10 lines of fastAI/lightning style code where almost everything is hidden by the APIs. Sure!
mmoskal 3y ago
All the nvidia/torch stuff is "only" to make it run two orders of magnitude faster or so. For the actual principles behind LLMS, 500 lines seems quite accurate. How long would it take to explain what an LLMs is doing to someone from 1800s? It is remarkably simple.
- imjonse 3y ago
  The article talked about building a basic version of an LLM. Even basic versions like Karpathy's TinyStories ~100 million parameter models which manage to generate comprehensive English for a restricted vocabulary are trained for hours on an A100. If you want to skip the nvidia/torch stuff you'll be both training for days and having to write your own training code, which is not going to be just 500 lines. The principles are simple - but that is true for any important algorithm of the last 100 years. As opposed to LLMs, most of those can actually be done in a few hundreds of lines of code with no specialized hw and software stack required for it to be practical.
  - ckcheng 3y ago
    > Karpathy's TinyStories Did you mean Karpathy's tinyllamas? [1][2] Or did you mean Ronen Eldan and Yuanzhi Li's "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" [3][4] [1]: https://huggingface.co/karpathy/tinyllamas/tree/main https://huggingface.co/karpathy/tinyllamas/tree/main [2]: https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c [3]: https://www.microsoft.com/en-us/research/publication/tinystories-how-small-can-language-models-be-and-still-speak-coherent-english/ https://www.microsoft.com/en-us/research/publication/tinysto... [4]: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
    - imjonse 3y ago
      Karpathy's tinyllamas are based on the tiny stories dataset. I could have phrased it better, sorry.
imjonse 3y ago
And also this assumes base models. The breakthrough where the instruct/chat models which again are much more code than 500 lines of plain training.
- b112 3y ago
  It's like claiming a mod on top of an existing game makes it a new game, in a few line if code. It can, but obviously it's the engine and existing code doing all the work.
- hnfong 3y ago
  What are you talking about? Fine tunes are basically just more of the same training, optionally on selected layers for efficiency.
  - imjonse 3y ago
    RLHF or DPO are definitely not just the same thing as the basic torch training loop, hence my many more lines of code argument.
simonw 3y ago
Sure - there are indeed millions of lines of code if you include the operating system, the Python interpreter, PyTorch, CUDA etc. But... you don't have to write that code. A research group constructing a brand new LLM really does only need a few hundred lines of code for the core training and inference. They need to spend their time on the data instead. That's what makes LLMs "easy" to build. You don't need to write millions of lines of custom code to build an "AI", which I think is pretty surprising.

miki123211 3y ago

FOr inference, less than 1KLOC of pure, dependency-free C is enough (if you include the tokenizer and command line parsing)[1]. This was a non-obvious fact for me, in principle, you could run a modern LLM 20 years ago with just 1000 lines of code, assuming you're fine with things potentially taking days to run of course. Training wouldn't be that much harder, Micrograd[2] is 200LOC of pure Python, 1000 lines would probably be enough for training an (extremely slow) LLM. By "extremely slow", I mean that a training run that normally takes hours could probably take dozens of years, but the results would, in principle, be the same. If you were writing in C instead of Python and used something like Llama CPP's optimization tricks, you could probably get somewhat acceptable training performance in 2 or 3 KLOC. You'd still be off by one or two orders of magnitude when compared to a GPU cluster, but a lot better than naive, loopy Python. [1] https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c [2] https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd

IanCal 3y ago

I don't think it's misleading, and the responses of exactly how much code you can get away with is IMO missing the point. The point is not about the total stack size or anything of the sort, it's that they didn't require much new stuff to be built. They're tools that can answer questions, role play, translate, write code and more and the architecture isn't a huge new system. It's not "First we do this, then we encode like this, then we perform a graph search, then we rank, then we decide which subsystem to start, then we start the custom planner, then a custom iteration over..." It's not a large, unique thing only one group knows how to make. > Otherwise you could also say it's 10 lines of fastAI/lightning style code where almost everything is hidden by the APIs. Sure!

mmoskal 3y ago

All the nvidia/torch stuff is "only" to make it run two orders of magnitude faster or so. For the actual principles behind LLMS, 500 lines seems quite accurate. How long would it take to explain what an LLMs is doing to someone from 1800s? It is remarkably simple.

imjonse 3y ago
The article talked about building a basic version of an LLM. Even basic versions like Karpathy's TinyStories ~100 million parameter models which manage to generate comprehensive English for a restricted vocabulary are trained for hours on an A100. If you want to skip the nvidia/torch stuff you'll be both training for days and having to write your own training code, which is not going to be just 500 lines. The principles are simple - but that is true for any important algorithm of the last 100 years. As opposed to LLMs, most of those can actually be done in a few hundreds of lines of code with no specialized hw and software stack required for it to be practical.
- ckcheng 3y ago
  > Karpathy's TinyStories Did you mean Karpathy's tinyllamas? [1][2] Or did you mean Ronen Eldan and Yuanzhi Li's "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" [3][4] [1]: https://huggingface.co/karpathy/tinyllamas/tree/main https://huggingface.co/karpathy/tinyllamas/tree/main [2]: https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c [3]: https://www.microsoft.com/en-us/research/publication/tinystories-how-small-can-language-models-be-and-still-speak-coherent-english/ https://www.microsoft.com/en-us/research/publication/tinysto... [4]: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
  - imjonse 3y ago
    Karpathy's tinyllamas are based on the tiny stories dataset. I could have phrased it better, sorry.

imjonse 3y ago

And also this assumes base models. The breakthrough where the instruct/chat models which again are much more code than 500 lines of plain training.

b112 3y ago
It's like claiming a mod on top of an existing game makes it a new game, in a few line if code. It can, but obviously it's the engine and existing code doing all the work.
hnfong 3y ago
What are you talking about? Fine tunes are basically just more of the same training, optionally on selected layers for efficiency.
- imjonse 3y ago
  RLHF or DPO are definitely not just the same thing as the basic torch training loop, hence my many more lines of code argument.

simonw 3y ago

Sure - there are indeed millions of lines of code if you include the operating system, the Python interpreter, PyTorch, CUDA etc. But... you don't have to write that code. A research group constructing a brand new LLM really does only need a few hundred lines of code for the core training and inference. They need to spend their time on the data instead. That's what makes LLMs "easy" to build. You don't need to write millions of lines of custom code to build an "AI", which I think is pretty surprising.