6 ms·

If you like this, it's also worth looking at llama2.c[1], an implementation of the Llama 2 architecture in about 1000 lines of plain, dependency-free C, tokeniz

If you like this, it's also worth looking at llama2.c[1], an implementation of the Llama 2 architecture in about 1000 lines of plain, dependency-free C, tokenizer and all. THe fact that this 960-line file and a somewhat modern C compiler is all you really need to run a state-of-the-art language model is really surprising to many. Of course, this is not all there is to a modern LLM, it would probably take another thousand lines or two to implement training, and many more than that to make it fast on all the major CPU and GPU architectures. If you want a flexible framework that lets a developer define any model you want and still goes as fast as it can, the complexity spirals. Most programmers have an intuition that duplicating a large software project from scratch, like Linux or Chromium for example, would require incredible amounts of expertise, manpower and time. It's not something that a small team can achieve in a few months. You're limited by talent, not hardware. LLMs are very different. THe code isn't that complicated, you could probably implement training and inference for a single model architecture, from scratch, on a single kind of GPU, with reasonable performance, as an individual with a background in programming and who still remembers their calculus and linear algebra, with a year or so of self study. What makes LLMs difficult is getting access to all the hardware to train them, getting the data, and being able to preprocess that data.

fooker 2y ago
> THe code isn't that complicated. This is an indication that we’re at the infancy of this field.
evanjrowley 2y ago
Links for llama2.c: https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c https://news.ycombinator.com/item?id=36838051 https://news.ycombinator.com/item?id=36838051
Fubarberry 2y ago
There's also a project where they have GPT-2 running off of an excel spreadsheet. https://arstechnica.com/information-technology/2024/03/once-too-scary-to-release-gpt-2-gets-squeezed-into-an-excel-spreadsheet/ https://arstechnica.com/information-technology/2024/03/once-...
bradfox2 2y ago
I feel like this ignores the complexity of the distributed training frameworks. The challenge is in making it fast at scale.
andy99 2y ago
And if you want to understand I'd recommend this post (gpt2 in 60 lines of numpy) and the post on attention it links to. The concepts are mostly identical to llama, just with a few minor architectural tweaks. https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/
- bhavesh2712 2y ago
  Thanks for sharing this!
nicklecompte 2y ago
One other thing to add is large-scale RLHF. Big Tech can pay literally hundreds of technically-sophisticated people throughout the world (e.g. college grads in developing countries) to improve LLM performance on all sorts of specific problems. It is not a viable way to get AGI, but it means your LLM can learn tons of useful tricks that real people might want, and helps avoid embarrassing "mix broken glass into your baby formula" mistakes. (Obviously it is not foolproof.) I suspect GPT-4's "secret sauce" in terms of edging out competitors is that OpenAI is better about managing data contractors than the other folks. Of course it's a haze of NDAs to learn specifics, and clearly the contractors are severely underpaid compared to OpenAI employees/executives. But a lone genius with a platinum credit card can't create a new world-class LLM without help from others.
- stephc_int13 2y ago
  Yes, this is the secret sauce and the moat. Not as easy as buying more compute with unlimited budget. … built on the back of a disposable workforce… There is something grim and dystopian, thinking about the countless small hands feeding the machine.
  - factormeta 2y ago
    >There is something grim and dystopian, thinking about the countless small hands feeding the machine. Dystopian indeed, this is pretty much how Manhattan Project and CERN were done, with many independent contractors doing different parts, and only a few has the overview. A page out of corporate management book, it very much allows concentration of power in the hands of a few.
    - pagekicker 2y ago
      Very generous to compare to Manhattan Project or CERN.
      - factormeta 2y ago
        >Comparing LLMs to the Manhattan Project based on budget alone is stupid and arrogant Just want to clarify. The comparison to Manhattan Project or CERN is referencing "the countless small hands feeding the machine." In projects such as these, roles and jobs are divided into small parts, that people who are working on it don't really see the forest from the tree, and that only a few that has the picture of the whole project.
barrkel 2y ago
The code is much more similar, in principle, to a virtual machine. The actual code, the bit that contains the logic which has the semantics we intend, is in the trained weights, where the level of complexity is much higher and more subtle.
deleted 2y ago
[deleted]
netdevnet 2y ago
> What makes LLMs difficult is getting access to all the hardware to train them, getting the data, and being able to preprocess that data. Yes, that's my opinion too. GAOs (Grassroots AI Organisations) are constrained by access to data and the hardware needed to process the data and train the model on it. I look forward to a future where GAOs will crowdsource their computations in the same way many science labs borrow computing power from people around the world.
- miki123211 2y ago
  This is hard because you need high bandwidth between the GPUs in your cluster, bandwidth far higher than broadband could provide. I'm not even sure whether the time spend synchronizing between far-away machines would offset the increase in computational power.
AnthonyMouse 2y ago
> Most programmers have an intuition that duplicating a large software project from scratch, like Linux or Chromium for example, would require incredible amounts of expertise, manpower and time. It's not something that a small team can achieve in a few months. You're limited by talent, not hardware. But only for the same reasons. Linux runs on very nearly every piece of hardware ever made. The APIs you have to implement in order to run "Linux programs" are large and full of old complexity that exists for compatibility. Chromium is full of code to try to make pages render even though they were designed for Internet Explorer 6. Conversely, some university programs have students create a basic operating system from scratch. It's definitely something a small team can do as long as you don't care about broad hardware support or compatibility with existing applications. In principle a basic web browser is even simpler.
isaacfung 2y ago
I recommend reading https://github.com/bkitano/llama-from-scratch https://github.com/bkitano/llama-from-scratch over the article op linked. It actually teaches you how to build llama iteratively, test, debug and interpret the training loss rather than just desribing the code.
Const-me 2y ago
> you could probably implement training and inference for a single model architecture, from scratch, on a single kind of GPU, with reasonable performance… with a year or so I have implemented inference of Whisper https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper and Mistral https://github.com/Const-me/Cgml/tree/master/Mistral/MistralChat https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... models on all GPUs which support Direct3D 11.0 API. The performance is IMO very reasonable. A year might be required when the only input is the research articles. In practice, we also have reference Python implementations of these models. Possible to test different functions or compute shaders against the corresponding pieces from the reference implementations, by comparing saved output tensors between the reference and the newly built implementation. Due to that simple trick, I think I have spent less than 1 month part-time for each of these two projects.
- miki123211 2y ago
  I'd say a year for somebody who doesn't know what a linear layer is and couldn't explain why a GPU might be of any use if you're not playing games, but who knows what the derivative of 3x^2 is.
gmays 2y ago
> The code isn't that complicated, you could probably implement training and inference for a single model architecture, from scratch, on a single kind of GPU, with reasonable performance, as an individual with a background in programming and who still remembers their calculus and linear algebra, with a year or so of self study. Great overview. One gap I've been working on (daily) since October is the math working towards MA's Mathematics for Machine Learning course (https://mathacademy.com/courses/mathematics-for-machine-learning https://mathacademy.com/courses/mathematics-for-machine-lear...). I wrote about my progress (http://gmays.com/math http://gmays.com/math) if anyone else is interested in a similar path. I recently crossed 200 days of doing math daily (at least a lesson a day). It's definitely taking longer than I want, but I also have limited time (young kids + startup + investing). The 'year of self study' definitely depends on where you're starting from and how much time you have, but it's very doable if you can dedicate an hour or two a day.
deleted 2y ago
[deleted]
fasa99 2y ago
>" THe fact that this 960-line file and a somewhat modern C compiler is all you really need to run a state-of-the-art language model is really surprising to many." "the code for AGI will be simple" - John Deremetrius Carmack

fooker 2y ago

> THe code isn't that complicated. This is an indication that we’re at the infancy of this field.

evanjrowley 2y ago

Links for llama2.c: https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c https://news.ycombinator.com/item?id=36838051 https://news.ycombinator.com/item?id=36838051

Fubarberry 2y ago

There's also a project where they have GPT-2 running off of an excel spreadsheet. https://arstechnica.com/information-technology/2024/03/once-too-scary-to-release-gpt-2-gets-squeezed-into-an-excel-spreadsheet/ https://arstechnica.com/information-technology/2024/03/once-...

bradfox2 2y ago

I feel like this ignores the complexity of the distributed training frameworks. The challenge is in making it fast at scale.

andy99 2y ago

And if you want to understand I'd recommend this post (gpt2 in 60 lines of numpy) and the post on attention it links to. The concepts are mostly identical to llama, just with a few minor architectural tweaks. https://jaykmody.com/blog/gpt-from-scratch/ https://jaykmody.com/blog/gpt-from-scratch/

bhavesh2712 2y ago
Thanks for sharing this!

nicklecompte 2y ago

One other thing to add is large-scale RLHF. Big Tech can pay literally hundreds of technically-sophisticated people throughout the world (e.g. college grads in developing countries) to improve LLM performance on all sorts of specific problems. It is not a viable way to get AGI, but it means your LLM can learn tons of useful tricks that real people might want, and helps avoid embarrassing "mix broken glass into your baby formula" mistakes. (Obviously it is not foolproof.) I suspect GPT-4's "secret sauce" in terms of edging out competitors is that OpenAI is better about managing data contractors than the other folks. Of course it's a haze of NDAs to learn specifics, and clearly the contractors are severely underpaid compared to OpenAI employees/executives. But a lone genius with a platinum credit card can't create a new world-class LLM without help from others.

stephc_int13 2y ago
Yes, this is the secret sauce and the moat. Not as easy as buying more compute with unlimited budget. … built on the back of a disposable workforce… There is something grim and dystopian, thinking about the countless small hands feeding the machine.
- factormeta 2y ago
  >There is something grim and dystopian, thinking about the countless small hands feeding the machine. Dystopian indeed, this is pretty much how Manhattan Project and CERN were done, with many independent contractors doing different parts, and only a few has the overview. A page out of corporate management book, it very much allows concentration of power in the hands of a few.
  - pagekicker 2y ago
    Very generous to compare to Manhattan Project or CERN.
    - factormeta 2y ago
      >Comparing LLMs to the Manhattan Project based on budget alone is stupid and arrogant Just want to clarify. The comparison to Manhattan Project or CERN is referencing "the countless small hands feeding the machine." In projects such as these, roles and jobs are divided into small parts, that people who are working on it don't really see the forest from the tree, and that only a few that has the picture of the whole project.

barrkel 2y ago

The code is much more similar, in principle, to a virtual machine. The actual code, the bit that contains the logic which has the semantics we intend, is in the trained weights, where the level of complexity is much higher and more subtle.

deleted 2y ago

[deleted]

netdevnet 2y ago

> What makes LLMs difficult is getting access to all the hardware to train them, getting the data, and being able to preprocess that data. Yes, that's my opinion too. GAOs (Grassroots AI Organisations) are constrained by access to data and the hardware needed to process the data and train the model on it. I look forward to a future where GAOs will crowdsource their computations in the same way many science labs borrow computing power from people around the world.

miki123211 2y ago
This is hard because you need high bandwidth between the GPUs in your cluster, bandwidth far higher than broadband could provide. I'm not even sure whether the time spend synchronizing between far-away machines would offset the increase in computational power.

AnthonyMouse 2y ago

> Most programmers have an intuition that duplicating a large software project from scratch, like Linux or Chromium for example, would require incredible amounts of expertise, manpower and time. It's not something that a small team can achieve in a few months. You're limited by talent, not hardware. But only for the same reasons. Linux runs on very nearly every piece of hardware ever made. The APIs you have to implement in order to run "Linux programs" are large and full of old complexity that exists for compatibility. Chromium is full of code to try to make pages render even though they were designed for Internet Explorer 6. Conversely, some university programs have students create a basic operating system from scratch. It's definitely something a small team can do as long as you don't care about broad hardware support or compatibility with existing applications. In principle a basic web browser is even simpler.

isaacfung 2y ago

I recommend reading https://github.com/bkitano/llama-from-scratch https://github.com/bkitano/llama-from-scratch over the article op linked. It actually teaches you how to build llama iteratively, test, debug and interpret the training loss rather than just desribing the code.

Const-me 2y ago

> you could probably implement training and inference for a single model architecture, from scratch, on a single kind of GPU, with reasonable performance… with a year or so I have implemented inference of Whisper https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper and Mistral https://github.com/Const-me/Cgml/tree/master/Mistral/MistralChat https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... models on all GPUs which support Direct3D 11.0 API. The performance is IMO very reasonable. A year might be required when the only input is the research articles. In practice, we also have reference Python implementations of these models. Possible to test different functions or compute shaders against the corresponding pieces from the reference implementations, by comparing saved output tensors between the reference and the newly built implementation. Due to that simple trick, I think I have spent less than 1 month part-time for each of these two projects.

miki123211 2y ago
I'd say a year for somebody who doesn't know what a linear layer is and couldn't explain why a GPU might be of any use if you're not playing games, but who knows what the derivative of 3x^2 is.

gmays 2y ago

> The code isn't that complicated, you could probably implement training and inference for a single model architecture, from scratch, on a single kind of GPU, with reasonable performance, as an individual with a background in programming and who still remembers their calculus and linear algebra, with a year or so of self study. Great overview. One gap I've been working on (daily) since October is the math working towards MA's Mathematics for Machine Learning course (https://mathacademy.com/courses/mathematics-for-machine-learning https://mathacademy.com/courses/mathematics-for-machine-lear...). I wrote about my progress (http://gmays.com/math http://gmays.com/math) if anyone else is interested in a similar path. I recently crossed 200 days of doing math daily (at least a lesson a day). It's definitely taking longer than I want, but I also have limited time (young kids + startup + investing). The 'year of self study' definitely depends on where you're starting from and how much time you have, but it's very doable if you can dedicate an hour or two a day.

deleted 2y ago

[deleted]

fasa99 2y ago

>" THe fact that this 960-line file and a somewhat modern C compiler is all you really need to run a state-of-the-art language model is really surprising to many." "the code for AGI will be simple" - John Deremetrius Carmack