3 ms·

All the nvidia/torch stuff is "only" to make it run two orders of magnitude faster or so. For the actual principles behind LLMS, 500 lines seems quite accurate.

• imjonse 3y ago
  The article talked about building a basic version of an LLM. Even basic versions like Karpathy's TinyStories ~100 million parameter models which manage to generate comprehensive English for a restricted vocabulary are trained for hours on an A100. If you want to skip the nvidia/torch stuff you'll be both training for days and having to write your own training code, which is not going to be just 500 lines. The principles are simple - but that is true for any important algorithm of the last 100 years. As opposed to LLMs, most of those can actually be done in a few hundreds of lines of code with no specialized hw and software stack required for it to be practical.
  • ckcheng 3y ago
    > Karpathy's TinyStories Did you mean Karpathy's tinyllamas? [1][2] Or did you mean Ronen Eldan and Yuanzhi Li's "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" [3][4] [1]: https://huggingface.co/karpathy/tinyllamas/tree/main https://huggingface.co/karpathy/tinyllamas/tree/main [2]: https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c [3]: https://www.microsoft.com/en-us/research/publication/tinystories-how-small-can-language-models-be-and-still-speak-coherent-english/ https://www.microsoft.com/en-us/research/publication/tinysto... [4]: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
    • imjonse 3y ago
      Karpathy's tinyllamas are based on the tiny stories dataset. I could have phrased it better, sorry.