markusheimerl

3 ms·

1. 1.▲
   by markusheimerl 4mo ago

   https://github.com/markusheimerl/gpt/blob/main/train.c - in this file, search for the line "const int batch_size = 15;" - reduce this number
2. 2.▲
   by markusheimerl 4mo ago

   Sure it could be extended to support LoRA finetuning but this implementation has the goal to be as lean and efficient as possible for a pre-training stack as you can be.
3. 3.▲
   by markusheimerl 4mo ago

   the data gets downloaded via curl from huggingface - sure you can make your own data, simply dump all text you want the model to be trained on into "corpus.txt" and skip "make data". As the tokenizer adds substantial com
4. 4.▲
   by markusheimerl 4mo ago

   I did run it as a test on the NVIDIA Jetson Orin Nano Super Dev. Kit once - so yea it works on arm like a charm ;)
5. 5.▲
   by markusheimerl 4mo ago

   Reduce batch size in train.c
6. 6.▲
   by markusheimerl 4mo ago

   I deleted the numerical checks a while back after confirming the backward pass is correct to keep the code base lean - running https://github.com/markusheimerl/gpt/blob/main/transformer/a... is also
7. 7.▲
   Tiny hackable CUDA language model implementation(github.com)

   81 points by markusheimerl 4mo ago|13 comments
8. 8.▲
   Stick in Bike Wheel OOP(imgflip.com)

   1 points by markusheimerl 2y ago|0 comments