6 ms·
Pure Rust implementation of a minimal GPT language model
- keyvank 3y agofemtoGPT is a pure Rust implementation of a minimal Generative Pretrained Transformer. Everything is implemented from scratch, including the tensor processing logic along with training/inference code of a minimal GPT architecture.
- allisdust 3y agoThanks for making this. Was searching for a pure rust GPT implementation just the other day!
- mrkeen 3y ago> After hours of training on the Shakespeare database, on a 300k parameter model, this has been the output I'm a bit out of the loop. What does it mean to train on Shakespeare? Does this imply that it's Shakespeare and a whole bunch of other English text? I tried generating Shakespeare a few years back using something quite primitive by today's standards. (Maybe it was word2vec? I don't remember.). I was surprised that it emitted real Shakespeare-sounding stuff! But then someone explained that there is so little Shakespeare to train on, that you'll only ever reproduce parts of the actual text (which indeed I had). So what are the expectations of this GPT model? Should it be bound by the same limitations?
- gandalfff 3y agoDoes this have potential for running on an SBC? Mine only has 2GB of RAM and I'm interested in running a toy language model on it.