4 ms·

This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-tr

This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-trained model locally? Surely the specs required to run a model are much, much lower than those required to train the model?

nerdponx 4y ago
It's the equivalent of building from source versus downloading a compiled binary. Also you can perform "fine tuning" which means you start with a trained model and train it further on your own data, allowing you to customize the model for specific tasks.
ausbah 4y ago
inference can still be a bottleneck i think since you usually load the whole thing into memory which is 32-64GB+ usually?
- visarga 4y ago
  Language models range from 1 to 300+ GB when loaded. It depends on how you load them, if you load in int8 you get 4x reduction.
anon291 4y ago
If you're only using pre-trained models, it's going to be harder to differentiate yourself. Training / specialization of models is where the moat-building is (due to access to different data sets / better ideas). By specializing / training, more of the token limit can be used for generation rather than prompting / better prompts can be made. The lower the cost of training, the more profitable any resultant business. You can even envision businesses that train the model regularly to bring in new knowledge. The cheaper this is, the more opportunities open up.
code_runner 4y ago
I believe the training is where the architecture of the model is most apparent. You can absolutely download plenty of pre-trained models. You will also probably need to fine tune for a specific use case, so a common approach is downloading a pre-trained model and fine tuning. I think including the “from scratch” tuning script is educational more than anything else.

4 ms·

This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-tr

by justusthane 4y ago

nerdponx 4y ago
It's the equivalent of building from source versus downloading a compiled binary. Also you can perform "fine tuning" which means you start with a trained model and train it further on your own data, allowing you to customize the model for specific tasks.
ausbah 4y ago
inference can still be a bottleneck i think since you usually load the whole thing into memory which is 32-64GB+ usually?
- visarga 4y ago
  Language models range from 1 to 300+ GB when loaded. It depends on how you load them, if you load in int8 you get 4x reduction.
anon291 4y ago
If you're only using pre-trained models, it's going to be harder to differentiate yourself. Training / specialization of models is where the moat-building is (due to access to different data sets / better ideas). By specializing / training, more of the token limit can be used for generation rather than prompting / better prompts can be made. The lower the cost of training, the more profitable any resultant business. You can even envision businesses that train the model regularly to bring in new knowledge. The cheaper this is, the more opportunities open up.
code_runner 4y ago
I believe the training is where the architecture of the model is most apparent. You can absolutely download plenty of pre-trained models. You will also probably need to fine tune for a specific use case, so a common approach is downloading a pre-trained model and fine tuning. I think including the “from scratch” tuning script is educational more than anything else.