3 ms·
I think people are overestimating how many parameters GPT-4 is really trained on. While also overestimating how many parameters anybody really needs after fine
by yieldcrv 4y ago
I think people are overestimating how many parameters GPT-4 is really trained on.
While also overestimating how many parameters anybody really needs after fine tuning.
The market will find the sweet spot. Right now everyone’s tinkering with the 7B parameter LLM and then going to move up to the 65B one once they've refined the process. I think its fiction that anybody really needs a 10 trillion parameter LLM at all. It will be completely niche.
- bitL 4y agoFine-tuning is a solved problem using transformer adapters which are fast, small and match regular fine-tuning, so that's not an issue. As for smaller models, their usefulness will quickly wane when they will be seen as producing trivial gibberish in the light of cloud models available in 5 years.