3 ms·
I want an open version of this that I can train on any data I choose, can give any prompt I want, and runs on my hardware. However, I feel like the best and mo
by monkpit 4y ago
I want an open version of this that I can train on any data I choose, can give any prompt I want, and runs on my hardware.
However, I feel like the best and most useful versions of this type of software would (by necessity) only be able to run at a huge scale, cost millions to train, etc.
Now instead of having to worry about the bias of the sources I read, I have to question the bias of the language model based upon who created it.
- haldujai 4y agoHow do we even define 'best and most useful' for a language model? If you have your own data and problems you are trying to solve I don't think using GPT4 in zero-few shot mode (even with CoT) is necessarily better/higher performing than using a smaller fine-tuned or retrieval augmented model, it certainly wasn't true for GPT3.5 although we don't have as much insight into GPT4. If we're talking about something as general/open-ended as ChatGPT or a 'poor mans AGI' well yeah, the data curation and preprocessing alone takes a small team to figure out let alone training it. How useful this type of model is when it's still not good enough (i.e. hallucination) to be used in most production cases despite the exorbitant cost in development remains to be seen.
- kjreact 4y agoI would also like to avoid a future where we are slaves to corporate AI. If training the whole model individually isn’t feasible, would it be possible to decentralize the training?
- monkpit 4y agoThat’s an interesting idea. Something like Folding at home, or coin mining.
- inawarminister 4y agoCheck out KoboldAI community, they're collaboratively finetuning and hosting GPT-J, NeoX, and now LLAMA models for free, as hobbies.