4 ms·
Could an open source/data GPT like chatbot be created?
The resulting InstructGPT models are much better at following instructions than GPT-3. They also make up facts less often, and show small decreases in toxic output generation. Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters. At the same time, we show that we don’t have to compromise on GPT-3’s capabilities, as measured by our model’s performance on academic NLP evaluations.
https://openai.com/blog/instruction-following/
It seems like the a 1.3B parameter model wouldn't require massive resources.
I understand training data could be hard to come by but is it in the realm of possiblity
- seydor 4y agoQuestion is when. In any case i would like to be able to run them locally and save all the chats we make to feed them to future versions. Over the years this would become a copy of my brain. And when i die we can keep on chatting
- cottonseed 4y agoYes. EleutherAI is doing it, probably one of many: https://www.eleuther.ai/projects/gpt-neox/ https://www.eleuther.ai/projects/gpt-neox/ https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox https://arxiv.org/abs/2204.06745 https://arxiv.org/abs/2204.06745 They have a 20B parameter model. I think the primary dataset for these open models is The Pile: https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 (web scrape, pubmed, arxiv, github, wikipedia, etc. There is a nice diagram on page 2 that summarizes the contents.)
- sharemywin 4y agofrom what I gather the pile is only a first step. it would require more steps. task oriented chats. as well as building something that can rate answers.