3 ms·
How good is the quality of this? BLOOM is a 176B parameter model, but it doesn't seem to compare to GPT-3 (175B parameters) in terms of output quality.
by VadimPR 4y ago
How good is the quality of this? BLOOM is a 176B parameter model, but it doesn't seem to compare to GPT-3 (175B parameters) in terms of output quality.
- lossolo 4y agoIt's because BLOOM is undertrained, you can prune a lot of weights in BLOOM and it doesn't impact performance. Look at Chinchilla paper[1], 70B model outperforms 175B GPT-3 model. https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556
- Der_Einzige 4y agoIn general, most giant LLMs are extremely undertrained at this time. Consider that most of the gains in RoBerta vs bert were from just continuing to train.
- stevenhuang 4y agoCases of undertraining can be observed whenever the output is repeating gibberish or loops. Happened a lot in GPT2 ai dungeon days
- leobg 4y agoSo can we continue training RoBERTa to get it to, say, GPT3 Ada level
- rnosov 4y agoOut of curiosity, how did your measure their respective performances? My understanding is that BLOOM roughly comparable to GPT-3 in performance on most NLP tasks. Were you comparing OpenAI davinci to raw BLOOM by any chance?
- VadimPR 4y agoCompared ChatGPT to BLOOM - which I know doesn't benefit from RLHF.