3 ms·
even if true which it does not seem to be the case, the whole thing sounds pretty marginal, in order to train a model that is most likely significantly bigger t
by wdpk 4y ago
even if true which it does not seem to be the case, the whole thing sounds pretty marginal, in order to train a model that is most likely significantly bigger than 100b parameters, one also needs orders of magnitude more training data than the small 120k chat that were shared on the ShareGPT website
- halfeatenscone 4y agoSuch logs would not be used for training the base model, but rather for fine-tuning the model for instruction following. Instruction tuning requires far less data than is needed for pre-training the foundation model. Stanford Alpaca showed surprisingly strong results from fine-tuning Meta's LLaMA model on just 52k ChatGPT-esque interactions (https://crfm.stanford.edu/2023/03/13/alpaca.html https://crfm.stanford.edu/2023/03/13/alpaca.html).
- wdpk 4y agowell, the initial twitter rant was pretty bombastic: "The cat is finally out of the bag – Google relied heavily on @ShareGPT 's data when training Bard. This was also why we took down ShareGPT's Explore page – which has over 112K shared conversations – last week. Insanity." Fine-tunning is not exactly the same as "relying heavily", I bet they got way more fine-tunning data from simply asking their 100k employees to pre-beta test for a couple of months