4 ms·
GPT-3 is the name for the architecture, but there are a few different versions/sizes. The OpenAI version that impressed us all was ~170B parameters, this is far
by nmfisher 6y ago
GPT-3 is the name for the architecture, but there are a few different versions/sizes. The OpenAI version that impressed us all was ~170B parameters, this is far smaller.
To go from 2.7B to 170B parameters will need more than just a few config tweaks. There's a whole bunch of hacks and tricks needed to coax a model to train at that scale, the Eleuther version is almost guaranteed to fail out-of-the-box.
- minimaxir 6y agoIt's worth noting that the GPT-3 paper did train models with more sane sizes (e.g. 1.5B) as a point of comparison. I am surprised/annoyed they never released them though.
- stellaathena 6y agoIt's because OpenAI sells them for profit. The "Ada" model is the same size as the larger of these two EleutherAI models.
- minimaxir 6y agoHuh, I was wondering what the size of the non-davinci models were; guess that make sense. It's still telling that a "small" GPT-3 model can risk cannibalizing a larger model.
- stellaathena 6y agoAda is 2.7B, Babbage is 6.7B, Curie is 13.0B, and DaVinci is 175B. The new one they announced last month is in the 20-50B range I think, not totally sure though.