35 ms·
They reference "a GPT-3 model with 356M parameters" So GPT-3 Medium (from the GPT-3 paper) - feels pretty disingenuous to list that as no one is referencing th
by jasonjmcghee 2y ago
They reference "a GPT-3 model with 356M parameters"
So GPT-3 Medium (from the GPT-3 paper) - feels pretty disingenuous to list that as no one is referencing that model when they say "GPT-3", but the 175B model.
I wasn't aware that size of the model (356M) was released- what am I missing here?
I also think it's relatively well understood that (with our current methods) transformers have a tipping point with parameter count, and I don't know of any models less than ~3B that are useful- arguably 7B.
Compare these benchmarks to, say, the RWKV 5/6 paper https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2404.05892
- CuriouslyC 2y agophi3 mini is surprisingly capable given its size. You can teach small transformers to do stuff well, you just can't have good general purpose small models.
- jasonjmcghee 2y agoTotally. But they aren't fine tuning these afaict- but comparing general purpose capabilities.
- Der_Einzige 2y agoThe point still stands, Phi3 is an excellent model and shows that good models don’t need that many parameters You should see the work on ReFT coming from mannings group showing that you can instruction fine tune models by modifying like, 0.00001% of the parameters. By doing it this way, you significantly mitigate the risk of catastrophic forgetting.