4 ms·
I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.
by andai 3y ago
I heard GPT-4 described as "eight GPT-3's in a trenchcoat" but I'm not sure how accurate that is.
- brandall10 3y agoSeveral people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models. * https://arxiv.org/pdf/2101.03961.pdf https://arxiv.org/pdf/2101.03961.pdf
- j45 3y agoI wonder if additional layers of factories of factories approach can continue to improve it. I'm not familiar enough with the technology but could it be possible to create a prompt, or multiple prompts to stitch together 8 similtaneous calls to GPT3.5 pulled together and see if the quality is similar to GPT4?
- andai 3y agoI think the experts part implies that each "GPT-3" has been specialized in some way?
- brandall10 3y agoThey're not 3.5 models, which are 175B param, they're 220B and the idea is the fine tuning is different for each which is where the 'expert' comes in.
- j45 3y agoAh, that makes sense, thanks for clarifying
- ttul 3y agoThe mixture of experts approach was found to be inferior than a straightforward transformer. However, researchers discovered that MoE models give substantially better results when guided by fine-tuning. My guess is that GPT-4 was an experiment to prove out that theory at huge scale and I also suspect that its performance astonished OpenAI as much as everyone else.
- s1gnp0st 3y agoIf it was an experiment I'd like to know why I'm still paying for access to something claiming to be it.
- nicky0 3y agoBecause you are getting value from it. Why else would you pay?
- babushkanazi 3y ago[dead]
- insanitybit 3y agoBecause you signed up to pay for a beta service???? You can stop paying at any time.
- JohnFen 3y agoIf you're paying for it, it's a released product, not a beta. No matter how much a company wants to claim it's a "beta", it's simply not. The company just wants to be able to adhere to lower standards. OpenAI is hardly the only company that pulls these shenanigans, too.
- insanitybit 3y agoNo, beta has nothing to do with if it's paid or not. Lots of products are released, and you pay for them, with early access/ beta. No one lied to you. No one tricked you. It is in beta. You chose to pay for it.
- JohnFen 3y agoI never said anyone lied to or tricked anyone. I'm saying that "beta" is a term referring to pre-release testing. If people are paying for it, it's been released and is therefore no longer a beta. Calling it a "beta" at that point is just pure PR.
- rogerkirkness 3y agoIt's actually 16 GPT 3.5s in a trenchcoat where each one is slightly different, like the minions.
- andai 3y agoCan anyone elaborate on the specialization? Does it have to do with segmenting the dataset, or with training on different kinds of tasks?
- rogerkirkness 3y agoFine tuning to a specific more constrained set of tasks is my understanding.