3 ms·
If the rumors around GPT4 being a mixture of expert models are true, the this comparison is not fair. What would be interesting is compare GPT4 at a certain ta
by tarruda 3y ago
If the rumors around GPT4 being a mixture of expert models are true, the this comparison is not fair.
What would be interesting is compare GPT4 at a certain task with a small model fine tuned for that task.
- johnthewise 3y agoGPT4 being a mixture of experts is irrelevant imo like we don't care about how many layers there are in a network and how wide those layers are or which type of activation functions are actually used etc. all that matters are we can run it on a specific hardware and the results.
- redox99 3y agoExactly. I don't get why people (non AI researchers) discount MoE like they are cheating or fake parameters. Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.