4 ms·
GPT4 being a mixture of experts is irrelevant imo like we don't care about how many layers there are in a network and how wide those layers are or which type of
by johnthewise 3y ago
GPT4 being a mixture of experts is irrelevant imo like we don't care about how many layers there are in a network and how wide those layers are or which type of activation functions are actually used etc. all that matters are we can run it on a specific hardware and the results.
- redox99 3y agoExactly. I don't get why people (non AI researchers) discount MoE like they are cheating or fake parameters. Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.