4 ms·
"The MOE architecture uses 20 times the parameters, is this comparison fair? Can it be compared with a single model that also uses 20 times the parameters?"
by deyiao 3y ago
"The MOE architecture uses 20 times the parameters, is this comparison fair? Can it be compared with a single model that also uses 20 times the parameters?"
- jiggawatts 3y agoIt has more parameters, but not all of them are used during inference. They compared models that use equal numbers of parameters.