4 ms·
I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?
by josu 2mo ago
I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?
- l33tman 2mo agoCould it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?
- marci 2mo agoNothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks. https://machinelearning.apple.com/research/introducing-third-generation-of-apple-foundation-models https://machinelearning.apple.com/research/introducing-third...
- josu 2mo agoThanks for responding.