7 ms·

They do give a hint: "This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture." One thing you coul

They do give a hint: "This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture." One thing you could do with MoE is giving each expert different subsets of the input tokens. And that would definitely do what they claim here: it would allow search. You want to find where someone said "the password is X" in a 50 hour audio file, this would be perfect. If your question is "what is the first AND last thing person X said" ... it's going to suck badly. Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work.

declaredapple 3y ago
> One thing you could do with MoE is giving each expert different subsets of the input tokens. Don't MoE's route tokens to experts after the attention step? That wouldn't solve the n^2 issue the attention step has. If you split the tokens before the attention step, that would mean those tokens would have no relationship to each other - it would be like inferring two prompts in parallel. That would defeat the point of a 10M context
deskamess 3y ago
Is MOE then basically divide and conquer? I have no deep knowledge of this so I assumed MOE was where each expert analyzed the problem in a different way and then there was some map-reduce like operation on the generated expert results. Kinda like random forest but for inference.
- declaredapple 3y ago
  > I assumed MOE was where each expert analyzed the problem in a different way Uh sorta but not like parent described at all. You have multiple "experts" and you have a routing layer(s) that decide which expert to send it to. Usually every token is sent to at least 2. You can't just send half the tokens to one expert and half to another. Also the "experts" are not "domain experts" - there is not a "programming expert" and an "essay expert".
spott 3y ago
> Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work. They kinda address that in the technical report[0]. On page 12 they show results from a "multiple needle in a haystack" evaluation. https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_...
- candiodari 3y ago
  I would perhaps add that it's worrying. Every youtube evaluation of this model in GCP AI Studio I've seen have commented on the constant hallucinations of this model.

declaredapple 3y ago

> One thing you could do with MoE is giving each expert different subsets of the input tokens. Don't MoE's route tokens to experts after the attention step? That wouldn't solve the n^2 issue the attention step has. If you split the tokens before the attention step, that would mean those tokens would have no relationship to each other - it would be like inferring two prompts in parallel. That would defeat the point of a 10M context

deskamess 3y ago

Is MOE then basically divide and conquer? I have no deep knowledge of this so I assumed MOE was where each expert analyzed the problem in a different way and then there was some map-reduce like operation on the generated expert results. Kinda like random forest but for inference.

declaredapple 3y ago
> I assumed MOE was where each expert analyzed the problem in a different way Uh sorta but not like parent described at all. You have multiple "experts" and you have a routing layer(s) that decide which expert to send it to. Usually every token is sent to at least 2. You can't just send half the tokens to one expert and half to another. Also the "experts" are not "domain experts" - there is not a "programming expert" and an "essay expert".

spott 3y ago

> Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work. They kinda address that in the technical report[0]. On page 12 they show results from a "multiple needle in a haystack" evaluation. https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_...

candiodari 3y ago
I would perhaps add that it's worrying. Every youtube evaluation of this model in GCP AI Studio I've seen have commented on the constant hallucinations of this model.