3 ms·
> 1. They don't talk about how they get to 10M token context > 2. They don't talk about how they get to 10M token context Yes. I wonder if they're using a "li
by cs702 3y ago
> 1. They don't talk about how they get to 10M token context
> 2. They don't talk about how they get to 10M token context
Yes. I wonder if they're using a "linear RNN" type of model like Linear Attention, Mamba, RWKV, etc.
Like Transformers with standard attention, these models train efficiently in parallel, but their compute is O(N) instead of O(N²), so in theory they can be extended to much longer sequences much efficiently. They have shown a lot of promise recently at smaller model sizes.
Does anyone here have any insight or knowledge about the internals of Gemini 1.5?
- sebzim4500 3y agoThe fact they are getting perfect recall with millions of tokens rules out any of the existing linear attention methods.
- cs702 3y agoI wouldn't be so sure perfect recall rules out linear RNNs, because I haven't seen any conclusive data on their ability to recall. Have you?
- candiodari 3y agoThey do give a hint: "This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture." One thing you could do with MoE is giving each expert different subsets of the input tokens. And that would definitely do what they claim here: it would allow search. You want to find where someone said "the password is X" in a 50 hour audio file, this would be perfect. If your question is "what is the first AND last thing person X said" ... it's going to suck badly. Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work.
- declaredapple 3y ago> One thing you could do with MoE is giving each expert different subsets of the input tokens. Don't MoE's route tokens to experts after the attention step? That wouldn't solve the n^2 issue the attention step has. If you split the tokens before the attention step, that would mean those tokens would have no relationship to each other - it would be like inferring two prompts in parallel. That would defeat the point of a 10M context
- deskamess 3y agoIs MOE then basically divide and conquer? I have no deep knowledge of this so I assumed MOE was where each expert analyzed the problem in a different way and then there was some map-reduce like operation on the generated expert results. Kinda like random forest but for inference.
- declaredapple 3y ago> I assumed MOE was where each expert analyzed the problem in a different way Uh sorta but not like parent described at all. You have multiple "experts" and you have a routing layer(s) that decide which expert to send it to. Usually every token is sent to at least 2. You can't just send half the tokens to one expert and half to another. Also the "experts" are not "domain experts" - there is not a "programming expert" and an "essay expert".
- spott 3y ago> Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work. They kinda address that in the technical report[0]. On page 12 they show results from a "multiple needle in a haystack" evaluation. https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_...
- candiodari 3y agoI would perhaps add that it's worrying. Every youtube evaluation of this model in GCP AI Studio I've seen have commented on the constant hallucinations of this model.
- deleted 3y ago[deleted]