3 ms·
> It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. How are small isolated language models more similar to tha
by stingraycharles 3mo ago
> It's actually how organic brains work - specialized tasks are offloaded to local cortical columns.
How are small isolated language models more similar to that than MoE in LLMs?
- simianwords 3mo agoRight MoE is a tradeoff between efficiency and intelligence.
- roadside_picnic 3mo agoMoEs don't route like most people imagine. They aren't learning topic based experts despite the name The original Mixtral paper [0] (in the "Routing analysis" section) found: "surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic" A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing). 0. https://arxiv.org/pdf/2401.04088 https://arxiv.org/pdf/2401.04088