2 ms·
MoE models will have far more world knowledge than dense models with the same amount of active parameters. MoE is a no-brainer if your inference setup is ultim
by zozbot234 6mo ago
MoE models will have far more world knowledge than dense models with the same amount of active parameters. MoE is a no-brainer if your inference setup is ultimately limited by compute or memory throughput - not total memory footprint - or alternately if it has fast, high-bandwidth access to lower-tier storage to fetch cold model weights from on demand.
- regularfry 6mo agoYes, this. I can run the 122B Qwen3.5 MoE usably on one 4090 + 64GB RAM. That's a monster of a model, comparatively speaking.
- aitchnyu 6mo agoTangential. I'm a newb, can you name the concept of partitioning weights so we dont need to load whole thing?
- agunapal 6mo agoDo you mean model sharding?