also isn't DeepSeek's Mixture of Experts? meaning not all params get ever activated on one forward pass? 70B feels like the best balance between usable locally

also isn't DeepSeek's Mixture of Experts? meaning not all params get ever activated on one forward pass? 70B feels like the best balance between usable locally and decent for regular use. maybe not SOTA, but a great first step.