3 ms·
Isn't that exactly what some of these models that have 30b params but only activate 3b at a time
by typpilol 1y ago
Isn't that exactly what some of these models that have 30b params but only activate 3b at a time
- rvba 1y agoHumans also only use X% of their brains (the one needed for a specific task)
- koakuma-chan 1y agoDoes that mean I'm a mixture of experts.
- RLAIF 1y ago[dead]
- clarionbell 1y agoThat's mixture of experts pattern.
- magicalhippo 1y agoAs mentioned that would be more like mixture-of-experts, where you instead effectively just multiply the rows with some of the columns in your weight matrices. That said, I wrote my post after bed time and I'm now pretty sure I wasn't thinking straight.