4 ms·
Given that the token space is large enough to waste on such "low quality" tokens, has there been work done to use a smaller token space in order for quantized m
by magicalhippo 1y ago
Given that the token space is large enough to waste on such "low quality" tokens, has there been work done to use a smaller token space in order for quantized models to perform better?
Just a silly thought that crossed my mind when I saw those "ad tokens".
- typpilol 1y agoIsn't that exactly what some of these models that have 30b params but only activate 3b at a time
- rvba 1y agoHumans also only use X% of their brains (the one needed for a specific task)
- koakuma-chan 1y agoDoes that mean I'm a mixture of experts.
- RLAIF 1y ago[dead]
- clarionbell 1y agoThat's mixture of experts pattern.
- magicalhippo 1y agoAs mentioned that would be more like mixture-of-experts, where you instead effectively just multiply the rows with some of the columns in your weight matrices. That said, I wrote my post after bed time and I'm now pretty sure I wasn't thinking straight.