2 ms·
Kind of interesting, really what experts do is sort/organise weights into categories that are optimal to work together. Seems like a lot of research could be do
by jeffybefffy519 3mo ago
Kind of interesting, really what experts do is sort/organise weights into categories that are optimal to work together. Seems like a lot of research could be done to extend this concept to group weights together for common inputs ahead of time to achieve the same purpose.
- gitpusher42 3mo agoYeah, I tried both rearranging experts on disk and predicting the next expert using statistical approach. Reordering helped on the test prompt, but failed on another prompt. Markov and cross layer prediction didn't work either