2 ms·
Just to add onto this point, you expect different experts to be activated for every token, so not having all of the weights in fast memory can still be quite sl
by bick_nyers 2y ago
Just to add onto this point, you expect different experts to be activated for every token, so not having all of the weights in fast memory can still be quite slow as you need to load/unload memory every token.
- valine 2y agoProbably better to be moving things from fast memory to faster memory than from slow disk to fast memory.