3 ms·
You can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any d
by spockz 8mo ago
You can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any decent throughput.
- stavros 8mo agoAh interesting, I didn't realize MoE doesn't need to all run in the same place.