4 ms·
Most people are vram constrained not compute constrained.
by msp26 2y ago
Most people are vram constrained not compute constrained.
- kaibee 2y agoCloud providers aren’t though.
- Manabu-eo 2y agoBut those people usually have more system RAM than VRAM. At those scales, most people become bandwidth and compute constrained using CPU inference instead of multiple GPUs. In those cases, an MOE with a low number of active parameters is the fastest.