2 ms·
The large models are actually MoE these days so they're usable on ordinary hardware with weights streaming from SSD, just very slow. You're nonethess right that
by zozbot234 7mo ago
The large models are actually MoE these days so they're usable on ordinary hardware with weights streaming from SSD, just very slow. You're nonethess right that it makes the cloud-based offering more popular, since you can use that for convenience after testing a few inferences locally.
- vlovich123 7mo ago“Usable but slow” is how you could run regardless of MoE or not, the model architecture has nothing to do with it. MoE might run N times faster than non MoE where N is the number of experts but that’s it.