4 ms·
Is it that silly? You could run multiple 27B models in parallel.
by orsorna 6d ago
Is it that silly? You could run multiple 27B models in parallel.
- peri-cl 6d agoYou actually don't need more RAM to batch multiple inference tasks of the same model. (Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
- orsorna 6d agoYou definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.