2 ms·
You actually don't need more RAM to batch multiple inference tasks of the same model. (Each task needs its own context, but the (e.g.) 27B of constant paramete
by peri-cl 12d ago
You actually don't need more RAM to batch multiple inference tasks of the same model.
(Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
- orsorna 12d agoYou definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.