3 ms·
5090 has 32GB VRAM. It'd be silly to buy the 18k model to run a tiny model like Qwen 27B. You use models like GLM Flash and Qwen Next which won't fit on a sing
by nacs 5d ago
5090 has 32GB VRAM.
It'd be silly to buy the 18k model to run a tiny model like Qwen 27B. You use models like GLM Flash and Qwen Next which won't fit on a single 5090.
- orsorna 5d agoIs it that silly? You could run multiple 27B models in parallel.
- peri-cl 5d agoYou actually don't need more RAM to batch multiple inference tasks of the same model. (Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
- orsorna 5d agoYou definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.
- deleted 5d ago[deleted]
- asimovDev 5d agocan run multiple subagents of Qwen 27B though, right? Unless I am fundamentally misunderstanding how VRAM constraints work
- Eisenstein 5d agoYou might be. Running another agent doesn't load a set of new weights. It creates a new KV cache for the agent and adds the prompts to the queue. Its just another inference turn.