4 ms·
The model being tested is 18k as configured. I didn't expect this to make the 5090 to look like a good deal.
by ApolloFortyNine 13d ago
The model being tested is 18k as configured.
I didn't expect this to make the 5090 to look like a good deal.
- nacs 13d ago5090 has 32GB VRAM. It'd be silly to buy the 18k model to run a tiny model like Qwen 27B. You use models like GLM Flash and Qwen Next which won't fit on a single 5090.
- orsorna 13d agoIs it that silly? You could run multiple 27B models in parallel.
- peri-cl 13d agoYou actually don't need more RAM to batch multiple inference tasks of the same model. (Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
- orsorna 13d agoYou definitely need more RAM if you are not satisfied with small context windows, especially if the weights take a large % of the total memory to boot.
- deleted 13d ago[deleted]
- asimovDev 13d agocan run multiple subagents of Qwen 27B though, right? Unless I am fundamentally misunderstanding how VRAM constraints work
- Eisenstein 13d agoYou might be. Running another agent doesn't load a set of new weights. It creates a new KV cache for the agent and adds the prompts to the queue. Its just another inference turn.