2 ms·

I have no problem running two or three sequences of qwen 27B with a 3090. It's basically the recommended way, LLM inference without batching is super inefficien

by redox99 2mo ago

I have no problem running two or three sequences of qwen 27B with a 3090. It's basically the recommended way, LLM inference without batching is super inefficient.