3 ms·
Yeah. My 3090 gets like ~5 tokens/s on 70B Q3KL. This is a good idea, as splitting up llms is actually pretty efficient with pipelined requests.
by brucethemoose2 3y ago
Yeah. My 3090 gets like ~5 tokens/s on 70B Q3KL.
This is a good idea, as splitting up llms is actually pretty efficient with pipelined requests.