4 ms·
> It generates the output slightly faster than reading speed For 33b? It should be much faster. What stack are you running? Llama.cpp and exLlama are SOTA as
by brucethemoose2 3y ago
> It generates the output slightly faster than reading speed
For 33b? It should be much faster.
What stack are you running? Llama.cpp and exLlama are SOTA as far as I know.