3 ms·
Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp? I’ve sweeped concurrency across many models and many different kinds of har
by dannyw 2mo ago
Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?
I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.
- mhitza 2mo agoWhat do you use instead of llama.cpp? With vllm for example most models don't seem to be supported out of the box.