4 ms·
Just added support for Ollama and LM Studio servers! I'm getting 66 tokens/sec and 0.17s time to first token for Llama 3.1, it's pretty mind-blowing. https://x
by Charlieholtz 2y ago
Just added support for Ollama and LM Studio servers! I'm getting 66 tokens/sec and 0.17s time to first token for Llama 3.1, it's pretty mind-blowing.
https://x.com/charliebholtz/status/1873798821526069258 https://x.com/charliebholtz/status/1873798821526069258