One nice thing about Ollama vs. stock llama.cpp is Ollama supports both ggml and gguf models. If you've still got a lot of old ggml bins around you can easily c

One nice thing about Ollama vs. stock llama.cpp is Ollama supports both ggml and gguf models. If you've still got a lot of old ggml bins around you can easily create a model file and use them. I haven't done benchmarking vs. vLLM, but it's quite fast; in my tests on an A100-80g w/ llama2 70b I was getting over 25 tok/sec which is just mind blowing. I was even getting around 30 tok/sec on llama2 7b on an old RTX 1070, which is equally crazy.