3 ms·
Ollama replacement 2-4x faster for no extra compute cost
- deleted 26d ago[deleted]
- smokel 26d agoThis seems to be a wrapper around llama.cpp with several tuned parameter settings. Most of the speedup is gained by enabling speculative decoding [1]. No amazing breakthroughs, just good tuning! [1] https://en.wikipedia.org/wiki/Speculative_decoding https://en.wikipedia.org/wiki/Speculative_decoding
- wowitsbase 26d agoSpeculative decoding was on in all of my tests, the gain came from moving the round's inputs off host memory.
- dTal 26d agoI have never understood how it is that "Ollama" manages to stay some sort of de-facto standard local hosting solution when all it does is wrap llama.cpp poorly. It's not as if llama.cpp is hard to use - it offers pre-compiled single file binaries! It implements every extant endpoint format! It has a web interface! What more could you ask for? Yet everything from agentic frameworks to vibe coded chat apps offer configurations for a variety of proprietary cloud providers, plus "Ollama". Never "llama.cpp". Why?
- wowitsbase 26d agoagreed, if not for this project I've been making I would at least be using base llama.cpp Personally I think it comes down to simplicity, but there's no reason for it's performance drops compared to llama.cpp while it's a wrapper of it.