3 ms·
Drop in replacement for what exactly? Can I use it with llama.cpp and Vulkan? Or vLLM and ROCm?
by npodbielski 5mo ago
Drop in replacement for what exactly? Can I use it with llama.cpp and Vulkan?
Or vLLM and ROCm?
- pythongiant 5mo agoKVBoost is a drop-in replacement for AutoModelForCausalLM. Same API surface (KVBoost.from_pretrained(...), engine.generate(...)), but with cross-request KV reuse, FlashAttention-2, AWQ layer streaming, and speculative decoding bolted on.