4 ms·

do you know if they did this to it? https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/ https://research.google/blog/tur

• kgeist 6mo ago
  Llama.cpp already uses an idea from it internally for the KV cache [0] So a quantized KV cache now must see less degradation [0] https://github.com/ggml-org/llama.cpp/pull/21038 https://github.com/ggml-org/llama.cpp/pull/21038