3 ms·

So, llama.cpp already somewhat supports this: https://github.com/ggerganov/llama.cpp/issues/3440 https://github.com/ggerganov/llama.cpp/issues/3440