3 ms·
This is the state of the art for such a setup. Really good performance! https://github.com/kvcache-ai/ktransformers https://github.com/kvcache-ai/ktransformers
by phonon 1y ago
This is the state of the art for such a setup. Really good performance!
https://github.com/kvcache-ai/ktransformers https://github.com/kvcache-ai/ktransformers