3 ms·
It’s not instantly fast though. Context is probably ~20gb of VRAM at max context size. That’s gonna take some time to get from SSD no matter what. TtFT will ge
by jychang 1y ago
It’s not instantly fast though. Context is probably ~20gb of VRAM at max context size. That’s gonna take some time to get from SSD no matter what.
TtFT will get slower if you export kv cache to SSD.