2 ms·
Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
by puzzlingcaptcha 3mo ago
Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
- trollbridge 3mo agoI haven't, since the machine has almost no GPU to speak of. It's an old machine, although in theory it would make sense to pop an R9700 in there and load a bunch of models into RAM and swap them in and out.