3 ms·

That much good RAM in itself isn't super expensive. So does the rest of the hardware have to be particularly powerful?

• esperent 3y ago
  It has to be GPU RAM from my understanding, unless you're happy to wait several minutes/hours for each response.
• GaggiX 3y ago
  From what I can find online LLAMA-65B 4-bit quantized can run 1 token/s on a Ryzen 7 3700X (using llama.cpp).