3 ms·

You can get llama CPP or kobold.cpp binaries and load a quantized model right into them on the CPU only, no need to install Python or have an Nvidia GPU.

• tormeh 3y ago
  Well, I'd like it to respond in something close to real-time, and since I have a pretty good non-Nvidia GPU, it makes more sense to wait for the WebGPU port.
  • programd 3y ago
    7 tokens per sec on an i5-11400 CPU using llama.cpp - that's pretty real time for personal use I would think.