3 ms·
I'm running the llama.cpp/gguf Q8 version, with 30 layers offloaded to the laptop's GPU (RTX 3070, 8G VRAM), and I get around 20-25 tokens/second. It really fe
by tarruda 3y ago
I'm running the llama.cpp/gguf Q8 version, with 30 layers offloaded to the laptop's GPU (RTX 3070, 8G VRAM), and I get around 20-25 tokens/second.
It really feels like I have one the the earlier versions of ChatGPT 3.5 installed on my computer.