I'm running vicuna on a machine with a 6GB Nvidia card using llama.cpp. I get 300ms per token which is totally usable. I tried to tune it using the ngl switch b

I'm running vicuna on a machine with a 6GB Nvidia card using llama.cpp. I get 300ms per token which is totally usable. I tried to tune it using the ngl switch but never got much better than that and that was fine for interacting with it. I'm blown away at the performance increases on consumer hardware in the last few months.