3 ms·
Similar situation happened to me too "and 24GB RAM, this took 20 seconds to start up. After posing the question "What is the most common way of transportation i
by kgbcia 3y ago
Similar situation happened to me too "and 24GB RAM, this took 20 seconds to start up. After posing the question "What is the most common way of transportation in Amsterdam?", the Vicuna model began to generate its response word by word, taking *15 minutes* to complete the task"
Time is money and the openai is pennies
- xrd 3y agoI'm running vicuna on a machine with a 6GB Nvidia card using llama.cpp. I get 300ms per token which is totally usable. I tried to tune it using the ngl switch but never got much better than that and that was fine for interacting with it. I'm blown away at the performance increases on consumer hardware in the last few months.
- Proven 3y ago[dead]