3 ms·
Which GPU is capable of all that? And which do you have a write up for this?
by starik36 3y ago
Which GPU is capable of all that? And which do you have a write up for this?
- ramesh31 3y ago> Which GPU is capable of all that? And which do you have a write up for this? Anything down to an RTX3090. You need 16GB VRAM to comfortably fit llama2-7b-chat with 8 bit quantization and a 2048 context length with llama.cpp. I'm getting completions (in real time) that are easily on par with ChatGPT on a VM with a single V100. People are even getting decent performance on Mac silicon with Metal.