3 ms·
llama.cpp is CPU only but llama runs on GPU using the HuggingFace Transformers library. You can run the llama model which far outpaces GPT-3.5 on a couple of $2
by MacsHeadroom 4y ago
llama.cpp is CPU only but llama runs on GPU using the HuggingFace Transformers library. You can run the llama model which far outpaces GPT-3.5 on a couple of $200 Tesla P40 GPUs at faster speeds than GPT-3.5 Turbo, completely locally.