4 ms·
> Also, note that there's zero CUDA dependency. So does this mean I can run this on AMD? And on a consumer 9000 series card?
by d3Xt3r 6mo ago
> Also, note that there's zero CUDA dependency.
So does this mean I can run this on AMD? And on a consumer 9000 series card?
- randomgermanguy 6mo agoIf you found a rare 9000 card with 200+ GB of VRAM, sure
- HarHarVeryFunny 6mo agoIf you don't have the source code then it makes no difference. If you have the weights and are running some model via llama.cpp, then you are using whatever API llama.cpp is using, not the API that was used to train the model or that anyone else may be using to serve it.
- Eisenstein 6mo agoIf the card supports vulkan and the model has gguf weights. llamacpp has excellent vulkan support that is being actively developed and is not that far behind CUDA where speed is concerned. * https://github.com/ggml-org/llama.cpp/releases https://github.com/ggml-org/llama.cpp/releases