4 ms·
UPDATE: I was using the llama.cpp CPU backend and was still getting gibberish. On Google colab they're running with CUDA. I turned Claude loose on the problem a
by UncleOxidant 7mo ago
UPDATE: I was using the llama.cpp CPU backend and was still getting gibberish. On Google colab they're running with CUDA. I turned Claude loose on the problem and it discovered a problem in the llama.cpp CPU backend code where a float was being converted to an int and basically going to 0. Now it runs fine locally with the CPU backend.
- gorgonical 7mo agoMind sharing the fix as a patch? I would like to run it this way, too.
- UncleOxidant 7mo agohttps://github.com/philtomson/llama.cpp https://github.com/philtomson/llama.cpp
- gorgonical 7mo agoWorks like a charm for me, just like for you. Getting ~8 tok/s on my middle-of-the-road workstation laptop. Thanks for this!