4 ms·
Nice, main.exe seems to work just fine with the 7B quantized model - generates a token every 400ms on an AMD Ryzen 5 2600! But, quantize.exe doesn't seem to wo
by GordonS 4y ago
Nice, main.exe seems to work just fine with the 7B quantized model - generates a token every 400ms on an AMD Ryzen 5 2600!
But, quantize.exe doesn't seem to work - any valid command (such as below) pauses for a split second, then returns with no output?
$ quantize.exe ggml-model-f16.bin ggml-model-q4_0.bin 2
- GordonS 4y agoIn case this helps anyone else: I built it myself on Windows with CMake, and then everything just works.
- starik36 4y agoDo you mind sharing the binaries?
- GordonS 4y agoSure! https://filetransfer.io/data-package/8hxKAiaH#link https://filetransfer.io/data-package/8hxKAiaH#link I wasn't sure where to upload them, and that link is only good for 50 downloads. Can put them somewhere else if you know a better location that doesn't require signup.
- starik36 4y agoThank you. llama.exe is basically main.exe? I actually learned how to compile this code via CMake/VS2019. It's sure a whole lot more complicated then it was 25 years ago when I was writing C.