3 ms·
I'm experimenting with these settings. On my machine (with a 6GB NVidia GEForce, 32 GB of RAM and Intel i5-10400 CPU @ 2.90GHz) I get worse performance offloadi
by xrd 3y ago
I'm experimenting with these settings. On my machine (with a 6GB NVidia GEForce, 32 GB of RAM and Intel i5-10400 CPU @ 2.90GHz) I get worse performance offloading to the GPU.
276.23 ms per token without GPU, using only -t 12
296.04 ms per token with GPU using -t 12 -ngl 22
How do you determine the right fit for -t and -ngl? I definitely compiled using make LLAMA_CUBLAS=1
I'm using models/ggml-vic7b-uncensored-q5_1.bin, does the model matter significantly?
- rahimnathwani 3y ago> How do you determine the right fit for -t and -ngl? t: the number of physical cores on your CPU ngl: just try increasing by one or two, until you start seeing inference crash due to 'out of memory' errors > i5-10400 You have 6 cores, so try `-t 6`
- xrd 3y agoThank you. I used 6 and that dropped the token time to 220ms. For -ngl, I tried using 24, and then 30 and then 40, and never got to an out of memory error, and got exactly the same token timing, stuck at 220ms. But, this is very helpful, thank you!
- rahimnathwani 3y agoI'm curious whether there's any difference if you try with a longer prompt or ask for a longer completion: https://news.ycombinator.com/item?id=35940365 https://news.ycombinator.com/item?id=35940365 Also curious to know whether the wall clock time (just prepend your command with 'time ') is any different.