6 ms·
On my system, using `-ngl 22` (running 22 layers on the GPU) cuts wall clock time by ~60%. My system: GPU: NVidia RTX 2070S (8GB VRAM) CPU: AMD Ryzen 5 3600
by rahimnathwani 3y ago
On my system, using `-ngl 22` (running 22 layers on the GPU) cuts wall clock time by ~60%.
My system:
GPU: NVidia RTX 2070S (8GB VRAM)
CPU: AMD Ryzen 5 3600 (16GB VRAM)
Here's the performance difference I see:
CPU only (./main -t 12)
llama_print_timings: load time = 15459.43 ms
llama_print_timings: sample time = 23.64 ms / 38 runs ( 0.62 ms per token)
llama_print_timings: prompt eval time = 9338.10 ms / 356 tokens ( 26.23 ms per token)
llama_print_timings: eval time = 31700.73 ms / 37 runs ( 856.78 ms per token)
llama_print_timings: total time = 47192.68 ms
GPU (./main -t 12 -ngl 22)
llama_print_timings: load time = 10285.15 ms
llama_print_timings: sample time = 21.60 ms / 35 runs ( 0.62 ms per token)
llama_print_timings: prompt eval time = 3889.65 ms / 356 tokens ( 10.93 ms per token)
llama_print_timings: eval time = 8126.90 ms / 34 runs ( 239.03 ms per token)
llama_print_timings: total time = 18441.22 ms
- rain1 3y agoThat is a crazy speedup!!
- GordonS 3y agoIs it really? Going from CPU to GPU, I would have expected a much better improvement.
- qwertox 3y agoI feel the same. For example some stats from Whisper [0] (audio transcoding, 30 seconds) show the following for the medium model (see other models in the link): --- GPU medium fp32 Linear 1.7s CPU medium fp32 nn.Linear 60.7s CPU medium qint8 (quant) nn.Linear 23.1s --- So the same model runs 35.7 times faster on GPU, and compared to an "optimized" model still 13.6. I was expecting around an order or magnitude of improvement. Then again, I do not know if in the case of this article the entire model was in the GPU, or just a fraction of it (22 layers) and the remainder on CPU, which might explain the result. Apparently that's the case, but I don't know much about this stuff. [0] https://github.com/MiscellaneousStuff/openai-whisper-cpu https://github.com/MiscellaneousStuff/openai-whisper-cpu
- rahimnathwani 3y agoYou last paragraph is correct. Only about half the model was running on the GPU.
- pavelstoev 3y agoTraining and inference on GPUs significantly underutilize …the GPUs. So tuning and various tricks need to be applied to achieve dramatic performance gains. If I am not good at cooking, giving me a larger kitchen will not make me faster or better.
- rahimnathwani 3y agoYou can think of it this way: if half the model is running on the GPU, and the GPU is infinitely fast, then the total calculation time would go down by 50%, compared with everything running on the CPU.
- ethbr0 3y agoRef Amdahl's Law: https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law
- quickthrower2 3y agoWow seems like common sense turned into a law. Maybe I can get a law :-). Anyone who has commuted on public transport probably knows this intuitively. (Using a kick scooter instead of walking cut my travel time by a good 5% which was excellent, as I still needed to be on a bus where that made no difference.)
- ethbr0 3y agoIt's a little more formalized and generalized observation: https://en.m.wikipedia.org/wiki/Amdahl%27s_law#Derivation https://en.m.wikipedia.org/wiki/Amdahl%27s_law#Derivation But yeah! There's a lot of common sense that, with some mathematical formalization, yields useful and extensible laws. Amdahl's has been robust because it opined on the 60s supercomputer coprocessor architectures, and then in more recent decades with consumer multicore chips. Laws aren't famous because they're insightful -- they're famous because they're useful.
- mejutoco 3y agoThese laws make more sense if one reads them as "heuristics", which is what they are.
- PaulWaldman 3y agoAny way to know the differences in power consumption?
- Tuna-Fish 3y agoProbably significant savings.
- guardiangod 3y agoI am testing it on an AWS instance and the speedup effect is not as consistent as I hope. The speedup varies between runs. Intel Xeon Platinum 8259CL CPU @ 2.50GHz 128 GB RAM Tesla T4 ./main -t 12 -m models/gpt4-alpaca-lora-30B-4bit-GGML/gpt4-alpaca-lora-30b.ggml.q5_0.bin llama_print_timings: load time = 3725.08 ms llama_print_timings: sample time = 612.06 ms / 536 runs ( 1.14 ms per token) llama_print_timings: prompt eval time = 13876.81 ms / 259 tokens ( 53.58 ms per token) llama_print_timings: eval time = 221647.40 ms / 534 runs ( 415.07 ms per token) llama_print_timings: total time = 239423.46 ms ./main -t 12 -m models/gpt4-alpaca-lora-30B-4bit-GGML/gpt4-alpaca-lora-30b.ggml.q5_0.bin -ngl 30 llama_print_timings: load time = 7638.95 ms llama_print_timings: sample time = 280.81 ms / 294 runs ( 0.96 ms per token) llama_print_timings: prompt eval time = 2197.82 ms / 2 tokens ( 1098.91 ms per token) llama_print_timings: eval time = 112790.25 ms / 293 runs ( 384.95 ms per token) llama_print_timings: total time = 120788.82 ms
- rahimnathwani 3y agoThanks. BTW: - the model I used was gpt4-x-vicuna-13B.ggml.q5_1.bin - I used 'time' to measure the wall clock time of each command. - My prompt was: Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Write a long blog post with 5 sections, about the pros and cons of emphasising procedural fluency over conceptual understanding, in high school math education. ### Response:
- samstave 3y agoMay you please ELI5 what is happening here... Imagine I am first ever hearing about this, ;; what did you do?
- rahimnathwani 3y ago0. Have a PC with an NVidia GPU, running Ubuntu, with the NVidia drivers and CUDA Toolkit already set up. 1. Download the weights for the model you want to use, e.g. gpt4-x-vicuna-13B.ggml.q5_1.bin 2. Clone the llama.cpp repo, and use 'make LLAMA_CUBLAS=1' to compile it with support for CUBLAS (BLAS on GPU). 3. Run the resulting 'main' executable, with the -ngl option set to 18, so that it tries to load 18 layers of the model into the GPU's VRAM, instead of the system's RAM.
- rain1 3y ago> 1. Download the weights for the model you want to use, e.g. gpt4-x-vicuna-13B.ggml.q5_1.bin I think you need to quantize the model yourself from the float/huggingface versions. My understanding is that the quantization formats have changed recently. and old quantized models no longer work.
- rahimnathwani 3y agoThat was true until 2 days ago :) The repo has now been updated with requantized models that work with the latest version, so you don't need to do that any more. https://huggingface.co/TheBloke/gpt4-x-vicuna-13B-GGML/commit/b4b5f7e523f35306412d10ea9c4922b6f5923719 https://huggingface.co/TheBloke/gpt4-x-vicuna-13B-GGML/commi...
- rain1 3y agowonderful! thank you
- samstave 3y agoCertainly! Here are the detailed steps for the tasks you mentioned: 0. Make sure you have a PC with an NVidia GPU running Ubuntu, and you have already set up the NVidia drivers and CUDA Toolkit. 1. Download the weights for the model you want to use, such as "gpt4-x-vicuna-13B.ggml.q5_1.bin". You can use a web browser or a command-line tool like `wget` to download the file. For example: ```bash wget https://example.com/path/to/gpt4-x-vicuna-13B.ggml.q5_1.bin https://example.com/path/to/gpt4-x-vicuna-13B.ggml.q5_1.bin ``` 2. Clone the llama.cpp repository using `git`. Make sure you have `git` installed on your system. Open a terminal and navigate to the directory where you want to clone the repository, then execute the following command: ```bash git clone https://github.com/example/llama.cpp.git https://github.com/example/llama.cpp.git ``` 3. Navigate into the cloned `llama.cpp` directory: ```bash cd llama.cpp ``` 4. Compile the code with support for CUBLAS (BLAS on GPU) by running the following command: ```bash make LLAMA_CUBLAS=1 ``` 5. After the compilation is successful, you can run the resulting `main` executable with the `-ngl` option set to 18. This will attempt to load 18 layers of the model into the GPU's VRAM instead of the system's RAM. Use the following command to run the executable: ```bash ./main -ngl 18 ``` Make sure to replace the example URLs and filenames with the actual ones you are using. Please note that the steps provided are based on the information you provided, and it assumes you have the necessary dependencies and permissions to perform these actions. If you encounter any issues or errors during the process, please provide more details, and I'll be happy to assist you further. - Certainly! Here are the detailed steps for the tasks you mentioned in a table format: | Step | Description | |------|-----------------------------------------------------------------------------------------------------------------------| | 0 | Make sure you have a PC with an NVidia GPU running Ubuntu, and you have already set up the NVidia drivers and CUDA Toolkit. | | 1 | Download the weights for the model you want to use, such as "gpt4-x-vicuna-13B.ggml.q5_1.bin". | | 2 | Clone the llama.cpp repository using `git`. | | 3 | Compile the code with support for CUBLAS (BLAS on GPU) using `make LLAMA_CUBLAS=1`. | | 4 | Run the resulting `main` executable with the `-ngl` option set to 18. | Here's how the commands can be represented in a table format: | Step | Command | |------|---------------------------------------------------------------------------------------------| | 1 | `wget https://example.com/path/to/gpt4-x-vicuna-13B.ggml.q5_1.bin https://example.com/path/to/gpt4-x-vicuna-13B.ggml.q5_1.bin` | | 2 | `git clone https://github.com/example/llama.cpp.git https://github.com/example/llama.cpp.git` | | 3 | `make LLAMA_CUBLAS=1` | | 4 | `./main -ngl 18` | Make sure to replace the example URLs and filenames with the actual ones you are using. Please note that the steps provided are based on the information you provided, and it assumes you have the necessary dependencies and permissions to perform these actions. If you encounter any issues or errors during the process, please provide more details, and I'll be happy to assist you further.
- eightysixfour 3y agoYou will likely see a bit of a performance gain dropping your threads to 6. I'm on a 3700x and get a regression when using 16 threads instead of the real 8 cores.
- rahimnathwani 3y agoHa! You're right. Honestly, I couldn't recall how many cores I had, and I figured it was faster to run `cat /proc/cpuinfo |grep processor| wc -l` than to google the specs. It seems like using the right # cores saves ~25%. and then using the GPU for 26 layers saves another ~43%. Timings for comparison with `--temp 0`: -t 6 : 4m10.458s -t 12 : 5m34.772s -t 6 -ngl 26 : 2m23.278s
- shaklee3 3y agoJust type lscpu
- rahimnathwani 3y agoYes, that's much clearer. The output includes: Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1
- deleted 3y ago[deleted]
- cpill 3y agoWill this work with the leaked models or Alpaca?
- MPSimmons 3y agoYes
- xrd 3y agoI'm experimenting with these settings. On my machine (with a 6GB NVidia GEForce, 32 GB of RAM and Intel i5-10400 CPU @ 2.90GHz) I get worse performance offloading to the GPU. 276.23 ms per token without GPU, using only -t 12 296.04 ms per token with GPU using -t 12 -ngl 22 How do you determine the right fit for -t and -ngl? I definitely compiled using make LLAMA_CUBLAS=1 I'm using models/ggml-vic7b-uncensored-q5_1.bin, does the model matter significantly?
- rahimnathwani 3y ago> How do you determine the right fit for -t and -ngl? t: the number of physical cores on your CPU ngl: just try increasing by one or two, until you start seeing inference crash due to 'out of memory' errors > i5-10400 You have 6 cores, so try `-t 6`
- xrd 3y agoThank you. I used 6 and that dropped the token time to 220ms. For -ngl, I tried using 24, and then 30 and then 40, and never got to an out of memory error, and got exactly the same token timing, stuck at 220ms. But, this is very helpful, thank you!
- rahimnathwani 3y agoI'm curious whether there's any difference if you try with a longer prompt or ask for a longer completion: https://news.ycombinator.com/item?id=35940365 https://news.ycombinator.com/item?id=35940365 Also curious to know whether the wall clock time (just prepend your command with 'time ') is any different.