3 ms·
I am testing it on an AWS instance and the speedup effect is not as consistent as I hope. The speedup varies between runs. Intel Xeon Platinum 8259CL CPU @ 2.5
by guardiangod 3y ago
I am testing it on an AWS instance and the speedup effect is not as consistent as I hope. The speedup varies between runs.
Intel Xeon Platinum 8259CL CPU @ 2.50GHz
128 GB RAM
Tesla T4
./main -t 12 -m models/gpt4-alpaca-lora-30B-4bit-GGML/gpt4-alpaca-lora-30b.ggml.q5_0.bin
llama_print_timings: load time = 3725.08 ms
llama_print_timings: sample time = 612.06 ms / 536 runs ( 1.14 ms per token)
llama_print_timings: prompt eval time = 13876.81 ms / 259 tokens ( 53.58 ms per token)
llama_print_timings: eval time = 221647.40 ms / 534 runs ( 415.07 ms per token)
llama_print_timings: total time = 239423.46 ms
./main -t 12 -m models/gpt4-alpaca-lora-30B-4bit-GGML/gpt4-alpaca-lora-30b.ggml.q5_0.bin -ngl 30
llama_print_timings: load time = 7638.95 ms
llama_print_timings: sample time = 280.81 ms / 294 runs ( 0.96 ms per token)
llama_print_timings: prompt eval time = 2197.82 ms / 2 tokens ( 1098.91 ms per token)
llama_print_timings: eval time = 112790.25 ms / 293 runs ( 384.95 ms per token)
llama_print_timings: total time = 120788.82 ms
- rahimnathwani 3y agoThanks. BTW: - the model I used was gpt4-x-vicuna-13B.ggml.q5_1.bin - I used 'time' to measure the wall clock time of each command. - My prompt was: Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Write a long blog post with 5 sections, about the pros and cons of emphasising procedural fluency over conceptual understanding, in high school math education. ### Response: