2 ms·
I just tried out the 4-bit version of that model, asking it to write part of a standup comedy routine. It generated some decent ideas, and the speed isn't terri
by rahimnathwani 3y ago
I just tried out the 4-bit version of that model, asking it to write part of a standup comedy routine. It generated some decent ideas, and the speed isn't terrible.
(~3 tokens/second with 20 layers running on the GPU)
I'm using oobabooga in chat-instruct mode. Here's the last few lines of the server log.
llama_print_timings: load time = 5310.13 ms
llama_print_timings: sample time = 140.31 ms / 200 runs ( 0.70 ms per token)
llama_print_timings: prompt eval time = 1775.70 ms / 21 tokens ( 84.56 ms per token)
llama_print_timings: eval time = 47175.08 ms / 199 runs ( 237.06 ms per token)
llama_print_timings: total time = 61175.83 ms
Output generated in 61.48 seconds (3.25 tokens/s, 200 tokens, context 1346, seed 1510260097)
llama_tokenize: too many tokens
llama_tokenize: too many tokens
llama_tokenize: too many tokens
Llama.generate: prefix-match hit
llama_print_timings: load time = 5310.13 ms
llama_print_timings: sample time = 140.16 ms / 200 runs ( 0.70 ms per token)
llama_print_timings: prompt eval time = 19554.31 ms / 1740 tokens ( 11.24 ms per token)
llama_print_timings: eval time = 49292.75 ms / 199 runs ( 247.70 ms per token)
llama_print_timings: total time = 81038.76 ms
Output generated in 81.36 seconds (2.46 tokens/s, 200 tokens, context 1775, seed 1751805530)
Llama.generate: prefix-match hit
llama_print_timings: load time = 5310.13 ms
llama_print_timings: sample time = 140.75 ms / 200 runs ( 0.70 ms per token)
llama_print_timings: prompt eval time = 9141.41 ms / 709 tokens ( 12.89 ms per token)
llama_print_timings: eval time = 49406.39 ms / 199 runs ( 248.27 ms per token)
llama_print_timings: total time = 70887.22 ms
Output generated in 71.18 seconds (2.81 tokens/s, 200 tokens, context 1721, seed 1210059184)