4 ms·

Can you elaborate on scary smart and fast? It's been a month or two since I've tried but the results were depressingly slow and useless for more or less every

by andai 3y ago

Can you elaborate on scary smart and fast? It's been a month or two since I've tried but the results were depressingly slow and useless for more or less every task I tried. Every time a model is claimed to be "90% of GPT-3" I get excited and every time it's very disappointing. (On that note, after using GPT-4, GPT-3 now seems disappointing almost every time I interact with it.)

whimsicalism 3y ago
i think the falcon instruct is considered pretty good but if you are expectation set by gpt4 it still will not compare
PostOnce 3y ago
Different quantizations can give you a big speedup if you've had "depressingly slow" issues. Even the slowest ones (that fit in RAM) will run at basically interactive speed, not instant, but also not "email speed". I have a laptop with a 2018 CPU and I'm working with them just fine. Text generation style instead of chat style is another avenue that makes the feedback time not so annoying for a developer. at 100ms/token, it's faster than most people type, I think. That's what you might get on an old laptop with a 7B model. There's a useful leaderboard here to help you pick a model: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... It really depends on your task, lots and lots of natural language type tasks give great results, the models seem to have extensive knowledge of many fields. So for some kinds of Q&A bot (technical or not), for copy blurbs, for fiction, game NPCs, etc, the models (especially 13B and up) can be breathtaking, even moreso considering they run on bottom-dollar consumer hardware (I paid $250 for the laptop I'm developing on). There are of course some things that neither the local LLMs nor GPT4 can do, like create useful OpenSCAD models :) Things keep getting better, newer quantization methods give you more smarts in the same amount of RAM at basically the same speed -- the models are getting better, there are more permissively licensed ones now.
- RossBencina 3y ago
  How are you running inference? GPU or CPU? I'm trying to use GPT4All (ggml-based) on 32 cores of E5-v3 hardware and even the 4GB models are depressingly slow as far as I'm concerned (i.e. slower than the GPT4 API, which is barely usable for interactive work). I'd be much obliged if you could point me at a specific quantized model on HF that you think is "fast" and I'll download it and try it out.
- wokwokwok 3y ago
  Whaaaaat, how are you getting 100ms per token on an 5 year old potato without a graphics card? Like, not vaguely hand wavey stuff, specifically, what model and what inference code? I get nothing like that performance for the 7B models, forget the larger models, using llama.cpp on a pc without an nvidia GPU.
  - PostOnce 3y ago
    Here is a short test of a 7B 4bit model on an intel 8350U laptop with no AMD/Nvidia GPU. On that laptop CPU from 2017, using a copy of llama.cpp I compiled 2 days ago (just "make", no special options, no BLAS, etc): ./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "A short test for Hacker News:" llama_print_timings: sample time = 19.12 ms / 36 runs ( 0.53 ms per token, 1882.65 tokens per second) llama_print_timings: prompt eval time = 886.82 ms / 9 tokens ( 98.54 ms per token, 10.15 tokens per second) llama_print_timings: eval time = 5507.31 ms / 35 runs ( 157.35 ms per token, 6.36 tokens per second) and a second run: ./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "Sherlock Holmes favorite dinner was " llama_print_timings: sample time = 54.37 ms / 102 runs ( 0.53 ms per token, 1875.93 tokens per second) llama_print_timings: prompt eval time = 876.94 ms / 9 tokens ( 97.44 ms per token, 10.26 tokens per second) llama_print_timings: eval time = 16057.95 ms / 101 runs ( 158.99 ms per token, 6.29 tokens per second) at 158ms per token, if we guess a word is 2.5 tokens, then that's 151 words per minute, much faster than most people can type. On a $250 laptop. Isn't the future neat? the code I was running: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and the model: https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML There are other models that may perform better, I'm going to be doing a lot of screwing around with OpenLLaMA this weekend.
  - Rastonbury 3y ago
    This is a exactly a case in point why people decide to pay OpenAI instead of rolling their own. I'm non-technical but have setup an image gen app based custom SD model using diffusers, so not entirely clueless. But for LLM I have no where idea where to start quickly. Finding a model on a leaderboard, download and setup then customising it and benchmarking is way too much time for me, I'll just pay for GPT4 if ever need to instead of chasing and troubleshooting to get some magical result. It'll be easier in the future I'm sure when an open model merges as the SD1.5 of LLM
ApolloFortyNine 3y ago
Save for coding they've been pretty good in my experience. There's definitely some prompt magic openai does behind the scenes that helps beat the raw style local llms usually go for. With proper prompting you can get chatgpt like answers.
lhl 3y ago
In terms of speed, we're talking about 140t/s for 7B models, and 40t/s for 33B models on a 3090/4090 now.[1] (1 token ~= 0.75 word) It's quite zippy. llama.cpp performs close on Nvidia GPUs now (but they don't have a handy chart) and you can get decent performance on 13B models on M1/M2 Macs. You can take a look at a list of evals here: https://llm-tracker.info/books/evals/page/list-of-evals https://llm-tracker.info/books/evals/page/list-of-evals - for general usage, I think home-rolled evals like llm-jeopardy [2] and local-llm-comparison [3] by hobbyists are more useful than most of the benchmark rankings. That being said, personally I mostly use GPT-4 for code assistance to that's what I'm most interested in, and the latest code assistants are scoring quite well: https://github.com/abacaj/code-eval https://github.com/abacaj/code-eval - a recent replit-3b fine tune the human-eval results for open models (as a point of reference, GPT-3.5 gets 60.4 on pass@1 and 68.9 on pass@10 [4]) - I've only just started playing around with it since replit model tooling is not as good as llamas (doc here: https://llm-tracker.info/books/howto-guides/page/replit-models https://llm-tracker.info/books/howto-guides/page/replit-mode...). I'm interested in potentially applying reflexion or some of the other techniques that have been tried to even further increase coding abilities. (InterCode in particular has caught my eye https://intercode-benchmark.github.io/ https://intercode-benchmark.github.io/) [1] https://github.com/turboderp/exllama#results-so-far https://github.com/turboderp/exllama#results-so-far [2] https://github.com/aigoopy/llm-jeopardy https://github.com/aigoopy/llm-jeopardy [3] https://github.com/Troyanovsky/Local-LLM-comparison/tree/main https://github.com/Troyanovsky/Local-LLM-comparison/tree/mai... [4] https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder
- ignoramous 3y ago
  > https://github.com/turboderp/exllama https://github.com/turboderp/exllama Is exllama an alternative to llama.cpp?
  - juliensalinas 3y ago
    llama.cpp focuses on optimizing inference on a CPU, while exllama is for inference on a GPU.

by andai 3y ago

whimsicalism 3y ago

i think the falcon instruct is considered pretty good but if you are expectation set by gpt4 it still will not compare

PostOnce 3y ago

Different quantizations can give you a big speedup if you've had "depressingly slow" issues. Even the slowest ones (that fit in RAM) will run at basically interactive speed, not instant, but also not "email speed". I have a laptop with a 2018 CPU and I'm working with them just fine. Text generation style instead of chat style is another avenue that makes the feedback time not so annoying for a developer. at 100ms/token, it's faster than most people type, I think. That's what you might get on an old laptop with a 7B model. There's a useful leaderboard here to help you pick a model: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... It really depends on your task, lots and lots of natural language type tasks give great results, the models seem to have extensive knowledge of many fields. So for some kinds of Q&A bot (technical or not), for copy blurbs, for fiction, game NPCs, etc, the models (especially 13B and up) can be breathtaking, even moreso considering they run on bottom-dollar consumer hardware (I paid $250 for the laptop I'm developing on). There are of course some things that neither the local LLMs nor GPT4 can do, like create useful OpenSCAD models :) Things keep getting better, newer quantization methods give you more smarts in the same amount of RAM at basically the same speed -- the models are getting better, there are more permissively licensed ones now.

RossBencina 3y ago
How are you running inference? GPU or CPU? I'm trying to use GPT4All (ggml-based) on 32 cores of E5-v3 hardware and even the 4GB models are depressingly slow as far as I'm concerned (i.e. slower than the GPT4 API, which is barely usable for interactive work). I'd be much obliged if you could point me at a specific quantized model on HF that you think is "fast" and I'll download it and try it out.
wokwokwok 3y ago
Whaaaaat, how are you getting 100ms per token on an 5 year old potato without a graphics card? Like, not vaguely hand wavey stuff, specifically, what model and what inference code? I get nothing like that performance for the 7B models, forget the larger models, using llama.cpp on a pc without an nvidia GPU.
- PostOnce 3y ago
  Here is a short test of a 7B 4bit model on an intel 8350U laptop with no AMD/Nvidia GPU. On that laptop CPU from 2017, using a copy of llama.cpp I compiled 2 days ago (just "make", no special options, no BLAS, etc): ./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "A short test for Hacker News:" llama_print_timings: sample time = 19.12 ms / 36 runs ( 0.53 ms per token, 1882.65 tokens per second) llama_print_timings: prompt eval time = 886.82 ms / 9 tokens ( 98.54 ms per token, 10.15 tokens per second) llama_print_timings: eval time = 5507.31 ms / 35 runs ( 157.35 ms per token, 6.36 tokens per second) and a second run: ./main -m models/WizardLM-7B-uncensored.ggmlv3.q4_0.bin -n 128 -s 99 -p "Sherlock Holmes favorite dinner was " llama_print_timings: sample time = 54.37 ms / 102 runs ( 0.53 ms per token, 1875.93 tokens per second) llama_print_timings: prompt eval time = 876.94 ms / 9 tokens ( 97.44 ms per token, 10.26 tokens per second) llama_print_timings: eval time = 16057.95 ms / 101 runs ( 158.99 ms per token, 6.29 tokens per second) at 158ms per token, if we guess a word is 2.5 tokens, then that's 151 words per minute, much faster than most people can type. On a $250 laptop. Isn't the future neat? the code I was running: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp and the model: https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML There are other models that may perform better, I'm going to be doing a lot of screwing around with OpenLLaMA this weekend.
- Rastonbury 3y ago
  This is a exactly a case in point why people decide to pay OpenAI instead of rolling their own. I'm non-technical but have setup an image gen app based custom SD model using diffusers, so not entirely clueless. But for LLM I have no where idea where to start quickly. Finding a model on a leaderboard, download and setup then customising it and benchmarking is way too much time for me, I'll just pay for GPT4 if ever need to instead of chasing and troubleshooting to get some magical result. It'll be easier in the future I'm sure when an open model merges as the SD1.5 of LLM

ApolloFortyNine 3y ago

Save for coding they've been pretty good in my experience. There's definitely some prompt magic openai does behind the scenes that helps beat the raw style local llms usually go for. With proper prompting you can get chatgpt like answers.

lhl 3y ago

In terms of speed, we're talking about 140t/s for 7B models, and 40t/s for 33B models on a 3090/4090 now.[1] (1 token ~= 0.75 word) It's quite zippy. llama.cpp performs close on Nvidia GPUs now (but they don't have a handy chart) and you can get decent performance on 13B models on M1/M2 Macs. You can take a look at a list of evals here: https://llm-tracker.info/books/evals/page/list-of-evals https://llm-tracker.info/books/evals/page/list-of-evals - for general usage, I think home-rolled evals like llm-jeopardy [2] and local-llm-comparison [3] by hobbyists are more useful than most of the benchmark rankings. That being said, personally I mostly use GPT-4 for code assistance to that's what I'm most interested in, and the latest code assistants are scoring quite well: https://github.com/abacaj/code-eval https://github.com/abacaj/code-eval - a recent replit-3b fine tune the human-eval results for open models (as a point of reference, GPT-3.5 gets 60.4 on pass@1 and 68.9 on pass@10 [4]) - I've only just started playing around with it since replit model tooling is not as good as llamas (doc here: https://llm-tracker.info/books/howto-guides/page/replit-models https://llm-tracker.info/books/howto-guides/page/replit-mode...). I'm interested in potentially applying reflexion or some of the other techniques that have been tried to even further increase coding abilities. (InterCode in particular has caught my eye https://intercode-benchmark.github.io/ https://intercode-benchmark.github.io/) [1] https://github.com/turboderp/exllama#results-so-far https://github.com/turboderp/exllama#results-so-far [2] https://github.com/aigoopy/llm-jeopardy https://github.com/aigoopy/llm-jeopardy [3] https://github.com/Troyanovsky/Local-LLM-comparison/tree/main https://github.com/Troyanovsky/Local-LLM-comparison/tree/mai... [4] https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder

ignoramous 3y ago
> https://github.com/turboderp/exllama https://github.com/turboderp/exllama Is exllama an alternative to llama.cpp?
- juliensalinas 3y ago
  llama.cpp focuses on optimizing inference on a CPU, while exllama is for inference on a GPU.