23 ms·
LLaMA2 Chat 70B outperformed ChatGPT
- generalizations 3y ago* ChatGPT 3.5. But it's also within spitting distance of GPT4, which is very exciting.
- make3 3y agoperformance is logarithmic as a function of money invested in compute, so maybe it's close but it's also far away
- valine 3y agoGPT4 is more difficult to measure I think. The value I get from GPT4 is in the details it gets right on very obscure, complex questions. I'm not sure benchmarks are capturing how far GPT4 is ahead of other models. For simple stuff it's not that much better than 3.5.
- nmfisher 3y agoI haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?
- freedomben 3y agoChatGPT uses the GPT-4 API, so it's the same. With the API directly though you can change the system prompt, which can enable better results if you know what you're doing.
- bazmattaz 3y agoChatGPT uses the GPT-4, but there are conspiracy theories circling that ChatGPT is neutered and thus not as good as GPT-4 through the API. The theory being that OpenAI are thottling the free version of GPT-4 (ChatGPT)
- bestcoder69 3y agoFree ChatGPT runs 3.5. You have to upgrade to plus to use GPT-4. The APIs seem close to ChatGPT, but it’s a little opaque what they’re actually doing. If you inspect network requests the models are named something like “chat-render-3.5” instead of the API model names. I’d imagine OpenAI might run experiments on ChatGPT that they wouldn’t on the API, to avoid breaking 3rd party applications unannounced.
- cosmojg 3y agoThat's, uh, not a conspiracy theory. The free version of ChatGPT uses an entirely different model on the backend.
- zo1 3y agoI use the GPT4 API and still think it got neutered since I first started using it.
- Fergusonb 3y agoAnecdotal evidence here - I find that the API is less likely to ask questions about what you are doing and get straight to the answer. For example, if I were to ask how to do something with burp it will just answer instead of going into the "as an AI" monologue.
- bkanber 3y agoChatGPT is a wrapper to the GPT Completion API with some sane defaults. With a new beta feature you can edit the system prompt via ChatGPT, but you still can't adjust the other parameters you can reach with the API.
- knodi123 3y agoYeah, my experience has been that every one of these freely downloadable models can be measured as "percent of chatgpt quality". and getting up to 85% is shockingly good. *edit: oops, my brain inserted "by" in the middle of "outperformed chatgpt". I'll leave my wrong comment up as a testament to shame.
- fullshark 3y agoThe reality is probably some queries ChatGPT outperforms and vice versa. Regardless the premise that ChatGPT's secret sauce could be hidden forever is very dead.
- cj 3y agoIt looks like ChatGPT length is 827 while LLaMA2 length is more than double at 1790. Disclaimer from the site: > Caution: GPT-4 may favor models with longer outputs and/or those that were fine-tuned on GPT-4 outputs. > While AlpacaEval provides a useful comparison of model capabilities in following instructions, it is not a comprehensive or gold-standard evaluation of model abilities. For one, as detailed in the AlpacaFarm paper, the auto annotator winrates are correlated with length.
- cs702 3y agoAlso, Llama 2 is still a few percentage points below GPT-4. Which is not close, because performance is logarithmic in training compute. Each additional percentage point of performance requires exponentially greater investment in compute during pretraining. Llama 2 was pretrained on 2 trillion tokens -- a significant investment in compute, for sure, but still not enough to get close to GPT-4.
- freedomben 3y agoLlama2 beat ChatGPT 3.5 with a 92.66% win rate to 89.37%, but lost to GPT-4 which got 95.28%. Still pretty amazing though!
- cs702 3y agoNot really close, because performance is logarithmic in training compute. That is, each additional percentage point of performance requires exponentially greater investment in compute during pretraining. Llama 2 was pretrained on 2 trillion tokens -- a significant investment in compute, for sure, but still not enough to get close to GPT-4. And this is only one benchmark.
- accrual 3y agoDoes this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? I've used a 13B model with LLaMA1 and it's surprisingly good, but still nowhere near ChatGPT for coding questions.
- ramesh31 3y ago>Does this mean it may be possible to self-host a ChatGPT clone assuming you have a 70B model? Not only possible but quite easy. Inference for 70B can be done with llama.cpp using CPU only, on any commodity hardware with >64GB of RAM
- beefield 3y agoI have 64gb on my 5 year old thinkpad. What kind of performance (tokens per sec) I could expect on that nowadays for a 70B model?
- jerrygenser 3y agoLlama cpp speed is dramatically improved by avx instructions. If your CPU has those it would be much faster than not. And if it doesn't you need to do some workarounds with compiling and it gets a bit harder to run.
- Zambyte 3y agoWhen you say "coding questions" do you mean questions that should be answered by producing code, or questions about code ("explain this")? Or both?
- marcosdumay 3y agoI imagine that if you take the time to specialize it, you suddenly have a model that is better than anything from the large players on all the cases that you care about. But, well, I am currently not hyped enough about it to actually try.
- rvz 3y agoPossibly. Might need to be further optimized in size and 4-bit quantisation, perhaps and then you have a scaleable and fast self-hosted AI model. Lets just hope that there won’t be any embarrassing vulnerabilities coming out of this when someone could prompt the model to reveal its own environment variables or API keys or the internal prompt that it is using. But it seems the $0 free AI models are eating OpenAI’s lunch and Meta so far is winning the race to zero.
- seydor 3y agothis can apparently run on 48GB
- lolinder 3y agoWhen quantized to 4 bits, yes. You lose some quality by doing that, though, as compared to the full f16.
- evilduck 3y agoFrom what I've gathered when reading up on this topic, if RAM is your constraint the common thought has been that higher parameter models quantized down to smaller sizes will outperform lower parameter models running at higher quantization, i.e. it may still be preferable to use the 70B Llama model quantized to 4-bits than something like an unquantized f16 Falcon 40B or the "coming soon" f16 33B Llama2.
- lolinder 3y agoYes, that is true! But you lose enough performance that comparisons to GPT-3.5 stop working.
- treprinum 3y ago2xA6000 NVLinked Ampere can run 70B 8-bit which is almost as good as fp16. I bought another A6000 just for that.
- alecco 3y agoBetter evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closing the gap with 4. Source for GPT-3.5 and GPT-4.0 values (but mind it might not be the same # of shots) https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/30#64c08a5550f0d6578dc67069 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- lhl 3y agoIt depends on the eval, but I think it's fair to say that it's close. Here is the AGI Eval results organized into a table w/ averages (also I put in the new Hermes LLama2 13B model as well: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=711776706 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It beats out ChatGPT in every category except SAT-Math. We definitely need harder benchmarks. So far, there's BIG-Bench Hard https://github.com/suzgunmirac/BIG-Bench-Hard https://github.com/suzgunmirac/BIG-Bench-Hard and just published, Advanced Reasoning Benchmark https://arb.duckai.org/ https://arb.duckai.org/
- Tostino 3y agoThat seems more in-line with my experience. I have been using GPT-3.5 and GPT-4 for data cleaning pipelines, and have tried to swap out LLaMA2 70B in a few of the "easier" tasks, and it hasn't performed well enough yet for any of my tasks done by GPT-3.5.
- dr_kiszonka 3y agoHi! Could you please share a few words on what type of data you are cleaning using GPT? It is an intriguing idea and I would love to learn more to see if I could use a similar approach.
- luckystarr 3y agoThe value of GPT-4 also lies in its stored knowledge. A 70B model can't store that much.
- lolinder 3y agoThe advantage of LLaMA 2 is that a company can fine tune it on the knowledge that they actually care about and then run it on their own hardware without paying API fees or relying on an unstable dependency that's constantly being tweaked.
- meepmorp 3y ago> without paying API fees or relying on an unstable dependency that's constantly being tweaked. and without handing a whole bunch of data to a 3rd party and hope they're securing it properly
- lhl 3y agoNote, that just because you're hosting itself won't mean you're securing it properly... Here's a just published injection attack that only works on open source models (public model weights): https://twitter.com/random_walker/status/1683833600196714497 https://twitter.com/random_walker/status/1683833600196714497
- huggingmouth 3y agoThere's a world of difference between visiting links hallucinated by an unreliable ai and having a third party store (and possibly sell) every single thing you say to an ai and tying it to your identity, forever. Also there is nothing about that attack that makes it iinherently only applicable to self hosted models.
- lhl 3y agoThat specific attack requires adversarial inputs crafted crafted against gradients so only works against open models (requires known model weights). There are dangers that include leaking PII from your current context, but also worse if you are using the model with RAG or with other types of system access so I don't think it's as innocuous as you are assuming.
- RcouF1uZ4gsC 3y agoThose MacBook Pros with 96 GB of unified GPU/CPU memory are looking pretty good right now. It would be awesome to have all this running on a laptop in a completely offline mode.
- TillE 3y agoI think it'd be more fun to spend an extra $700 and get an M2 Ultra Mac Studio with way more GPU cores and 128GB of RAM, and set up a private server. But if you really want a portable offline thing, sure.
- shanusmagnus 3y agoI get more excited at the prospect of popping into some random cafe, SSHing from my iPad into some vast.ai server and setting loose this giant AI brain on whatever stuff. Feels badass.
- jacknews 3y agoon their own metric?
- Jackson__ 3y ago*When asked by GPT4 to compare the outputs. I'm a staunch believer that it would be foolish to rely on GPT4 for quality comparisons, and it has been mind boggling to see so many people do it and treat it as perfect proof of anything. It would be slightly more understandable if there was a study to see how human and gpt4 preferences compare, but I'm unaware of any such thing.
- letmevoteplease 3y agoThere is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. Despite different views, humans deemed GPT-4’s judgments reasonable in 75% of cases and are even willing to change their choices in 34% of cases."[1] [1] https://arxiv.org/abs/2306.05685 https://arxiv.org/abs/2306.05685
- lolinder 3y ago80% agreement is high, but the margins between models at the top are so low that even that remaining 20% could be enough to alter the final rankings, depending on which direction it errs.
- lhl 3y agoThis was just posted a few hours ago and when I tried it they were neck and neck (for me, LLama 2 won by a 1 question, but it was close): https://llmboxing.com/ https://llmboxing.com/ It looks like the eval is open sourced so you could easily build a version w/ your own questions for blind testing...
- charcircuit 3y agoAt least when I tried Llama 2's response was always the longer one so it was hard to remain unbiased.
- kuprel 3y agoWas LLaMA2 potentially trained on the raw text from this page? https://huggingface.co/datasets/tatsu-lab/alpaca_eval/viewer/alpaca_eval_annotations_alpaca_eval_gpt4/eval https://huggingface.co/datasets/tatsu-lab/alpaca_eval/viewer...
- api 3y agoLLaMA2 seems to compete with ChatGPT 3.5, which is great. It's nowhere near as large as GPT-4 so I would not expect it to be competitive with that. GPT-4 level models that regular people can run with a reasonable hardware budget are going to require innovations in optimization and model efficiency beyond just quantizing weights. Rumor has it that GPT-4 is a "committee" of ~220G models, which would require ~128GiB VRAM at 4-bit quantization to run each model.
- heliophobicdude 3y agoThe benchmark I care about the most for my development workflow is on structured output. Paul Gauthier made this benchmark [1] to measure correct git diffs. If you ask GPT-4 for help with your code, it can output a change in a git diff more reliably than 3.5. My hope is that we can do that with Llama 2. 1:https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html
- weare138 3y agoWhats up with WizardLM-13B-V1.2? I don't know anything about it but the description says it's based on Llama-2 with only 13B parameters and it's holding it's own in the top 5 with a fraction of the model size.
- treprinum 3y agoIsn't Wizard one of the uncensored versions like Luna etc.?
- aantix 3y agoWhat's the most straight forward way of downloading LLaMA2, and training it with additional documents? I have a whole host of personal pdf's and documentation that I would love to be able to ask questions about.
- ubj 3y agoThis may be relevant: https://www.sematic.dev/blog/tuning-and-testing-llama-2-flan-t5-and-gpt-j-with-lora-sematic-and-gradio https://www.sematic.dev/blog/tuning-and-testing-llama-2-flan... It's the most straightforward explanation I've found so far. I'd love to hear if anyone's found something better though.
- golergka 3y agoBut unlike ChatGPT, it's still exclusively English, right?
- ChatGTP 3y agoWe cannot let the Chinese or Russians access this tech \s
- europeanNyan 3y agoThere is a cool website where you can blind judge the outputs from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/ https://llmboxing.com/ Surprisingly, LLaMa 2 won 5-0 for me.
- speedgoose 3y agoIt was much closer to me. But llama 2 did surprisingly good. It’s looks like it’s a great alternative of chatGPT 3.5.
- andrei512 3y agoall the shorter answers were from GPT-3 - if you like long answers you pick llama 2...
- user_7832 3y agoThanks for the link! At least in my examples, the llama output was more verbose/comprehensive. Sometimes ChatGPT didn't expand enough, sometimes Llama missed the mark entirely (eg explaining the Eiffel's architecture.)
- SV_BubbleTime 3y agoInteresting exercise, and llama won for me with 1 GPT answer… but it would be VERY easy to cherry pick these results and select a winner for most people.
- drew-y 3y agoI got the opposite result. ChatGPT-3.5 won 5-0 for me. For me, LLaMa 2 gave longer answers that sometimes strayed away from the original question. They both gave great answers overall though.
- dsabanin 3y agoSame for me. Interesting..
- thorum 3y agoIn a response about the Turing test on this site, LLaMa 2 used the phrase “to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human” which appears to be copied verbatim from the first sentence of the Wikipedia article on the subject (as well as quite a few other pages in Google). Makes me wonder how many of the responses are just repeating and rephrasing memorized content written by humans, which will of course appear better, while ChatGPT makes more effort to avoid this (and might be able to generalize better to things it hasn’t memorized?).
- paxys 3y agoThere are more ways of evaluating LLMs than there are LLMs. All of the "X is better than Y" statements are pointless unless there is a very clear consensus.
- thewataccount 3y agoThe fact Guanaco 33B is at 65% with Vicuna 13B at 70.43% immediately makes these results non-sensical from my own experience with them. Heck from my experience Guanaco 33B is better then Vicuna 33B! Not to mention GPT4 at 95% and ChatGPT at 89% - I use chatgpt(3.5turbo)/gpt4 daily for work, and I rarely ever bother with 3.5turbo because of how unreliable it's answers are compared to gpt4. So whatever this is effectively measuring is useless for comparing these models, especially across work types.
- exo-pla-net 3y agoI wish it were convention to specify the model, such as "gpt-3.5", rather than "ChatGPT", which is a service that hosts multiple models. Talking about ChatGPT creates pointless ambiguity. (But maybe it's a good filter: if someone is talking about "ChatGPT's" performance, they probably don't have anything useful to say.)
- 0xbadc0de5 3y agoAnd Vicuna-33B is not far behind - which you can actually run on a 24GB 3090/4090 GPU unlike LLaMAv2-70B. Although for a lot of tasks, Guanaco outperforms Vicuna.
- krisknez 3y agoI've played a little bit around llama2 and gpt3.5 is still better but llama2 is not far behind.
- mk_stjames 3y agoHas anyone published a similar run of benchmarks with llama2 70B but at different quantization levels? I assume this benchmark is evaluated on the base model run at FP16. How much does it lose quantizing to INT8?
- kosolam 3y agoIs there some free service that allows chatting with the 70b llama2?
- coldblues 3y agoAwesome. What's great is that it can be unfiltered as well. So we'll be able to have models without all that incessant apologizing.
- iandanforth 3y agoIsn't the chat version of llama 2 trained on gpt-4 output, hence it's non-commercial license (as opposed to the base model) or am I just making things up?