5 ms·
How does LLaMA compare to GPT-3.5 has anyone done side by side comparisons?
by cloudking 4y ago
How does LLaMA compare to GPT-3.5 has anyone done side by side comparisons?
- tikkun 4y agoIn short, my experience is that it's much worse to the point that I won't use LLaMA. The main change needed seems to be InstructGPT style tuning (https://openai.com/research/instruction-following https://openai.com/research/instruction-following)
- simonw 4y agoYeah, it's MUCH harder to use because of the lack of tuning. You have to lean on much older prompt engineering tricks - there are a few initial tips in the LLaMA FAQ here: https://github.com/facebookresearch/llama/blob/main/FAQ.md#2-generations-are-bad https://github.com/facebookresearch/llama/blob/main/FAQ.md#2...
- davidb_ 4y agoAre you getting useful content out of the 7B model? It goes off the rails way too often for me to find it useful.
- rnosov 4y agoYou might want to tune the sampler. For example, set it to a lower temperature. Also, the 4bit RTN quantisation seems to be messing up the model. Perhaps, the GPTQ quantisation will be much better.
- spion 4y agoUse `--top_p 2 --top_k 40 --repeat_penalty 1.176 --temp 0.7` with llama.cpp
- datadeft 4y agoNot bad with these settings: ./main -m ./models/7B/ggml-model-q4_0.bin \ --top_p 2 --top_k 40 \ --repeat_penalty 1.176 \ --temp 0.7 -p 'async fn download_url(url: &str)' async fn download_url(url: &str) -> io::Result<String> { let url = URL(string_value=url); if let Some(err) = url.verify() {} // nope, just skip the downloading part else match err == None { // works now true => Ok(String::from(match url.open("get")?{ |res| res.ok().expect_str(&url)?, |err: io::Error| Err(io::ErrorKind(uint16_t::MAX as u8))), false => Err(io::Error
- yeeeloit 4y agoWhat fork are you using? repeat_penalty is not an option.
- knodi123 4y agolol, ./main -m ./models/7B/ggml-model-q4_0.bin \ --top_p 2 --top_k 40 \ --repeat_penalty 1.176 \ --temp 0.7 -p 'To seduce a woman, you first have to' output: import numpy as np from scipy.linalg import norm, LinAlgError np.random.seed(10) x = -2\*norm(LinAlgError())[0] # error message is too long for command line use print x [end of text]
- Tepix 4y agoHave you tried using the original repo?
- deleted 4y ago[deleted]
- DavidSJ 4y agoGPT-3.5 probably has substantially lower cross-entropy loss before instruction fine-tuning as well. Meta’s LLaMA v. GPT-3 comparisons are to OpenAI’s 2020 release of GPT-3, but there’s been significant progress since then.
- KennyBlanken 4y agoThe writeup includes example text where the algorithm is fed a sentence starting about George Washington and within half a sentence or so goes unhinged and starts praising Trump... Also, a reminder to folks that this model is not conversationally trained and won't behave like ChatGPT; it cannot take directions.
- superkuh 4y agoWell, gpt3.5-turbo fails the turing test due to it's censorship and legal liability butt covering openai bolted on, so almost anything else is better. Now, compared to openai's gpt3 davinci (text-davinci-003) ... llama is much worse.
- nodemaker 4y agoI dont know why youre getting downvoted! There is nothing out there at the moment that is as authentic as text-davinci-003. I really hope its not taken away
- flangola7 4y agoI thought LLaMA outscored GPT-3
- simonw 4y agoGPT-3 is a very different model from GPT-3.5. My understanding is that they were comparing LLaMA's performance to benchmark scores published for the original GPT-3, which came out in 2020 and had not yet had instruction tuning, so was significantly harder to use.
- flangola7 4y agoI know, that is why I said GPT-3 (Davinci) not GPT-3.5|ChatGPT.
- simonw 4y agoDa Vinci 002 and 003 are actually classified as GPT 3.5 by OpenAI: https://platform.openai.com/docs/models/gpt-3-5 https://platform.openai.com/docs/models/gpt-3-5 ChatGPT is GPT-3.5 Turbo.
- eshack94 4y agoWould you mind summarizing the difference between GPT 3.5 and GPT-3.5 Turbo? I'm not clear about that.
- bt1a 4y agoThe sampler needs some tuning, but the 65B model has impressive output https://twitter.com/theshawwn/status/1632569215348531201 https://twitter.com/theshawwn/status/1632569215348531201
- aroo 4y agoThe original LLaMA paper has some benchmarks. https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf