4 ms·
IMO Llama2 really isn’t close to 3.5. It still has regular mode collapse (or whatever you call getting repetitive and nonsensical responses after a while), it h
by dr_dshiv 3y ago
IMO Llama2 really isn’t close to 3.5. It still has regular mode collapse (or whatever you call getting repetitive and nonsensical responses after a while), it has very poor mathematical/logical reasoning and is not good at following multi-part instructions.
It just sounds like 3.5/4 because it was trained on it.
- geysersam 3y agoYou're mixing up the language model with the chat bot. The llama2 is a language model. I imagine the language model behind chatgpt is not much different (perhaps it's better, but not by many months AI research time). It likely also suffers from "mode collapse" issues etc. But 3.5 also has a lot of systems around it that detects mode collapse and applies some kind of mitigation, forcing the model to give a more reasonable output. Mathematical / logical reasoning questions are likely also detected hand passed on in some form to a separate system.
- dr_dshiv 3y agoSo this would be testable by showing that chatGPT makes more mistakes than prompting via API? Or would you consider the API a chatbot, too?
- geysersam 3y agoI don't think there's any public interface to the LLM underlying ChatGPT, so the only ones able to test this are openAI engineers.
- vorticalbox 3y agoThis is what presence_penalty and frequency_penalty are for.
- nl 3y agoLlama 2 wasn't trained on ChatGPT/GPT4. I think maybe you are thinking of the Vicuna models? https://lmsys.org/blog/2023-03-30-vicuna/ https://lmsys.org/blog/2023-03-30-vicuna/