3 ms·
Saying fine-tuned LLaMA can outperform GPT-3.5-turbo on some tasks would be reasonably generous. Saying it can outperform GPT-3.5-turbo on "many tasks" would b
by BoorishBears 3y ago
Saying fine-tuned LLaMA can outperform GPT-3.5-turbo on some tasks would be reasonably generous.
Saying it can outperform GPT-3.5-turbo on "many tasks" would be venturing into unreasonable territory.
Saying it can outperform GPT-3.5-turbo or even GPT-4 on many tasks is just setting people up for disappointment: There's a reason ARC (common sense reasoning) never gets mentioned when touting OS LLMs vs commercial, and the gap in ARC performance is amplified terribly when you try to apply chain-of-thought
- Palmik 3y agoI am totally with you when it comes to reasoning, or when it comes to generally / universally capable models. But many problems don't require reasoning, and are very concrete / specific.
- SOLAR_FIELDS 3y agoI use my models primarily for code generation and evaluation. Does the latter part fall into “reasoning” territory?
- Palmik 3y agoAdvanced code generation, the kind you would need for something like AutoGPT or other agent-like system, is definitely reasoning and GPT-4 or Claude 2 will dominate. But most of the code generation from GitHub Copilot or other systems (like Replit) is much simpler pattern matching (and indeed, Replit afaik uses very small models). What do you mean by evaluation?
- SOLAR_FIELDS 3y agoMy personal use case is most often supplying files and generating unit tests, as well as supplying a section of code with a prompt about that section to evaluate the approach contextually with some other unseen code. Which perhaps is another form of reasoning.
- benob 3y agoI am trying to find support for your last argument. https://paperswithcode.com/sota/common-sense-reasoning-on-arc-challenge https://paperswithcode.com/sota/common-sense-reasoning-on-ar... GPT-3 53.2 GPT-3.5 85.2 LLaMa-65B 56.0 Any idea of the performance of an instruction-fine-tuned version of LLaMa models? I can't seem to find non-aggregated performance figures on ARC.
- behnamoh 3y agoWow, the top 14 are all proprietary models (except 2 models which are not commonly used in OS community anyway). And surprisingly, Llama 33B performs _better_ than Llama 65B!
- NhanH 3y agoThat table is comparing few shot prompting GPT 4 to zero shot llama.
- BoorishBears 3y agoI'm not sure if these ARC scores here are fully comparable to those in scaling, but if they are: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... TigerResearch/tigerbot-70b-chat: ARC (76.79) Still below 3.5 and like most top entries no clear training objectives, which usually means they were fine-tuned on the benchmark itself. - ARC is easily the most important benchmark right now for widespread adoption of LLMs by laypeople: it's literal grade school multiple choice, created to the bar of an 8th grader. When you sit people down in front of an LLM and have them interact with it: ARC has by far the closest correlation of how "generally smart" the model will feel.