27 ms·
When will we reach an upper limit/dimishing returns in terms of number of parameters and mixture of experts?
by hubraumhugo 3y ago
When will we reach an upper limit/dimishing returns in terms of number of parameters and mixture of experts?
- andy99 3y agoWe may have already - data is more important than anything else which is why nobody has beat GPT4 yet. Throwing more parameters or more compute at the problem only gets you so far. But Grok was never a contender so there is room to improve on it. It is one of the biggest models open sourced as mentioned, so will be interesting to take a look at for sure.
- squigz 3y agoI think Groq is something else?
- lambdaba 3y agoClaude 3 has *decisively* beat GPT-4, I wonder how all their attributes compare.
- stainablesteel 3y agoi like some of claudes answers better, but it doesnt seem to be a better coder imo
- simonw 3y agoI've found it to be significantly better for code than GPT-4 - I've had multiple examples where the GPT-4 solution contained bugs but the Claude 3 Opus solution was exactly what I wanted. One recent example: https://fedi.simonwillison.net/@simon/112057299607427949 https://fedi.simonwillison.net/@simon/112057299607427949 How well models work varies wildly according to your personal prompting style though - it's possible I just have a prompting style which happens to work better with Claude 3.
- bugglebeetle 3y agoWhat is your code prompting style for Claude? I’ve tried to repurpose some of my GPT-4 ones for Claude and have noticed some degradation. I use the “Act as a software developer/write a spec/implement step-by-step” CoT style.
- simonw 3y agoAlmost impossible to describe prompting style, but here are some examples of how I've used Claude 3: https://gist.github.com/simonw/4cecde4a729f4da0b5059b50c8e01359 https://gist.github.com/simonw/4cecde4a729f4da0b5059b50c8e01... - writing a Python function https://gist.github.com/simonw/408fcf28e9fc6bb2233aae694f8cd1f4 https://gist.github.com/simonw/408fcf28e9fc6bb2233aae694f8cd... - most sophisticated example, building a JavaScript command palette https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83074 https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83... - asking it to refactor some existing code I don't use the "Act as a X" format any more, I'm not at all convinced it has a noticeable impact on quality. I think it's yet another example of LLM superstition.
- lgas 3y ago> I don't use the "Act as a X" format any more, I'm not at all convinced it has a noticeable impact on quality. I think it's yet another example of LLM superstition. It's very contextually dependent. You really have to things like this for your specific task, with your specific model, etc. Sometimes it helps, sometimes it hurts, and sometimes it does nothing at all.
- htrp 3y agocitation needed (other than 'vibes')
- deleted 3y ago[deleted]
- swalsh 3y agoI don't know if Claude is "smarter" in any significant way. But its harder working. I can ask it for some code, and I never get a placeholder. It dutifully gives me the code I need.
- lambdaba 3y agoIt understands instructions better, it's rarer to have it misunderstand, and I have to be less careful with prompting.
- orbital-decay 3y agoHas it, though? LMSys Arena Leaderboard (blind ranking by humans) [0] positions Opus just below GPT-4 with a negligible ELO gap. [0] https://chat.lmsys.org/ https://chat.lmsys.org/
- BoorishBears 3y agoChatbot Arena is not a blind ranking. Many, if not most, users intentionally ask the models questions to tease out their canned disclaimers: so they know exactly which model is answering. On one hand it's fair to say disclaimers affect the usefulness of the model, but on the other I don't think most people are solely asking these LLMs to produce meth or say "fuck", and that has an outsized effect on the usefulness of Chatbot Arena as a general benchmark. I personally recommend people use it at most as a way to directly test specific LLMs and ignore it as a benchmark.
- staticman2 3y agoThat "blind ranking" is limited to about 2,000 tokens of context. So it's certainly not evaluating how good the models are at complex assignments.
- espadrine 3y agoA number of AI companies have a naming/reproducibility issue. GPT4 Turbo, released last November, is a separate version that is much better than GPT-4 (winning 70% of human preferences in blind tests), released in March 2023. Claude 3 Opus beats release-day GPT-4 (winning 60% of human preferences), but not GPT-4 Turbo. In the LMSys leaderboard, release-day GPT-4 is labeled gpt-4-0314, and GPT4 Turbo is labeled gpt-4-1106-preview.
- YetAnotherNick 3y agoThere is no reason to believe GPT-4 had more(or higher quality) data than Google etc. has now. GPT-4 was entirely trained before the Microsoft deal. If OpenAI could pay to acquire data in 2023, >10 companies could acquire similar quality by now, and no one has similar quality model in a year.
- austhrow743 3y agoThe more disregard a company has for intellectual property rights, the more data they can use. Google had far more to lose from a "copyright? lol" approach than OpenAI did.
- YetAnotherNick 3y agoHaving used both Google's and OpenAI's models, the kind of issue they have are different. Google's models are superior or at least on par in knowledge. It's the instruction following and understanding where OpenAI is significantly better. I don't think pretraining data is the reason of this.
- supafastcoder 3y ago> Google had far more to lose from a "copyright? lol" approach than OpenAI did. The company that scrapes trillions of web pages has an issue with copyright?
- sib 3y agoWell... Googlebot does pay attention to robots.txt - I don't think (original) OpenAI-bot did.
- brookst 3y agoI was under the impression training was at best an undefined area of IP law. Is there any aspect of copyright that prohibits training models?
- simonw 3y ago
- ldjkfkdsjnv 3y agoClaude > GPT4. Anyone using these models on a daily basis knows this
- int_19h 3y agoI use these models regularly, and Claude is dumb as a rock compared to GPT-4.
- jstummbillig 3y agoIt is known
- deleted 3y ago[deleted]