4 ms·

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "g

Does anything currently beat GPT-4? I saw some comments here say to check out Claude. From what I can tell, Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet.

jxdxbx 3y ago
Claude is better than GPT 4 for my uses, and was able to help me do some simple coding things that GPT 4 could not. It’s worth trying at least.
monsieurbanana 3y ago
Isn't that something you get from the infrastructure surrounding the llm? I thought the "running code" feature didn't need specific support from the llm, besides being able to output conforming json or code when asked to.
- MuffinFlavored 3y ago
  The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)
  - avree 3y ago
    Doesn't seem like you are very informed on how LLMs work, but just so you know, there are many different versions of Claude, just like how ChatGPT can use different versions of GPT.
  - simonw 3y ago
    That's because the Claude web UI doesn't yet have the equivalent of the ChatGPT Code Interpreter tool (though they say they're working on it). That's not about the quality of the Claude 3 Opus model, which is the model which people think compares to or beats GPT-4. It's about the tooling that has been built for ChatGPT.
    - fragmede 3y ago
      Code interpreter is pretty neat, because you can tell ChatGPT to write some code and to make sure the code works, and then it'll write you some bad code, realize it's bad, and then iterate on it until it gets to a place that it's happy with. (Maybe I should say passes its test rather than anthropomorphize ChatGPT as being "happy".)
  - cosmojg 3y ago
    Right, like the other commenter suggested, that's an infrastructure-level thing, not a model-level thing. Given that you're talking about ChatGPT, I assume you aren't accessing GPT-3.5 or GPT-4 directly through the API but using the app or the interface provided at chat.openai.com. The magic that makes the kinds of interactions you're describing possible amounts to a bit of clever prompting sprinkled on top of some rather impressive frontend design and engineering. Correctly prompted, even Mistral-7B can write and run code in response to questions, and it's a model that can run on laptops from half a decade ago, with two or three orders of magnitude fewer parameters that GPT-4.
    - MuffinFlavored 3y ago
      > Right, like the other commenter suggested, that's an infrastructure-level thing, not a model-level thing. By default, the ChatGPT "model" knows to not try to do math and instead write code to do the math then run it. I get that it's set up infrastructure wise to be able to run it, but why is Claude's main chat UI not trying to instead respond "hey, do this calculation on your own since I can't" or something of this nature instead of responding to math incorrectly
      - cosmojg 3y ago
        Because ChatGPT ships with a system prompt which instructs the underlying model to do exactly that. A similar web application could be developed for Claude, and it would perform similarly with the right prompt, as it's quite good at tool use. For example, I'm able to get Claude-3-Opus to write Python and call a Python interpreter of its own accord when questioned about time series data in some of my data analysis workflows, though I haven't glued together a pretty GUI for it yet (e.g., plots are simply saved to disk). While I haven't run into any problems around calculations yet, I'm sure it wouldn't be too hard to further refine the system prompt and ensure that all calculations are performed or checked using Python.
rubymamis 3y ago
A programming task where Mistral-large beats both GPT-4 and Claude Opus: https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 (only Mistral got the current syntax) Although based on other tasks, overall, GPT-4 seems to be the best, but by a very small margin, so I cancelled my subscription. Although the native mobile app is really great.
- cosmojg 3y ago
  Do you prefer Mistral-Large or Claude-Opus?
  - rubymamis 3y ago
    Not sure. Most of the time GPT-4 is better. Since I'm using Vercel AI playground[1], on almost every query I get a response from all models so it's easy to compare. [1] https://sdk.vercel.ai/ https://sdk.vercel.ai/
- fragmede 3y ago
  Is there a way to use Mistral-large with TTS and STT engines so you can converse with it like you can ChatGPT in the mobile app? it's really great on long drives for learning/talking about stuff, like a customized personal podcast.
  - rubymamis 3y ago
    Exactly, I absolutely love this feature. And many times the conversation is quite natural and fluid (with good internet connection). I think I'll build something like that myself (:
geor9e 3y ago
I don't know what system and user prompt you are testing with, but as one anecdote, Claude 3 Opus (and only Opus) consistently gives me better coding answers than GPT-4. Maybe it's the type of stuff I am doing or how I phrase things, who knows. I was using GPT-4 since the day it came out but haven't felt like going back so far.
treprinum 3y ago
Claude-2 in some tasks albeit it's a bit slower, Mistral on some tasks and it's a bit faster.
yieldcrv 3y ago
The benchmark is Sora or whatever Open AI is working on right now or next, not trying to beat the model released a year ago and still failing so when looking at it that way, the real question is what do you need? all I need is Mixtral 7x8B Q5 in an 8,000 token context window, at the moment I think there are plenty of other people that can design their applications and problems around lower fidelity tools, or just pursue something else
jasonjmcghee 3y ago
Claude Opus (largest v3 model) consistently outperforms GPT-4 for me. Better at following prompts, _feels_ much better.
thorum 3y ago
I don’t know what the people who say Claude 3 is better than GPT-4 are using it for. It’s been consistently worse for everything I’ve thrown at it. Debugging a Python function this morning. Claude 3 Opus failed completely. GPT-4 found the bug, as well as two others I hadn’t even been looking for.
- simonw 3y ago
  I've had the opposite experience: coding prompts that GPT-4 makes mistakes on Claude 3 Opus gets right the first time. As always, your results will vary based on your personal prompting style. My style apparently works great with Opus. Here's one example: GPT-4 gave me code that was missing some async/await keywords: https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f3262594d https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f32... Claude 3 Opus with the same prompt got it right the first time: https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83074 https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83...
  - brianjking 3y ago
    Yeah, Opus has entirely taken over any code specific use for me over ChatGPT 4 or OpenAI GPT-4 API. Once Opus has the ability to run a code interpreter, it'll really be an exciting time.
drexlspivey 3y ago
According to Chatbot Arena where people vote on responses blindly and an ELO rating is determined for each LLM, gpt4 is on top slightly ahead of Claude 3 Opus https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
marviel 3y ago
Claude is excellent for brainstorming, being a thought partner, general knowledge acquisition tasks, and creative writing. The one mixed-bag weak spot I've found is in coding -- It tends to make more "d'oh" mistakes while coding, but comes up with more creative solutions at the same time ¯\_(ツ)_/¯
dragonwriter 3y ago
> Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet. GPT-4 didn't figure that out, either; that’s just tooling built around the model, not something the model “figures out”.

jxdxbx 3y ago

Claude is better than GPT 4 for my uses, and was able to help me do some simple coding things that GPT 4 could not. It’s worth trying at least.

monsieurbanana 3y ago

Isn't that something you get from the infrastructure surrounding the llm? I thought the "running code" feature didn't need specific support from the llm, besides being able to output conforming json or code when asked to.

MuffinFlavored 3y ago
The LLM (Claude) currently doesn't know to not hallucinate numbers and instead write code + run it (something ChatGPT used to do but they fixed it)
- avree 3y ago
  Doesn't seem like you are very informed on how LLMs work, but just so you know, there are many different versions of Claude, just like how ChatGPT can use different versions of GPT.
- simonw 3y ago
  That's because the Claude web UI doesn't yet have the equivalent of the ChatGPT Code Interpreter tool (though they say they're working on it). That's not about the quality of the Claude 3 Opus model, which is the model which people think compares to or beats GPT-4. It's about the tooling that has been built for ChatGPT.
  - fragmede 3y ago
    Code interpreter is pretty neat, because you can tell ChatGPT to write some code and to make sure the code works, and then it'll write you some bad code, realize it's bad, and then iterate on it until it gets to a place that it's happy with. (Maybe I should say passes its test rather than anthropomorphize ChatGPT as being "happy".)
- cosmojg 3y ago
  Right, like the other commenter suggested, that's an infrastructure-level thing, not a model-level thing. Given that you're talking about ChatGPT, I assume you aren't accessing GPT-3.5 or GPT-4 directly through the API but using the app or the interface provided at chat.openai.com. The magic that makes the kinds of interactions you're describing possible amounts to a bit of clever prompting sprinkled on top of some rather impressive frontend design and engineering. Correctly prompted, even Mistral-7B can write and run code in response to questions, and it's a model that can run on laptops from half a decade ago, with two or three orders of magnitude fewer parameters that GPT-4.
  - MuffinFlavored 3y ago
    > Right, like the other commenter suggested, that's an infrastructure-level thing, not a model-level thing. By default, the ChatGPT "model" knows to not try to do math and instead write code to do the math then run it. I get that it's set up infrastructure wise to be able to run it, but why is Claude's main chat UI not trying to instead respond "hey, do this calculation on your own since I can't" or something of this nature instead of responding to math incorrectly
    - cosmojg 3y ago
      Because ChatGPT ships with a system prompt which instructs the underlying model to do exactly that. A similar web application could be developed for Claude, and it would perform similarly with the right prompt, as it's quite good at tool use. For example, I'm able to get Claude-3-Opus to write Python and call a Python interpreter of its own accord when questioned about time series data in some of my data analysis workflows, though I haven't glued together a pretty GUI for it yet (e.g., plots are simply saved to disk). While I haven't run into any problems around calculations yet, I'm sure it wouldn't be too hard to further refine the system prompt and ensure that all calculations are performed or checked using Python.

rubymamis 3y ago

A programming task where Mistral-large beats both GPT-4 and Claude Opus: https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 https://sdk.vercel.ai/s/Thi7RDx9e8VOZo1Ee6We5 (only Mistral got the current syntax) Although based on other tasks, overall, GPT-4 seems to be the best, but by a very small margin, so I cancelled my subscription. Although the native mobile app is really great.

cosmojg 3y ago
Do you prefer Mistral-Large or Claude-Opus?
- rubymamis 3y ago
  Not sure. Most of the time GPT-4 is better. Since I'm using Vercel AI playground[1], on almost every query I get a response from all models so it's easy to compare. [1] https://sdk.vercel.ai/ https://sdk.vercel.ai/
fragmede 3y ago
Is there a way to use Mistral-large with TTS and STT engines so you can converse with it like you can ChatGPT in the mobile app? it's really great on long drives for learning/talking about stuff, like a customized personal podcast.
- rubymamis 3y ago
  Exactly, I absolutely love this feature. And many times the conversation is quite natural and fluid (with good internet connection). I think I'll build something like that myself (:

geor9e 3y ago

I don't know what system and user prompt you are testing with, but as one anecdote, Claude 3 Opus (and only Opus) consistently gives me better coding answers than GPT-4. Maybe it's the type of stuff I am doing or how I phrase things, who knows. I was using GPT-4 since the day it came out but haven't felt like going back so far.

treprinum 3y ago

Claude-2 in some tasks albeit it's a bit slower, Mistral on some tasks and it's a bit faster.

yieldcrv 3y ago

The benchmark is Sora or whatever Open AI is working on right now or next, not trying to beat the model released a year ago and still failing so when looking at it that way, the real question is what do you need? all I need is Mixtral 7x8B Q5 in an 8,000 token context window, at the moment I think there are plenty of other people that can design their applications and problems around lower fidelity tools, or just pursue something else

jasonjmcghee 3y ago

Claude Opus (largest v3 model) consistently outperforms GPT-4 for me. Better at following prompts, _feels_ much better.

thorum 3y ago

I don’t know what the people who say Claude 3 is better than GPT-4 are using it for. It’s been consistently worse for everything I’ve thrown at it. Debugging a Python function this morning. Claude 3 Opus failed completely. GPT-4 found the bug, as well as two others I hadn’t even been looking for.

simonw 3y ago
I've had the opposite experience: coding prompts that GPT-4 makes mistakes on Claude 3 Opus gets right the first time. As always, your results will vary based on your personal prompting style. My style apparently works great with Opus. Here's one example: GPT-4 gave me code that was missing some async/await keywords: https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f3262594d https://chat.openai.com/share/117fb1ad-6361-41e2-be59-110f32... Claude 3 Opus with the same prompt got it right the first time: https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83074 https://gist.github.com/simonw/2002e2b56a97053bd9302a34e0b83...
- brianjking 3y ago
  Yeah, Opus has entirely taken over any code specific use for me over ChatGPT 4 or OpenAI GPT-4 API. Once Opus has the ability to run a code interpreter, it'll really be an exciting time.

drexlspivey 3y ago

According to Chatbot Arena where people vote on responses blindly and an ELO rating is determined for each LLM, gpt4 is on top slightly ahead of Claude 3 Opus https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

marviel 3y ago

Claude is excellent for brainstorming, being a thought partner, general knowledge acquisition tasks, and creative writing. The one mixed-bag weak spot I've found is in coding -- It tends to make more "d'oh" mistakes while coding, but comes up with more creative solutions at the same time ¯\_(ツ)_/¯

dragonwriter 3y ago

> Claude hasn't figure out yet how to do the whole "generate Python code and run it in a Juptyer notebook" for math yet. GPT-4 didn't figure that out, either; that’s just tooling built around the model, not something the model “figures out”.