4 ms·

Not sure what you are using it for, but it is terrible for me for coding; claude beats it always and hands down. o1 just thinks forever to come up with stuff it

Not sure what you are using it for, but it is terrible for me for coding; claude beats it always and hands down. o1 just thinks forever to come up with stuff it already tried the previous time. People say that's just prompting without pointing to real million line+ repositories or realistic apps to show how that can be improved. So I say they are making todo and hello world apps and yes, there it works really well. Claude still beats it, every.. single.. time.. And yes, I use the Pro of all and yes, I do assume coding is done for most of people. Become a plumber or electrician or carpenter.

rubymamis 2y ago
I find that o1 and Sonnet 3.5 are good and bad quite equally on different things. That's why I keep asking both the same coding questions.
- anonzzzies 2y ago
  We do the same (all requests go to o1, sonnet and gemini and we store the results for later to compare) automatically for our research: Claude always wins. Even with specific prompting on both platforms. Especially frontend it seems o1 really is terrible.
  - ynniv 2y ago
    Claude is trained on principles. GPT is trained on billions of edge cases. Which student do you prefer?
    - deleted 2y ago
      [deleted]
  - rubymamis 2y ago
    Every time I try Gemini, it's really subpar. I found that qwen2.5-coder-32b-instruct can be better. Also, for me 50% 50% for Sonnet and o1, but although I'm not 100% sure about it, I think o1 is better with longer and more complicated (C++) code and debugging. At least from my brief testing. Also, OpenAI models seem to be more verbose - sometimes it's better - where I'd like additional explanation on chosen fields in a SQL schema, sometimes it's too much. EDIT: Just asked both o1 and Sonnet 3.5 the same QML coding question, and Sonnet 3.5 succeeded, o1 failed.
    - oceanplexian 2y ago
      Very anecdotal but I’ve found that for things that are well spec’d out with a good prompt Sonnet 3.5 is far better. For problems where I might have introduced a subtle logical error o1 seems to catch it extremely well. So better reasoning might be occurring but reasoning is only a small part of what we would consider intelligence.
  - CapcomGo 2y ago
    Wins? What does this mean? Do you have any results? I see the claims that Claude is better for coding a lot but using it and using Gemini 2.0 flash and o1 and it sure doesn't seem like it.
  - energy123 2y ago
    A new o1 was released on December 17th. Which one are you talking about
    - tigershark 2y ago
      Exactly. The previous version of o1 did actually worse in the coding benchmarks, so I would expect it to be worse in real life scenarios. The new version released a few days ago on the other hand is better in the benchmarks, so it would seem strange that someone used it and is saying that it’s worse than Claude.
h_tbob 2y ago
That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.
- master_crab 2y ago
  Claude also has a better workflow UI. It’ll maintain conversation context while opening up new windows to present code suggestions. When I was still subscribing to OpenAI (about 4 months ago) this didn’t exist.
  - rrrrrrrrrrrryan 2y ago
    It exists as of last week with Canvas.
  - fragmede 2y ago
    If you're using the web interface of either, you might consider looking into tools that focus on using LLMs for code, so you're not copy/pasting.
- A_D_E_P_T 2y ago
  They're both okay for coding, though for my use cases (which are niche and involve quite a lot of mathematics and formal logic) o1/o1-Pro is better. It seems to have a better native grasp of mathematical concepts, and it can even answer very difficult questions from vague inputs, e.g.: https://chatgpt.com/share/676020cb-8574-8005-8b83-4bed5b13e1cd https://chatgpt.com/share/676020cb-8574-8005-8b83-4bed5b13e1...
  - deleted 2y ago
    [deleted]
- orbital-decay 2y ago
  Different languages maybe? I find Sonnet v2 to be lacking in Rust knowledge compared to 4o 11-20, but excelling at Python and JS/TS. O1's strong side seems to be complex or quirky puzzle-like coding problems that can be answered in a short manner, it's meh at everything else, especially considering the price. Which is understandable given its purpose and training, but I have no use for it as that's exactly the sort of problem I wouldn't trust an LLM to solve. Sonnet v2 in particular seems to be a bit broken with its reasoning (?) feature. The one where it detects it might be hallucinating (what's even the condition?) and reviews the reply, reflecting on it. It can make it stop halfway into the reply and decide it wrote enough, or invent some ridiculous excuse to output a worse answer. Annoying, although it doesn't trigger too often.

rubymamis 2y ago

I find that o1 and Sonnet 3.5 are good and bad quite equally on different things. That's why I keep asking both the same coding questions.

anonzzzies 2y ago
We do the same (all requests go to o1, sonnet and gemini and we store the results for later to compare) automatically for our research: Claude always wins. Even with specific prompting on both platforms. Especially frontend it seems o1 really is terrible.
- ynniv 2y ago
  Claude is trained on principles. GPT is trained on billions of edge cases. Which student do you prefer?
  - deleted 2y ago
    [deleted]
- rubymamis 2y ago
  Every time I try Gemini, it's really subpar. I found that qwen2.5-coder-32b-instruct can be better. Also, for me 50% 50% for Sonnet and o1, but although I'm not 100% sure about it, I think o1 is better with longer and more complicated (C++) code and debugging. At least from my brief testing. Also, OpenAI models seem to be more verbose - sometimes it's better - where I'd like additional explanation on chosen fields in a SQL schema, sometimes it's too much. EDIT: Just asked both o1 and Sonnet 3.5 the same QML coding question, and Sonnet 3.5 succeeded, o1 failed.
  - oceanplexian 2y ago
    Very anecdotal but I’ve found that for things that are well spec’d out with a good prompt Sonnet 3.5 is far better. For problems where I might have introduced a subtle logical error o1 seems to catch it extremely well. So better reasoning might be occurring but reasoning is only a small part of what we would consider intelligence.
- CapcomGo 2y ago
  Wins? What does this mean? Do you have any results? I see the claims that Claude is better for coding a lot but using it and using Gemini 2.0 flash and o1 and it sure doesn't seem like it.
- energy123 2y ago
  A new o1 was released on December 17th. Which one are you talking about
  - tigershark 2y ago
    Exactly. The previous version of o1 did actually worse in the coding benchmarks, so I would expect it to be worse in real life scenarios. The new version released a few days ago on the other hand is better in the benchmarks, so it would seem strange that someone used it and is saying that it’s worse than Claude.

h_tbob 2y ago

That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.

master_crab 2y ago
Claude also has a better workflow UI. It’ll maintain conversation context while opening up new windows to present code suggestions. When I was still subscribing to OpenAI (about 4 months ago) this didn’t exist.
- rrrrrrrrrrrryan 2y ago
  It exists as of last week with Canvas.
- fragmede 2y ago
  If you're using the web interface of either, you might consider looking into tools that focus on using LLMs for code, so you're not copy/pasting.
A_D_E_P_T 2y ago
They're both okay for coding, though for my use cases (which are niche and involve quite a lot of mathematics and formal logic) o1/o1-Pro is better. It seems to have a better native grasp of mathematical concepts, and it can even answer very difficult questions from vague inputs, e.g.: https://chatgpt.com/share/676020cb-8574-8005-8b83-4bed5b13e1cd https://chatgpt.com/share/676020cb-8574-8005-8b83-4bed5b13e1...
- deleted 2y ago
  [deleted]
orbital-decay 2y ago
Different languages maybe? I find Sonnet v2 to be lacking in Rust knowledge compared to 4o 11-20, but excelling at Python and JS/TS. O1's strong side seems to be complex or quirky puzzle-like coding problems that can be answered in a short manner, it's meh at everything else, especially considering the price. Which is understandable given its purpose and training, but I have no use for it as that's exactly the sort of problem I wouldn't trust an LLM to solve. Sonnet v2 in particular seems to be a bit broken with its reasoning (?) feature. The one where it detects it might be hallucinating (what's even the condition?) and reviews the reply, reflecting on it. It can make it stop halfway into the reply and decide it wrote enough, or invent some ridiculous excuse to output a worse answer. Annoying, although it doesn't trigger too often.