4 ms·

Is o1 even that good? It's doesn't even rank first on LMArena..

by zb3 2y ago

Tostino 2y ago
For very specific tasks, yeah it's good. For my usual coding tasks...no way.
mistercow 2y ago
I haven’t been super impressed with it, and haven’t encountered any practical tasks I wanted to solve with an LLM where o1 worked any better than prompting 4o or Sonnet to use more extensive CoT. There might be some narrow band of practical problems in between what other LLMs can do and what o1 can’t, but I don’t think that really matters for most use cases, especially given how much slower it is. Day to day, you just don’t really want to prompt a model near the limits of its capabilities, because success quickly becomes a coin flip. So if a model needs five times as long to work, it needs to dramatically expand the range of problems that can be solved reliably.
- torginus 2y ago
  I think the true edge of CoT models will come from layman usability. While I can easily prompt Claude for examples and then manually modify the code to fill in the gaps, general domain knowledge and technical understanding is absolutely required from the human sitting in front of the screen. With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. So it's not so much about increased capability, but removing the expert human in the loop.
  - itsoktocry 2y ago
    >With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. I just punched that prompt into Sonnet 3.5 and o1 and I wouldn't say that o1 is doing anything better than Sonnet. o1 certainly didn't "do it", it gave me a very broad outline of how to accomplish that, from "Define requirements" to "Test and deply on Vercel"
    - torginus 2y ago
      Honestly I had pretty good success with it. I wanted to try AWS batch for an example app after people here suggested it, and I had something running with like 2 prompts.
foundry27 2y ago
My understanding was that the metric for LMArena is that one answer is “better” than another, for a deliberately 100% subjective definition of better. My experience has been that typical LLMs will have more “preamble” to what they say, easing the reader (and priming themselves autoregressively) into answers with some relevant introduction of the subject, sometimes justifying the rationale and implications behind things. But for o1, that transient period and the underlying reasoning behind things is part of OpenAI’s special sauce, and they deliberately and aggressively take steps to hide it from users. o1 will get correct answers to hard problems more often than other models (look at the math/coding/hard subsections on the leaderboard, where anecdotal experiences aside, it is #1), and there’s a strong correlation between correctness and a high score in those domains because getting code or math “right” matters more than the justification or explanation. But in more general domains where there isn’t necessarily an objective right or wrong, I know the vibe matters a lot more to me, and that’s something o1 struggles with.
whimsicalism 2y ago
yes, it’s extremely good. don’t overindex on the lmsys arena, the median evaluator is kinda mid
IAmGraydon 2y ago
Good at what? It's great at breaking down complex problems into small, logical steps. Claude Sonnet 3.5 is still the best for coding. They can be leveraged together by using Aider's architect mode. It gives your request to the "architect" model first and returns a list of steps to implement your idea, but it doesn't write any code at that point. You then approve this and it sends the list to the coding model to actually write the code. This technique creates better quality code than any one model by itself. In Aider, you can assign any model you want as the architect and any other model as the coder. It's really great and I'm looking forward to the AI coding extensions for VSCode doing the same thing since I prefer to work in VSC than on the command line as is necessary with Aider. My only real problem with o1 is that it's ridiculously expensive, to the point that it makes no sense to use it for actual code. In architect mode, however, you can keep the costs under control as there are far fewer input/output tokens.

Is o1 even that good? It's doesn't even rank first on LMArena..

by zb3 2y ago

Is o1 even that good? It's doesn't even rank first on LMArena..

Tostino 2y ago

For very specific tasks, yeah it's good. For my usual coding tasks...no way.

mistercow 2y ago

I haven’t been super impressed with it, and haven’t encountered any practical tasks I wanted to solve with an LLM where o1 worked any better than prompting 4o or Sonnet to use more extensive CoT. There might be some narrow band of practical problems in between what other LLMs can do and what o1 can’t, but I don’t think that really matters for most use cases, especially given how much slower it is. Day to day, you just don’t really want to prompt a model near the limits of its capabilities, because success quickly becomes a coin flip. So if a model needs five times as long to work, it needs to dramatically expand the range of problems that can be solved reliably.

torginus 2y ago
I think the true edge of CoT models will come from layman usability. While I can easily prompt Claude for examples and then manually modify the code to fill in the gaps, general domain knowledge and technical understanding is absolutely required from the human sitting in front of the screen. With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. So it's not so much about increased capability, but removing the expert human in the loop.
- itsoktocry 2y ago
  >With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. I just punched that prompt into Sonnet 3.5 and o1 and I wouldn't say that o1 is doing anything better than Sonnet. o1 certainly didn't "do it", it gave me a very broad outline of how to accomplish that, from "Define requirements" to "Test and deply on Vercel"
  - torginus 2y ago
    Honestly I had pretty good success with it. I wanted to try AWS batch for an example app after people here suggested it, and I had something running with like 2 prompts.

foundry27 2y ago

My understanding was that the metric for LMArena is that one answer is “better” than another, for a deliberately 100% subjective definition of better. My experience has been that typical LLMs will have more “preamble” to what they say, easing the reader (and priming themselves autoregressively) into answers with some relevant introduction of the subject, sometimes justifying the rationale and implications behind things. But for o1, that transient period and the underlying reasoning behind things is part of OpenAI’s special sauce, and they deliberately and aggressively take steps to hide it from users. o1 will get correct answers to hard problems more often than other models (look at the math/coding/hard subsections on the leaderboard, where anecdotal experiences aside, it is #1), and there’s a strong correlation between correctness and a high score in those domains because getting code or math “right” matters more than the justification or explanation. But in more general domains where there isn’t necessarily an objective right or wrong, I know the vibe matters a lot more to me, and that’s something o1 struggles with.

whimsicalism 2y ago

yes, it’s extremely good. don’t overindex on the lmsys arena, the median evaluator is kinda mid

IAmGraydon 2y ago

Good at what? It's great at breaking down complex problems into small, logical steps. Claude Sonnet 3.5 is still the best for coding. They can be leveraged together by using Aider's architect mode. It gives your request to the "architect" model first and returns a list of steps to implement your idea, but it doesn't write any code at that point. You then approve this and it sends the list to the coding model to actually write the code. This technique creates better quality code than any one model by itself. In Aider, you can assign any model you want as the architect and any other model as the coder. It's really great and I'm looking forward to the AI coding extensions for VSCode doing the same thing since I prefer to work in VSC than on the command line as is necessary with Aider. My only real problem with o1 is that it's ridiculously expensive, to the point that it makes no sense to use it for actual code. In architect mode, however, you can keep the costs under control as there are far fewer input/output tokens.