3 ms·

I haven’t been super impressed with it, and haven’t encountered any practical tasks I wanted to solve with an LLM where o1 worked any better than prompting 4o o

I haven’t been super impressed with it, and haven’t encountered any practical tasks I wanted to solve with an LLM where o1 worked any better than prompting 4o or Sonnet to use more extensive CoT. There might be some narrow band of practical problems in between what other LLMs can do and what o1 can’t, but I don’t think that really matters for most use cases, especially given how much slower it is. Day to day, you just don’t really want to prompt a model near the limits of its capabilities, because success quickly becomes a coin flip. So if a model needs five times as long to work, it needs to dramatically expand the range of problems that can be solved reliably.

torginus 2y ago
I think the true edge of CoT models will come from layman usability. While I can easily prompt Claude for examples and then manually modify the code to fill in the gaps, general domain knowledge and technical understanding is absolutely required from the human sitting in front of the screen. With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. So it's not so much about increased capability, but removing the expert human in the loop.
- itsoktocry 2y ago
  >With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. I just punched that prompt into Sonnet 3.5 and o1 and I wouldn't say that o1 is doing anything better than Sonnet. o1 certainly didn't "do it", it gave me a very broad outline of how to accomplish that, from "Define requirements" to "Test and deply on Vercel"
  - torginus 2y ago
    Honestly I had pretty good success with it. I wanted to try AWS batch for an example app after people here suggested it, and I had something running with like 2 prompts.

torginus 2y ago

I think the true edge of CoT models will come from layman usability. While I can easily prompt Claude for examples and then manually modify the code to fill in the gaps, general domain knowledge and technical understanding is absolutely required from the human sitting in front of the screen. With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. So it's not so much about increased capability, but removing the expert human in the loop.

itsoktocry 2y ago
>With o1, a layman can sit in front of the computer, and ask 'I want a website for tracking deliveries for my webshop and make it pretty', and the model will do it. I just punched that prompt into Sonnet 3.5 and o1 and I wouldn't say that o1 is doing anything better than Sonnet. o1 certainly didn't "do it", it gave me a very broad outline of how to accomplish that, from "Define requirements" to "Test and deply on Vercel"
- torginus 2y ago
  Honestly I had pretty good success with it. I wanted to try AWS batch for an example app after people here suggested it, and I had something running with like 2 prompts.