4 ms·

Yes but prompt evaluation is far faster than inference as it can be done (mostly) in parallel, so I don't think that's true.

danenania 1y ago
The problem is that input token cost dominates output token cost for the majority of tasks. Once you've given the model your prompt and are reading the first output token for classification, you've already paid most of the cost of just prompting it directly. That said, there could definitely be exceptions for short prompts where output costs dominate input costs. But these aren't usually the interesting use cases.
- redox99 1y ago
  That's usually not the case for thinking models. And usually hard problems have a very short prompt.
  - danenania 1y ago
    For me personally (using mostly for coding and project planning) it's nearly always the case, including with thinking models. I'm usually pasting in a bunch of files, screenshots, etc., and having long conversations. Input nearly always heavily dominates output. I don't disagree that there are hard problems which use short prompts, like math homework problems etc., but they mostly aren't what I would categorize as "real work". But of course I can only speak to my own experience /shrug.
    - redox99 1y ago
      Yeah coding is definitely a situation where context is usually very very large. But at the same time in those situations something like Sonnet is fine.
- energy123 1y ago
  No, you're talking about costs to user, which are oversimplifications of the costs that providers bear. One output token with a million input tokens is incredibly cheap for providers
  - danenania 1y ago
    > One output token with a million input tokens is incredibly cheap for providers Source? Afaik this is incorrect.
    - danielbln 1y ago
      Chevk out any LLM API providers pricing. Output tokens are always significantly more expensive than input (which can also be cached).
      - danenania 1y ago
        Input tokens usually dominate output tokens by a lot more than 2x though. It’s often 10x or more input. It can even easily be 100x or more. Again in realistic workflows. Caching does help the situation, but you always at least pay the initial cache write. And prompts need to be structured carefully to be cacheable. It’s not a free lunch.

danenania 1y ago

The problem is that input token cost dominates output token cost for the majority of tasks. Once you've given the model your prompt and are reading the first output token for classification, you've already paid most of the cost of just prompting it directly. That said, there could definitely be exceptions for short prompts where output costs dominate input costs. But these aren't usually the interesting use cases.

redox99 1y ago
That's usually not the case for thinking models. And usually hard problems have a very short prompt.
- danenania 1y ago
  For me personally (using mostly for coding and project planning) it's nearly always the case, including with thinking models. I'm usually pasting in a bunch of files, screenshots, etc., and having long conversations. Input nearly always heavily dominates output. I don't disagree that there are hard problems which use short prompts, like math homework problems etc., but they mostly aren't what I would categorize as "real work". But of course I can only speak to my own experience /shrug.
  - redox99 1y ago
    Yeah coding is definitely a situation where context is usually very very large. But at the same time in those situations something like Sonnet is fine.
energy123 1y ago
No, you're talking about costs to user, which are oversimplifications of the costs that providers bear. One output token with a million input tokens is incredibly cheap for providers
- danenania 1y ago
  > One output token with a million input tokens is incredibly cheap for providers Source? Afaik this is incorrect.
  - danielbln 1y ago
    Chevk out any LLM API providers pricing. Output tokens are always significantly more expensive than input (which can also be cached).
    - danenania 1y ago
      Input tokens usually dominate output tokens by a lot more than 2x though. It’s often 10x or more input. It can even easily be 100x or more. Again in realistic workflows. Caching does help the situation, but you always at least pay the initial cache write. And prompts need to be structured carefully to be cacheable. It’s not a free lunch.