4 ms·

With caching that's 10 cents per million in. Most of the cheap open source models (which this claims to beat, except glm 4.6) have limited and not as effective

by Bolwin 1y ago

With caching that's 10 cents per million in. Most of the cheap open source models (which this claims to beat, except glm 4.6) have limited and not as effective caching. This could be massive.

logicchains 1y ago
$1/M is hardly a big improvement over GPT5's $1.250/M (or Gemini Pro's $1.5/M), and given how much worse Haiku is than those at any kind of difficult problem (or problems with a large context size), I can't imagine it being a particularly competitive alternative for coding. Especially for anything math/logic related, I find GPT5 and Gemini Pro to be significantly better even than Opus (which reflects in their models having won Olympiad prizes while Anthropic's have not).
- HarHarVeryFunny 1y ago
  GPT-5 is $10/M for output tokens, twice the cost of Haiku 4.5 at $5/M, despite Haiku apparently being better at some tasks (SWE Bench). I suppose it depends on how you are using it, but for coding isn't output cost more relevant than input - requirements in, code out ?
  - criemen 1y ago
    > I suppose it depends on how you are using it, but for coding isn't output cost more relevant than input - requirements in, code out ? Depends on what you're doing, but for modifying an existing project (rather than greenfield), input tokens >> output tokens in my experience.
  - logicchains 1y ago
    Unless you're working on a small greenfield project, you'll usually have 10s-100s of thousands of relevant words (~tokens) of relevant code in context for every query, vs a few hundred words of changes being output per query. Because most changes to an existing project are relatively small in scope.
Tiberium 1y ago
The funny thing is that even in this area Anthropic is behind other 3 labs (Google, OpenAI, xAI). It's the only one out of those 4 that requires you to manually set cache breakpoints, and the initial cache costs 25% more than usual context. The other 3 have fully free implicit caching. Although Google also offers paid, explicit caching. https://docs.claude.com/en/docs/build-with-claude/prompt-caching https://docs.claude.com/en/docs/build-with-claude/prompt-cac... https://ai.google.dev/gemini-api/docs/caching https://ai.google.dev/gemini-api/docs/caching https://platform.openai.com/docs/guides/prompt-caching https://platform.openai.com/docs/guides/prompt-caching https://docs.x.ai/docs/models#cached-prompt-tokens https://docs.x.ai/docs/models#cached-prompt-tokens
- tempusalaria 1y ago
  I vastly prefer the manual caching. There are several aspects of automatic caching that are suboptimal, with only moderately less developer burden. I don’t use Anthropic much but I wish the others had manual cache options
  - simonw 1y ago
    What's sub-optimal about the OpenAI approach, where you get 90% discount on tokens that you've previously sent within X minutes?
    - stavros 1y ago
      Is it wherever the tokens are, or is it the N first tokens they've seen before? Ie if my prompt is 99% the same, except for the first token, will it be cached?
      - simonw 1y ago
        The prefix has to be stable. If you are 99% the same but the first token is different it won't cache at all. You end up having to design your prompts to accommodate this.

by Bolwin 1y ago

With caching that's 10 cents per million in. Most of the cheap open source models (which this claims to beat, except glm 4.6) have limited and not as effective caching. This could be massive.

logicchains 1y ago

$1/M is hardly a big improvement over GPT5's $1.250/M (or Gemini Pro's $1.5/M), and given how much worse Haiku is than those at any kind of difficult problem (or problems with a large context size), I can't imagine it being a particularly competitive alternative for coding. Especially for anything math/logic related, I find GPT5 and Gemini Pro to be significantly better even than Opus (which reflects in their models having won Olympiad prizes while Anthropic's have not).

HarHarVeryFunny 1y ago
GPT-5 is $10/M for output tokens, twice the cost of Haiku 4.5 at $5/M, despite Haiku apparently being better at some tasks (SWE Bench). I suppose it depends on how you are using it, but for coding isn't output cost more relevant than input - requirements in, code out ?
- criemen 1y ago
  > I suppose it depends on how you are using it, but for coding isn't output cost more relevant than input - requirements in, code out ? Depends on what you're doing, but for modifying an existing project (rather than greenfield), input tokens >> output tokens in my experience.
- logicchains 1y ago
  Unless you're working on a small greenfield project, you'll usually have 10s-100s of thousands of relevant words (~tokens) of relevant code in context for every query, vs a few hundred words of changes being output per query. Because most changes to an existing project are relatively small in scope.

Tiberium 1y ago

The funny thing is that even in this area Anthropic is behind other 3 labs (Google, OpenAI, xAI). It's the only one out of those 4 that requires you to manually set cache breakpoints, and the initial cache costs 25% more than usual context. The other 3 have fully free implicit caching. Although Google also offers paid, explicit caching. https://docs.claude.com/en/docs/build-with-claude/prompt-caching https://docs.claude.com/en/docs/build-with-claude/prompt-cac... https://ai.google.dev/gemini-api/docs/caching https://ai.google.dev/gemini-api/docs/caching https://platform.openai.com/docs/guides/prompt-caching https://platform.openai.com/docs/guides/prompt-caching https://docs.x.ai/docs/models#cached-prompt-tokens https://docs.x.ai/docs/models#cached-prompt-tokens

tempusalaria 1y ago
I vastly prefer the manual caching. There are several aspects of automatic caching that are suboptimal, with only moderately less developer burden. I don’t use Anthropic much but I wish the others had manual cache options
- simonw 1y ago
  What's sub-optimal about the OpenAI approach, where you get 90% discount on tokens that you've previously sent within X minutes?
  - stavros 1y ago
    Is it wherever the tokens are, or is it the N first tokens they've seen before? Ie if my prompt is 99% the same, except for the first token, will it be cached?
    - simonw 1y ago
      The prefix has to be stable. If you are 99% the same but the first token is different it won't cache at all. You end up having to design your prompts to accommodate this.