2 ms·

With cache hit rates being effectively free, harnesses like Reasonix have let me do a month of work for less than 2 dollars. It's not even the subsidies making

With cache hit rates being effectively free, harnesses like Reasonix have let me do a month of work for less than 2 dollars. It's not even the subsidies making it cheap, American providers like Digital Ocean or Cloudflare host the same model with similar pricing.

ForHackernews 4mo ago
I think this is very likely and something that everyone seems to be missing when valuing these AI firms. AI is not the new industrial revolution, it's the new cloud VM: a very useful commodity software offering.
- antonvs 4mo ago
  The parallels to the Industrial Revolution are so close that we even have a new generation of Luddites. (Not saying they don’t have some valid points; so did the original group.) The reason it’s like the Industrial Revolution is simply that there’s no question it’s going to completely transform jobs. It can make a very similar difference to the difference between a craftsman and a factory worker. The latter is massively more productive.
Scaevolus 4mo ago
Cloudflare's Deepseek V4 Pro prices are 4x more than Deepseek's for input and output tokens, and 100x more for cached input tokens, which is crucial for the tool uses of agents which cause multi-turn conversations.
- arikrahman 4mo ago
  Cache hit is less than a cent with Deepseek Flash and 3 cents with Cloudflare, it's free vs almost free. Where are you finding the statistics on Deepseek Pro? I don't see Cloudflare as a provider on openrouter for Pro, only flash.
pjc50 4mo ago
How does caching help here? How much repetition is there in queries?
- AnthonyMouse 4mo ago
  It probably depends on what you're doing, but imagine you're something in the shape of a search engine. How many user queries are unique vs. the same thing someone else searched for an hour ago?
- jcparkyn 4mo ago
  Agent loops (particularly coding agents) have a huge amount of repetition, because the entire context is included in every model request. So long as it's at the start of the input and doesn't change, it will be able to hit the KV cache (assuming the model provider actually has the prefix in cache). This only works because prompt caching is done by matching prefixes, not the entire input.
- crazylogger 4mo ago
  In a typical agent loop your N-th LLM request naturally becomes prefix for the (N+1)-th request. As the thread grows longer, cache hit rate converges to 100% and unit pricing for cached tokens is 10-100x cheaper.
- arikrahman 4mo ago
  Their blob explains it best, although this is a link to an older version design: https://github.com/esengine/DeepSeek-Reasonix/blob/v1/docs/ARCHITECTURE.md#pillar-1--cache-first-loop https://github.com/esengine/DeepSeek-Reasonix/blob/v1/docs/A... From my understanding if previous tokens are frozen and guaranteed to be immutable you can leverage that.

ForHackernews 4mo ago

I think this is very likely and something that everyone seems to be missing when valuing these AI firms. AI is not the new industrial revolution, it's the new cloud VM: a very useful commodity software offering.

antonvs 4mo ago
The parallels to the Industrial Revolution are so close that we even have a new generation of Luddites. (Not saying they don’t have some valid points; so did the original group.) The reason it’s like the Industrial Revolution is simply that there’s no question it’s going to completely transform jobs. It can make a very similar difference to the difference between a craftsman and a factory worker. The latter is massively more productive.

Scaevolus 4mo ago

Cloudflare's Deepseek V4 Pro prices are 4x more than Deepseek's for input and output tokens, and 100x more for cached input tokens, which is crucial for the tool uses of agents which cause multi-turn conversations.

arikrahman 4mo ago
Cache hit is less than a cent with Deepseek Flash and 3 cents with Cloudflare, it's free vs almost free. Where are you finding the statistics on Deepseek Pro? I don't see Cloudflare as a provider on openrouter for Pro, only flash.

pjc50 4mo ago

How does caching help here? How much repetition is there in queries?

AnthonyMouse 4mo ago
It probably depends on what you're doing, but imagine you're something in the shape of a search engine. How many user queries are unique vs. the same thing someone else searched for an hour ago?
jcparkyn 4mo ago
Agent loops (particularly coding agents) have a huge amount of repetition, because the entire context is included in every model request. So long as it's at the start of the input and doesn't change, it will be able to hit the KV cache (assuming the model provider actually has the prefix in cache). This only works because prompt caching is done by matching prefixes, not the entire input.
crazylogger 4mo ago
In a typical agent loop your N-th LLM request naturally becomes prefix for the (N+1)-th request. As the thread grows longer, cache hit rate converges to 100% and unit pricing for cached tokens is 10-100x cheaper.
arikrahman 4mo ago
Their blob explains it best, although this is a link to an older version design: https://github.com/esengine/DeepSeek-Reasonix/blob/v1/docs/ARCHITECTURE.md#pillar-1--cache-first-loop https://github.com/esengine/DeepSeek-Reasonix/blob/v1/docs/A... From my understanding if previous tokens are frozen and guaranteed to be immutable you can leverage that.