3 ms·

What people don't realize is that cache is free, well not free, but compared to the compute required to recompute it? Relatively free. If you remove the cach

What people don't realize is that cache is *free*, well not free, but compared to the compute required to recompute it? Relatively free. If you remove the cached token cost from pricing the overall api usage drops from around $5000 to $800 (or $200 per week) on the $200 max subscription. Still 4x cheaper over API, but not costing money either - if I had to guess it's break even as the compute is most likely going idle otherwise.

eru 8mo ago
> [...] if I had to guess it's break even as the compute is most likely going idle otherwise. Why would it go idle? It would go to their next best use. At least they could help with model training or let their researchers run experiments etc.
- himata4113 8mo ago
  inference compute is vastly different versus training, also it has to stay hot in vram which probably takes up most of it. There is limited use for THAT much compute as well, they are running things like claude code compiler and even then they're scratching the surface of the amount of compute they have. Training currently requires nvidia's latest and greatest for the best models (they also use google TPU's now which are also technically the latest and greatest? However, they're more of a dual purpose than anything afaik so that would be a correct assesment in that case) Inference can run on a hot potato if you really put your mind to it
  - eru 8mo ago
    They can run any number of inference experiments. Like a lot of the alignment work they have going on. I am not saying this would be a great use of their compute, but idle is far from the only alternative. (Unless electricity is the binding constraint?)
    - himata4113 8mo ago
      Electricity is charged whenever you use it or not, so very unlikely, but sure, they can find uses for it. Although they are not going to make that much money compared to claude code subscriptions.
      - eru 8mo ago
        > Electricity is charged whenever you use it or not, [...] Huh, what? You know you can turn off unused equipment, and at least my nvidia GPU can use more or less Watts even when turned on? Or does Anthropic have a flatline deal for electricity and cooling?
        himata4113 8mo ago
        in datacenters power allocation is a fixed cost.
        eru 8mo ago
        If you are big enough, you can and will negotiate.
        himata4113 8mo ago
        the datacenter has a fixed cost for power, industrial power is not consumer power especially at large scale. Scale really kicks in if you own your power plant (ex: hydro, wind, solar).
        eru 8mo ago
        Data centres are more complicated than that. For an example, even if you have a fixed power budget at the data centre level, you still have opportunity costs: if you turn some unused GPUs off, you can run other things hotter.
  - rafaelmn 8mo ago
    I think I've heard multiple time that a large % of training compute for SoTA models is inference to generate training tokens, this is bound to happen with RL training
criemen 8mo ago
> What people don't realize is that cache is free I'm incredibly salty about this - they're essentially monetizing intensely something that allows them to sell their inference at premium prices to more users - without any caching, they'd have much less capacity available.
mike_hearn 8mo ago
Cache definitely isn't free! We're in a global RAM shortage and KV caches sit around consuming RAM in the hope that there will be a hit. The gamble with caching is to hold a KV cache in the hope that the user will (a) submit a prompt that can use it and (b) that will get routed to the right server which (c) won't be so busy at the time it can't handle the request. KV caches aren't small so if you lose that bet you've lost money (basically, the opportunity cost of using that RAM for something else).
- otterley 8mo ago
  Why do you believe that caches are held in RAM? They don’t need RAM performance, and disk is orders of magnitude cheaper.
  - mike_hearn 8mo ago
    Because OpenAI specifically say that: https://developers.openai.com/api/docs/guides/prompt-caching/ https://developers.openai.com/api/docs/guides/prompt-caching... > When using the in-memory policy, cached prefixes generally remain active for 5 to 10 minutes of inactivity, up to a maximum of one hour. In-memory cached prefixes are only held within volatile GPU memory. You can opt-in to storing the caches on local disk but it's not the default. I haven't done the calculations for why they do this, but given that disaggregated parallel prefill and RDMA can recompute the KV cache very fast, you'd need a huge amount of bandwidth from disk to beat it (and flash drives wear out!).
- himata4113 8mo ago
  free relative to gpu cost even at these costs
agenthustler 8mo ago
[dead]

What people don't realize is that cache is *free*, well not free, but compared to the compute required to recompute it? Relatively free. If you remove the cach

eru 8mo ago

> [...] if I had to guess it's break even as the compute is most likely going idle otherwise. Why would it go idle? It would go to their next best use. At least they could help with model training or let their researchers run experiments etc.

himata4113 8mo ago
inference compute is vastly different versus training, also it has to stay hot in vram which probably takes up most of it. There is limited use for THAT much compute as well, they are running things like claude code compiler and even then they're scratching the surface of the amount of compute they have. Training currently requires nvidia's latest and greatest for the best models (they also use google TPU's now which are also technically the latest and greatest? However, they're more of a dual purpose than anything afaik so that would be a correct assesment in that case) Inference can run on a hot potato if you really put your mind to it
- eru 8mo ago
  They can run any number of inference experiments. Like a lot of the alignment work they have going on. I am not saying this would be a great use of their compute, but idle is far from the only alternative. (Unless electricity is the binding constraint?)
  - himata4113 8mo ago
    Electricity is charged whenever you use it or not, so very unlikely, but sure, they can find uses for it. Although they are not going to make that much money compared to claude code subscriptions.
    - eru 8mo ago
      > Electricity is charged whenever you use it or not, [...] Huh, what? You know you can turn off unused equipment, and at least my nvidia GPU can use more or less Watts even when turned on? Or does Anthropic have a flatline deal for electricity and cooling?
      - himata4113 8mo ago
        in datacenters power allocation is a fixed cost.
        eru 8mo ago
        If you are big enough, you can and will negotiate.
        himata4113 8mo ago
        the datacenter has a fixed cost for power, industrial power is not consumer power especially at large scale. Scale really kicks in if you own your power plant (ex: hydro, wind, solar).
        eru 8mo ago
        Data centres are more complicated than that. For an example, even if you have a fixed power budget at the data centre level, you still have opportunity costs: if you turn some unused GPUs off, you can run other things hotter.
- rafaelmn 8mo ago
  I think I've heard multiple time that a large % of training compute for SoTA models is inference to generate training tokens, this is bound to happen with RL training

criemen 8mo ago

> What people don't realize is that cache is free I'm incredibly salty about this - they're essentially monetizing intensely something that allows them to sell their inference at premium prices to more users - without any caching, they'd have much less capacity available.

mike_hearn 8mo ago

Cache definitely isn't free! We're in a global RAM shortage and KV caches sit around consuming RAM in the hope that there will be a hit. The gamble with caching is to hold a KV cache in the hope that the user will (a) submit a prompt that can use it and (b) that will get routed to the right server which (c) won't be so busy at the time it can't handle the request. KV caches aren't small so if you lose that bet you've lost money (basically, the opportunity cost of using that RAM for something else).

otterley 8mo ago
Why do you believe that caches are held in RAM? They don’t need RAM performance, and disk is orders of magnitude cheaper.
- mike_hearn 8mo ago
  Because OpenAI specifically say that: https://developers.openai.com/api/docs/guides/prompt-caching/ https://developers.openai.com/api/docs/guides/prompt-caching... > When using the in-memory policy, cached prefixes generally remain active for 5 to 10 minutes of inactivity, up to a maximum of one hour. In-memory cached prefixes are only held within volatile GPU memory. You can opt-in to storing the caches on local disk but it's not the default. I haven't done the calculations for why they do this, but given that disaggregated parallel prefill and RDMA can recompute the KV cache very fast, you'd need a huge amount of bandwidth from disk to beat it (and flash drives wear out!).
himata4113 8mo ago
free relative to gpu cost even at these costs

agenthustler 8mo ago

[dead]