4 ms·

Well that really depends where you're caching the data. Is it a lot for caching in L1 on a chip somewhere? No that'd be wildly cheap. Is it a lot for "caching

by IanCal 2y ago

Well that really depends where you're caching the data. Is it a lot for caching in L1 on a chip somewhere? No that'd be wildly cheap. Is it a lot for "caching" on a tape somewhere? Yes. So where on this scale does keeping it quick to get to gpu memory lie? > That's two million times more expensive than the storage cost of standard S3 ( You're not comparing to s3 at all.

bastawhiz 2y ago
"RAM near a GPU" is ~the same cost as "RAM near literally any other piece of hardware". Even if it has to traverse the network, that's a fairly low, fairly fixed (in, say, the same rack) cost. Hell, it's probably even fast enough to use an NVME disk. Google can search the entire Internet in a fraction of a second, they can keep a million tokens within a few dozen milliseconds of a GPU for less than a dollar an hour.
- IanCal 2y ago
  Is that fast enough? And how much data is being stored? They're not storing the tokens you pass on but the activations after processing them. I'll take a wild stab that the activations for Claude 3.5 aren't anywhere near 4 meg.
  - bastawhiz 2y ago
    You've already frowned on my comparison to S3 but I think it's apt: it's many times more expensive than S3, but (for even a gigabyte of activations) it doesn't even need to be two orders of magnitude faster than standard S3. If you use the Elasticache pricing, which is $0.125/gb per hour, it's still eight times more expensive. So even if a million tokens is a full gigabyte of data, it's still almost an order of magnitude more expensive than an in-memory cache adjacent to the inference boxes. When your managed cache costs right times as much as a general purpose managed cache _in the cloud_, you've jumped the shark on pricing.
    - IanCal 2y ago
      I do not understand your first paragraph. It's more expensive than s3 and it's entirely different. It's cheaper than writing on vellum, and faster! > If you use the Elasticache pricing, which is $0.125/gb per hour, it's still eight times more expensive. So even if a million tokens is a full gigabyte of data Is it a gigabyte of data and is it fast enough? You've guessed 4mb and 1gb. What's the actual data size here? What speed do you need to get it into the GPU ram? The entire point here is to lower latency and costs so it has to be close and fast. Guessing at sizes isn't helping anything here.

by IanCal 2y ago

bastawhiz 2y ago

"RAM near a GPU" is ~the same cost as "RAM near literally any other piece of hardware". Even if it has to traverse the network, that's a fairly low, fairly fixed (in, say, the same rack) cost. Hell, it's probably even fast enough to use an NVME disk. Google can search the entire Internet in a fraction of a second, they can keep a million tokens within a few dozen milliseconds of a GPU for less than a dollar an hour.

IanCal 2y ago
Is that fast enough? And how much data is being stored? They're not storing the tokens you pass on but the activations after processing them. I'll take a wild stab that the activations for Claude 3.5 aren't anywhere near 4 meg.
- bastawhiz 2y ago
  You've already frowned on my comparison to S3 but I think it's apt: it's many times more expensive than S3, but (for even a gigabyte of activations) it doesn't even need to be two orders of magnitude faster than standard S3. If you use the Elasticache pricing, which is $0.125/gb per hour, it's still eight times more expensive. So even if a million tokens is a full gigabyte of data, it's still almost an order of magnitude more expensive than an in-memory cache adjacent to the inference boxes. When your managed cache costs right times as much as a general purpose managed cache _in the cloud_, you've jumped the shark on pricing.
  - IanCal 2y ago
    I do not understand your first paragraph. It's more expensive than s3 and it's entirely different. It's cheaper than writing on vellum, and faster! > If you use the Elasticache pricing, which is $0.125/gb per hour, it's still eight times more expensive. So even if a million tokens is a full gigabyte of data Is it a gigabyte of data and is it fast enough? You've guessed 4mb and 1gb. What's the actual data size here? What speed do you need to get it into the GPU ram? The entire point here is to lower latency and costs so it has to be close and fast. Guessing at sizes isn't helping anything here.