25 ms·

576 CS-3 nodes costs around $900 million, which is $1.56 million per node. It takes 4 nodes to serve one 70B model or $6.24m. It is unclear how many requests t

576 CS-3 nodes costs around $900 million, which is $1.56 million per node. It takes 4 nodes to serve one 70B model or $6.24m. It is unclear how many requests they can serve concurrently for this, they only report on tokens. A cluster of 128 MI300x, which has a combined 24,576GB and can serve up a whole ton of models and users, 4 racks total, is in the ~$5m range, if you don't go big on networking/disk/ram (which you don't need for inference anyway). While speed might be an issue here, I don't think people are going to be able to justify the price for the speed (always the tradeoff) unless they can get their costs down significantly.

modeless 2y ago
You are right assuming that model capabilities are determined only by model size. But consider that OpenAI is saying they have a way of scaling intelligence with inference time compute, not just model size. If that proves out, reducing latency per output token potentially becomes as valuable as or even possibly more valuable than scaling model size. Speed becomes intelligence. And Cerebras has 1/10 the latency per token of anything else.
krasin 2y ago
You're correct on $/bandwidth. The point about low latency continues to be ignored, though.
- menaerus 2y ago
  It's maybe because the assumption about low latency because everything fits in SRAM is not valid? CS-1 had 18G of SRAM, CS-2 extended it to 40G and CS-3 has 44G of SRAM. None of these is sufficient to run the inference of Llama 70B and much less so of even larger models.
  - latchkey 2y ago
    Exactly. Latency is less relevant if you have to have 4 literal servers (each taking up a whole rack) to push out one single 70B model and we don't know how many concurrent user requests that actually services (probably 1).
arisAlexis 2y ago
That's a benchmark or shower thought?
campers 2y ago
On Google Cloud a server with 8 TPU v5e will do 2175 token/seconds on Llama2 70B. https://cloud.google.com/blog/products/compute/updates-to-ai-hypercomputer-software-stack/ https://cloud.google.com/blog/products/compute/updates-to-ai... From https://cloud.google.com/tpu/pricing https://cloud.google.com/tpu/pricing and https://cloud.google.com/vertex-ai/pricing#prediction-prices https://cloud.google.com/vertex-ai/pricing#prediction-prices (search for ct5lp-hightpu-8t on the page) the cost for that appears to be $11.04/hr which is just under $100k for a year. Or half that on a 3-year commit. That seems like a better deal than millions for a few CS-3 nodes. And they've just announced the v6 TPU: Compared to TPU v5e, Trillium delivers: Over 4x improvement in training performance Up to 3x increase in inference throughput A 67% increase in energy efficiency An impressive 4.7x increase in peak compute performance per chip Double the High Bandwidth Memory (HBM) capacity Double the Interchip Interconnect (ICI) bandwidth https://cloud.google.com/blog/products/compute/trillium-sixth-generation-tpu-is-in-preview/ https://cloud.google.com/blog/products/compute/trillium-sixt...

modeless 2y ago

You are right assuming that model capabilities are determined only by model size. But consider that OpenAI is saying they have a way of scaling intelligence with inference time compute, not just model size. If that proves out, reducing latency per output token potentially becomes as valuable as or even possibly more valuable than scaling model size. Speed becomes intelligence. And Cerebras has 1/10 the latency per token of anything else.

krasin 2y ago

You're correct on $/bandwidth. The point about low latency continues to be ignored, though.

menaerus 2y ago
It's maybe because the assumption about low latency because everything fits in SRAM is not valid? CS-1 had 18G of SRAM, CS-2 extended it to 40G and CS-3 has 44G of SRAM. None of these is sufficient to run the inference of Llama 70B and much less so of even larger models.
- latchkey 2y ago
  Exactly. Latency is less relevant if you have to have 4 literal servers (each taking up a whole rack) to push out one single 70B model and we don't know how many concurrent user requests that actually services (probably 1).

arisAlexis 2y ago

That's a benchmark or shower thought?

campers 2y ago

On Google Cloud a server with 8 TPU v5e will do 2175 token/seconds on Llama2 70B. https://cloud.google.com/blog/products/compute/updates-to-ai-hypercomputer-software-stack/ https://cloud.google.com/blog/products/compute/updates-to-ai... From https://cloud.google.com/tpu/pricing https://cloud.google.com/tpu/pricing and https://cloud.google.com/vertex-ai/pricing#prediction-prices https://cloud.google.com/vertex-ai/pricing#prediction-prices (search for ct5lp-hightpu-8t on the page) the cost for that appears to be $11.04/hr which is just under $100k for a year. Or half that on a 3-year commit. That seems like a better deal than millions for a few CS-3 nodes. And they've just announced the v6 TPU: Compared to TPU v5e, Trillium delivers: Over 4x improvement in training performance Up to 3x increase in inference throughput A 67% increase in energy efficiency An impressive 4.7x increase in peak compute performance per chip Double the High Bandwidth Memory (HBM) capacity Double the Interchip Interconnect (ICI) bandwidth https://cloud.google.com/blog/products/compute/trillium-sixth-generation-tpu-is-in-preview/ https://cloud.google.com/blog/products/compute/trillium-sixt...