3 ms·

Looking at their blog, they in fact ran gpt-oss-120b: https://furiosa.ai/blog/serving-gpt-oss-120b-at-5-8-ms-tpot-with-two-rngd-cards-compiler-optimizations-in-

by sanxiyn 9mo ago

Looking at their blog, they in fact ran gpt-oss-120b: https://furiosa.ai/blog/serving-gpt-oss-120b-at-5-8-ms-tpot-with-two-rngd-cards-compiler-optimizations-in-practice https://furiosa.ai/blog/serving-gpt-oss-120b-at-5-8-ms-tpot-... I think Llama 3 focus mostly reflects demand. It may be hard to believe, but many people aren't even aware gpt-oss exists.

reactordev 9mo ago
Many are aware, just can’t offload it onto their hardware. The 8B models are easier to run on an RTX to compare it to local inference. What llama does on an RTX 5080 at 40t/s, Furiosa should do at 40,000t/s or whatever… it’s an easy way to have a flat comparison across all the different hardware llama.cpp runs on.
zmmmmm 9mo ago
Now I'm interested ... It still kind of makes the point that you are stuck with a very limited range of models that they are hand implementing. But at least it's a model I would actually use. Give me that in a box I can put in a standard data center with normal power supply and I'm definitely interested. But I want to know the cost :-)
nl 9mo ago
> we demonstrated running gpt-oss-120b on two RNGD chips [snip] at 5.8 ms per output token That's 86 token/second/chip By comparison, a H100 will do 2390 token/second/GPU Am I comparing the wrong things somehow? [1] https://inferencemax.semianalysis.com/ https://inferencemax.semianalysis.com/
- binary132 9mo ago
  I thought they were saying it was more efficient, as in tokens per watt. I didn’t see a direct comparison on that metric but maybe I didn’t look well enough.
  - nl 9mo ago
    Probably. Companies sell on efficiency when they know they lose on performance.
    - binary132 9mo ago
      Right, but datacenters also very much operate on electrical cost so it’s not without merit.
    - tormeh 9mo ago
      If you have an efficient chip you can just have more of them and come out ahead. This isn't a CPU where single core performance is all that important.
      - avereveard 9mo ago
        Eh if there's a human on the other side single stream performance is going to matter to them.
      - fennecfoxy 9mo ago
        Only if the price is right...
- sanxiyn 9mo ago
  I think you are comparing latency with throughput. You can't take the inverse of latency to get throughput because concurrency is unknown. But then, RNGD result is probably with concurrency=1.

by sanxiyn 9mo ago

reactordev 9mo ago

Many are aware, just can’t offload it onto their hardware. The 8B models are easier to run on an RTX to compare it to local inference. What llama does on an RTX 5080 at 40t/s, Furiosa should do at 40,000t/s or whatever… it’s an easy way to have a flat comparison across all the different hardware llama.cpp runs on.

zmmmmm 9mo ago

Now I'm interested ... It still kind of makes the point that you are stuck with a very limited range of models that they are hand implementing. But at least it's a model I would actually use. Give me that in a box I can put in a standard data center with normal power supply and I'm definitely interested. But I want to know the cost :-)

nl 9mo ago

> we demonstrated running gpt-oss-120b on two RNGD chips [snip] at 5.8 ms per output token That's 86 token/second/chip By comparison, a H100 will do 2390 token/second/GPU Am I comparing the wrong things somehow? [1] https://inferencemax.semianalysis.com/ https://inferencemax.semianalysis.com/

binary132 9mo ago
I thought they were saying it was more efficient, as in tokens per watt. I didn’t see a direct comparison on that metric but maybe I didn’t look well enough.
- nl 9mo ago
  Probably. Companies sell on efficiency when they know they lose on performance.
  - binary132 9mo ago
    Right, but datacenters also very much operate on electrical cost so it’s not without merit.
  - tormeh 9mo ago
    If you have an efficient chip you can just have more of them and come out ahead. This isn't a CPU where single core performance is all that important.
    - avereveard 9mo ago
      Eh if there's a human on the other side single stream performance is going to matter to them.
    - fennecfoxy 9mo ago
      Only if the price is right...
sanxiyn 9mo ago
I think you are comparing latency with throughput. You can't take the inverse of latency to get throughput because concurrency is unknown. But then, RNGD result is probably with concurrency=1.