4 ms·
The title says "on commodity GPUs" but the only GPU mentioned (and the only one with benchmarks) are Nvidia H100s ($30K+ on Ebay)? Do these run on actual commo
by nacs 2y ago
The title says "on commodity GPUs" but the only GPU mentioned (and the only one with benchmarks) are Nvidia H100s ($30K+ on Ebay)?
Do these run on actual commodity GPUs like RTX 3090s and what kind of tokens/sec is expected on those?
Also, there's no paper, no open weights, no code. Just an API?
Companies like Groq and Cerebras already hit these kind of numbers over a year ago so I'm not seeing what's HN worthy here.
- dragonwriter 2y ago“Commodity” and “consumer” are not the same thing; H100 is commodity but not consumer, RTX 3090 is consumer and commodity.
- halJordan 2y agoAn h100 is absolutely not a commodity product. A commodity product is one that is fungible (and that interchangeability is also reflected by second order effects like price). A 4090 is replacable with a 7900xtx. Not the same with an h100 and an instinct mi360
- volodia 2y agoThat's a good point. In this context, we've been using "commodity GPUs" to refer to standard Nvidia hardware, in contrast to specialized chips like Groq and Cerebras. While these chips also achieve fast speeds, they are not nearly as ubiquitous as Nvidia GPUs. We think that matching their performance on standard Nvidia hardware can make AI much more affordable. We also support any GPUs, not just H100's. We're going to be releasing a tech report soon, stay tuned!