3 ms·
Shouldn't the right benchmark be performance per watt? It's easy enough to add more chips to do LLM training or inference in parallel. Maybe the benchmark shou
by zhyder 2y ago
Shouldn't the right benchmark be performance per watt? It's easy enough to add more chips to do LLM training or inference in parallel.
Maybe the benchmark should be performance per $... though I suspect power consumption will eclipse the cost of purchasing the chips from NVDA or AMD (and costs of chips will vary over time and with discounts). EDIT: was wrong on eclipsing; still am looking for a more durable benchmark (performance per billion transistors?) given it's suspected NVDA's chips are over-priced due to demand outstripping supply for now, and AMD's are under- to get a foothold in this market.
- mmoskal 2y agoNot quite. Assume 1kW power consumption (with cooling). At $0.08/kWh (avarage US industrial rate) this is $700 per year. Adjust for more cooling etc and for say 5 years of usage but you still won't be anywhere near the $25k MSRP for H100.