3 ms·
> hardware to run them Costs a few hundred thousand per server, it's a huge expense if you want it at your home but a rounding error for most organizations.
by fooker 8mo ago
> hardware to run them
Costs a few hundred thousand per server, it's a huge expense if you want it at your home but a rounding error for most organizations.
- bottlepalm 8mo agoYou're buying what exactly for a few hundred thousand? and running what model on it? to support how many users? at what tps?
- fooker 8mo agoNot every use case is a cloud provider or tech giant. Newer Blackwell does 200+ tokens per second on the largest models and tens of thousands on the smaller models. Most military applications require fast smaller models, I'd imagine. Also, custom chips are reportedly approaching an order of magnitude more for the price. It's a matter of availability right now, but that will be solved at some point.