4 ms·
In that case I'm misunderstanding you. Are you saying that it's "BS" that they are reaching ~1k tokens/s? If so, you may be misunderstanding what a Cerebras mac
by danpalmer 2y ago
In that case I'm misunderstanding you. Are you saying that it's "BS" that they are reaching ~1k tokens/s? If so, you may be misunderstanding what a Cerebras machine is. Also 8xH100 is still ~half the price of a single Cerebras machine, and that's even accounting for H100s being massively over priced. You've got easily twice the value in a Cerebras machine, they have nearly 1m cores on a single die.
- sam_dam_gai 2y agoHa ha. He probably means ”at a batch size of 1”, i.e. not even using some amortization tricks to get better numbers.
- danpalmer 2y agoAh! That does make more sense!