2 ms·
$5000 hardware can run GPT 120B at 40-60tps. Which is plenty to code actively through the day. You could instead use Cerebras to get 3,000 tps; but people real
by incomingpain 1y ago
$5000 hardware can run GPT 120B at 40-60tps. Which is plenty to code actively through the day.
You could instead use Cerebras to get 3,000 tps; but people really dont do that according to stats. 100x the speed but its not like everyone is rushing to their service.
The speed of LLMs after the advent of MOE has hit a good spot. What we now need is smarter.