3 ms·
14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
by edot 3mo ago
14 seconds per token? Not tokens per second. Seconds per token? That’s nowhere near the cusp!
- dotancohen 3mo agoIt's need it to be an order of magnitude cheaper, but for some queries ("What to discuss at tomorrow's meeting") I can wait 12+ hours.
- throw10920 3mo agoI'd rather just use my actual human brain to compute the answer at that point. I don't see the value at throughput that is this low.
- dotancohen 3mo agoI'd rather not be thinking about tomorrow's meeting all night! Just trust "the robot has got this"... and scramble in the morning when the robot has failed. But at least I got a good night's sleep!
- antonvs 3mo agoAt that rate, it would take me a mere 15 days to generate the number of tokens I typically use in a day. "Yes, I'm using AI to speed up development. I'll submit that PR in two weeks time!"
- adrian_b 3mo agoThat speed is significantly lower that what can be attained with streaming from SSDs. Not only in big desktops, but even in most recent mini-PCs, it is possible to read concurrently from one PCIe 5.0 SSDs and one PCIe 4.0 SSD, at a total sustained reading throughput of around 20 GB/s. With an optimized inference implementation, it should be possible to overlap completely the computations with streaming weights from SSDs. This should improve the inference speed to around at least 1 token per second, on a cheap computer, under $2000 even at the current super-inflated prices. There are enough tasks where this would be useful. Obviously one should use for most tasks a fast small LLM and use the big one only when this actually saves time.