4 ms·
6 to 8 t/s is decent for a GPU-less build, but speaking from experience... it will feel slow. Usable - but slow. Especially because these models <think> for a
by horsawlarway 2y ago
6 to 8 t/s is decent for a GPU-less build, but speaking from experience... it will feel slow. Usable - but slow.
Especially because these models <think> for a long bit before actually answering, so they generate for a longer period. (to be clear - I find the <think> section useful, but it also means waiting for more tokens)
Personally - I end up moving down to lower quality models (quants/less params) until I hit about 15 tokens/second. For chat, that seems to be the magical spot where I stop caring and it's "fast enough" to keep me engaged.
For inline code helpers (ex - copilot) you really need to be up near 30 tokens/second to make it feel fast enough to be helpful.
- k__ 2y agoWould DeepSeek-V3 be cheaper?
- sigmoid10 2y agoOnly in the sense that it will generate fewer tokens.
- guerrilla 2y agoThis is the same I get from a Core i5-9400 for smaller models. Is there no prosumer board that can take that much RAM? There must be a ThreadRipper that can do it, right? Why did he need EPYC?
- hollerith 2y agoIt's not just the amount of RAM: RAM bandwidth also matters.
- UncleOxidant 2y agoRAM channels is key here. That board has 24 channels of DDR5 RAM. A lot of lowend boards only have 2 channels. Most come in at 4. Some high end boards have 8, so 24 channels is quite wide.
- UncleOxidant 2y agoIs a token a character or a word? I don't think I can read 30 words/second so 8/second would be fine. But if it's characters then yes, it would be slow.
- neoromantique 2y agoSomething in between. example: https://gpt-tokenizer.dev/ https://gpt-tokenizer.dev/
- UncleOxidant 2y agoLooks like for the most part it's words, but some longer words get broken at prefix/suffix boundaries.
- ryao 2y agoIt typically somewhere between 3 to 5 characters.
- horsawlarway 2y agoIt varies a lot by model, tokenizer, and language. For DeekSeek 1 token ~= 3 English characters. See: https://api-docs.deepseek.com/quick_start/token_usage/ https://api-docs.deepseek.com/quick_start/token_usage/ It comes out to around 1-3 words/second. This is not so slow that it's maddening (ex 2 token/second is frustratingly slow, like walk away and make coffee while it's answering slow), but it's still slow enough to make it hard to functionally use and not break flow state. You get bored and distracted reading at that pace.
- SkyPuncher 2y agoI've been with RooCode the slower token speeds are acceptable. I'll ask it to do a task, then it will do it in a few steps and notify me when it's done. I use that time to take care of something else.