3 ms·
ChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.
by tomrod 2mo ago
ChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.
- sixtyj 2mo agoAt such output speed, I wouldn’t expect reasoning. (But I didn’t know it, thanks.) 700 TPS with reasoning is awesome and it speeds things up. Cerebras as public traded company is worth keeping an eye what they produce.
- msdz 2mo ago> At such output speed, I wouldn’t expect reasoning. As the sibling comment to yours mentioned, if they had a reasoning model “hardware-ified” onto a custom chip (as is their plan for IIRC this or next year, a new ASIC), it’d output fast decode speeds for the regular output as well as reasoning sections. Both would be ≈equally fast.
- senordevnyc 2mo agoYeah, I thought reasoning was literally just chain of thought in the output token stream, with the model itself adding delimiters to indicate what part of the output is internal reasoning, and what part is an answer to the user. Is that wrong?
- beering 2mo agoYou are right, reasoning is unrelated to tokens per second.
- msdz 2mo agoNo, as the sibling comment mentioned, your understanding was correct there. What’s more, the only technical difference in speeds could be, and likely also is with the HC1 chip, between prefill (prompt processing) and decode (text generation) speeds. I don’t know whether it’s the case with Taalas’ chip, but in the “software-based” LLMs we typically see and use so far, those two stages hit different parts of a computer (processing/compute-bound vs. memory/bandwidth-bound).
- notfromhere 2mo agoAnything will be fast if you etch it straight to silicon
- dzhiurgis 2mo agoThe knowledge of ChatJimmy is terrible. Even Qwen on my iPhone is better.
- headPoet 2mo agoChatJimmy isn't a model, it's Llama 3.1 8B hardwired into silicon. The point isn't to be a good llm, but to showcase the speedup that's possible
- senordevnyc 2mo agoHaha, at first I thought you meant that the knowledge of the existence of an LLM that’s so fast is terrible because it’s ruined every other LLM for you!
- walrus01 2mo agowell, yeah, it's based on a 2+ year old tiny model. It's very much an alpha proof of concept that they can perma-bake an LLM into silicon. https://huggingface.co/meta-llama/Llama-3.1-8B https://huggingface.co/meta-llama/Llama-3.1-8B