2 ms·
I checked the fine print on the product website: by “up to 4x faster LLM prompt processing,” they’re specifically referring to time to first token. So it’s not
by otterley 8mo ago
I checked the fine print on the product website: by “up to 4x faster LLM prompt processing,” they’re specifically referring to time to first token. So it’s not about token generation rate (tokens per second).
- jasonjmcghee 8mo agoIt would probably be worth finding a more friendly way to market this, but it's a reasonable / accurate way to say it. The prompt processing sped up. Not the output generation. M4 was notoriously slow at this compared to DGX etc.
- aurareturn 8mo agoYes. This is known. They added neural accelerators, aka Tensor core equivalent, in the GPU. This will make prompt processing competitive vs similar class GPUs.
- solarkraft 8mo agoIt’s a big deal! Prompt processing was previously the Mac’s weak point. Sure, output generation matters for file recital in programming, but in general conversation I’d rather have it output a short answer anyway (after extensive processing by a smart model).
- otterley 8mo agoGeneral conversation is already free with all the major providers (Claude, ChatGPT, etc.). That's not where the major gains in productivity lie.