3 ms·

I don't want to be a jerk but 31t/s prefill is basically unusable in an agentic situation. A mere 10k in context and you're sitting there for 5+ minutes before

by xienze 5mo ago

I don't want to be a jerk but 31t/s prefill is basically unusable in an agentic situation. A mere 10k in context and you're sitting there for 5+ minutes before the first token is generated.

aiscoming 5mo ago
if it's just the coding agent system prompt and tools, you can cache that
- xienze 5mo ago
  Yeah the problem is that's just the start of the context. There's, you know, all the tool call results and file reads and stuff.
  - deleted 5mo ago
    [deleted]
fgfarben 5mo ago
That prefill number isn't right. M4 Max hits 200-300: https://github.com/antirez/ds4/blob/main/speed-bench/m4_max_ts.svg https://github.com/antirez/ds4/blob/main/speed-bench/m4_max_...
- hadlock 5mo ago
  M5 studio is gonna sell like hot cakes
throwdbaaway 5mo ago
Hah, that's because the prompt itself was only about 30 tokens. We need a much bigger prompt to properly test PP.