3 ms·

That's the "prompt processing" speed, just reading the prompt. It can only generate text at 28 t/s according to that chart.

• behnamoh 3y ago
  no, prompt processing is 512 (look at pp).
  • sp332 3y ago
    512 means the prompt had 512 tokens total. The next column, labelled "t/s", shows tokens per second. The text generation benchmark is labelled "tg 64" because it generates 64 tokens for the benchmark.