It says at the top that prompts are processed with a batch size of 512, while new tokens can only be generated one at a time with batch size of 1. Prompt proce

It says at the top that prompts are processed with a batch size of 512, while new tokens can only be generated one at a time with batch size of 1. Prompt processing doesn't actually need to compute logits for each new token, just cache the KV values and so is much faster than actual inference.