2 ms·
They really do compute just a single token at a time, but the input to that decision is the number of tokens in the context window - so that's usually around 8,
by simonw 3y ago
They really do compute just a single token at a time, but the input to that decision is the number of tokens in the context window - so that's usually around 8,000 tokens (approximately 6000 words).
You can experiment with smaller LLMs on your own devices to get a better feeling for how that works.