3 ms·
> Another point is that many people don't realise that LLMs re-read their entire input for every word they generate! Not every word, for every generation (whic
by circuit10 4y ago
> Another point is that many people don't realise that LLMs re-read their entire input for every word they generate!
Not every word, for every generation (which on something like ChatGPT is a message)
> they can read really fast even when running on mobile device hardware. Think 10K to 100K words per second
That’s definitely not right, I’ll try to find some numbers
- jiggawatts 4y agoPlease do, I was about to include that statement in a presentation, so if I’m wrong I’d love to be corrected!
- circuit10 4y agoI’m really struggling to find any actual numbers so I could be very wrong but I vaguely remember input tokens being around 10x faster Basing it on pricing doesn’t seem like a good way to estimate it because it varies so much but: - OpenAI charges the same for input and output tokens, except for GPT4 where output tokens cost twice as much as inputs - textsynth.org charges 25 times as much for outputs than for inputs - goose.ai doesn’t charge for inputs Also you have limited context length so that limits how much data you can read
- mike_hearn 4y agoI don't think that's quite correct. The underlying algorithms start by processing the prompt tokens in parallel, and once that's processed, the intermediate KV vectors are re-used for each new token prediction by the feed-forward. What you may be thinking of is that once the model stops predicting at the end of an API call the state is dropped from RAM, and if you go back with another message in a chat, then the LLM will re-read the chat log up to that point as part of the prompt. So this is true on a per-message basis, but not on a per-word or per-token basis.