3 ms·
No. Autoregressive models don't have model specific limit to output tokens, it's just when to stop looping.
by sanxiyn 3y ago
No. Autoregressive models don't have model specific limit to output tokens, it's just when to stop looping.
- jxy 3y agoI guess I only know transformers and how BERT or GPT works, as there would be a limit in the context length. With GPT, you can certainly generate infinite amount of tokens but the previous tokens outside of the maximum context length would be outside of the context window. LLaMa has 2k, GPT-4 has 32k. Are you saying I can give unlimited tokens to PaLM and generate unlimited amount of tokens? So PaLM doesn't have a context limit?
- sanxiyn 3y agoNo, I am not saying that. Since PaLM 2 is a transformer model (they didn't disclose almost anything about the model architecture, but they did disclose that), it has a context length limit. What I am saying is that you can't infer that limit from the limit of maxOutputTokens parameter in the API.