3 ms·
Doesn't text-davinci-002 have a text input limit , what happens to stories that exceed that limit ?
by er1t0 4y ago
Doesn't text-davinci-002 have a text input limit , what happens to stories that exceed that limit ?
- wskish 4y agoWe use the GPT2 tokenizer (which is apparently the same as the GPT3 tokenizer) to count the number of tokens in the extracted input text and brutally truncate the input text to keep it under the limit. There is a lot of room for more finesse in this area if anyone wants to help out. See open issues in github.
- atgctg 4y agoYou can summarize them recursively: https://openai.com/blog/summarizing-books/ https://openai.com/blog/summarizing-books/