We use the GPT2 tokenizer (which is apparently the same as the GPT3 tokenizer) to count the number of tokens in the extracted input text and brutally truncate t

We use the GPT2 tokenizer (which is apparently the same as the GPT3 tokenizer) to count the number of tokens in the extracted input text and brutally truncate the input text to keep it under the limit. There is a lot of room for more finesse in this area if anyone wants to help out. See open issues in github.