2 ms·
GPT-4 turbo has a context window of 128k tokens, so you can divide every book into a few chunks for initial summarization, then merge the results for the final
by bookpecker 3y ago
GPT-4 turbo has a context window of 128k tokens, so you can divide every book into a few chunks for initial summarization, then merge the results for the final summary!
- cranberryturkey 3y agoWow. Sounds like it’ll get expensive quick with all the api calls. I hope you make your money back via Amazon.
- cranberryturkey 3y agoI assume you have pdfs. How do you split them up?
- bookpecker 3y agopdftotext is a Linux utility to extract text from PDF files. Then you can easily cut the text file into pieces. For best summarization, it's best that the pieces overlap a bit, so that important paragraphs that lie on the boundaries are not lost.