3 ms·
I really doubt you can fit all Harry Potter books in 1M tokens.
by koakuma-chan 1y ago
I really doubt you can fit all Harry Potter books in 1M tokens.
- PeterStuer 1y agoThe series is 1,084,170 words. At let's say 1.4 tokens per word, this would not fit, but it is getting close.
- koakuma-chan 1y agoIt's 2M tokens for Gemini.
- chrismustcode 1y agoThat was previous iterations, 2.5 is 1 million context window https://ai.google.dev/gemini-api/docs/models https://ai.google.dev/gemini-api/docs/models (context window is details under model variant section with + signs) They were meant to crank 2.5 to 2 million at some point though, maybe waiting now till 3?
- koakuma-chan 1y agoI mean the Harry Potter books are 2M tokens.
- bredren 1y agoMaybe consuming the resources internally.
- magicalhippo 1y agoHow do they do if you test[1] them for attention deficit disorder? [1]: https://www.imdb.com/title/tt0766092/quotes/?item=qt1440870 https://www.imdb.com/title/tt0766092/quotes/?item=qt1440870
- gcr 1y agoThe entire HP series is about one million words.
- koakuma-chan 1y agoHarry Potter and the Order of Phoenix alone is 400K tokens.
- kridsdale3 1y agoAnd takes up a proportional width of everyone's bookshelves along side the others.
- llm_nerd 1y agoCurious, I found an epub, converted it to a txt, and dumped it into the Qwen3 tokenizer. It yielded 359,088 tokens, end to end. Using the GPT-4 tokenizer (cl100k_base) yields 349,371 tokens. Recent Google and Anthropic models do not have local tokenizers and ridiculously make you call their APIs to do it, so no idea about those. Just thought that was interesting.