Important to note that these tokens _can_ be words, but oftentimes a word will be comprised of multiple tokens, so 8192 tokens = 8192 words isn't strictly corre

Important to note that these tokens _can_ be words, but oftentimes a word will be comprised of multiple tokens, so 8192 tokens = 8192 words isn't strictly correct. That said, your point stands. Most short stories are low-to-mid four-digit words, and a jump from 2048 tokens to 8192 squarely fits in that window. As someone who's been working on multi-layered approaches to using GPT-like models for long text generation (e.g. synopsis -> outline -> paragraph expansions) to get around the limited context window, it'll be interesting to see if people will keep working towards that end or if it'll all become a moot point as the effective context window continues to scale up.