4 ms·

> Longer context. The context length of the new model is increased by a factor of four, from 2048 to 8192, making it more convenient to work with long documents

by gok 4y ago

> Longer context. The context length of the new model is increased by a factor of four, from 2048 to 8192, making it more convenient to work with long documents. 8192 words is getting into the range of short stories or a masters thesis, which opens the door to some interesting applications.

drusepth 4y ago
Important to note that these tokens _can_ be words, but oftentimes a word will be comprised of multiple tokens, so 8192 tokens = 8192 words isn't strictly correct. That said, your point stands. Most short stories are low-to-mid four-digit words, and a jump from 2048 tokens to 8192 squarely fits in that window. As someone who's been working on multi-layered approaches to using GPT-like models for long text generation (e.g. synopsis -> outline -> paragraph expansions) to get around the limited context window, it'll be interesting to see if people will keep working towards that end or if it'll all become a moot point as the effective context window continues to scale up.
- janalsncm 4y ago
  Do we know exactly what the GPT3 tokenizer is? It could be combining multiple words together, for example common bigrams. Never mind, it looks like they have a tokenizer tool online and every bigram I’ve given it BPEs to multiple tokens: https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer
minimaxir 4y ago
It's worth noting that OpenAI mixed it up and is not using the GPT-2 tokenizer this time. It's unclear what tokenizer they are using and the documentation is being coy about it. It could be a more efficient or a less efficient tokenizer.
- bryanh 4y ago
  https://twitter.com/sherwinwu/status/1603522777968832512 https://twitter.com/sherwinwu/status/1603522777968832512 appears to include it.
  - minimaxir 4y ago
    So it does. The code there implies cl100k_base has a vocab size of 100k (I guess it's in the name lol) which means it is more comprehensive than GPT-2's 50k, so fewer tokens will be necessary.

by gok 4y ago

drusepth 4y ago

Important to note that these tokens _can_ be words, but oftentimes a word will be comprised of multiple tokens, so 8192 tokens = 8192 words isn't strictly correct. That said, your point stands. Most short stories are low-to-mid four-digit words, and a jump from 2048 tokens to 8192 squarely fits in that window. As someone who's been working on multi-layered approaches to using GPT-like models for long text generation (e.g. synopsis -> outline -> paragraph expansions) to get around the limited context window, it'll be interesting to see if people will keep working towards that end or if it'll all become a moot point as the effective context window continues to scale up.

janalsncm 4y ago
Do we know exactly what the GPT3 tokenizer is? It could be combining multiple words together, for example common bigrams. Never mind, it looks like they have a tokenizer tool online and every bigram I’ve given it BPEs to multiple tokens: https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer

minimaxir 4y ago

It's worth noting that OpenAI mixed it up and is not using the GPT-2 tokenizer this time. It's unclear what tokenizer they are using and the documentation is being coy about it. It could be a more efficient or a less efficient tokenizer.

bryanh 4y ago
https://twitter.com/sherwinwu/status/1603522777968832512 https://twitter.com/sherwinwu/status/1603522777968832512 appears to include it.
- minimaxir 4y ago
  So it does. The code there implies cl100k_base has a vocab size of 100k (I guess it's in the name lol) which means it is more comprehensive than GPT-2's 50k, so fewer tokens will be necessary.