5 ms·
> A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 1
by capableweb 3y ago
> A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words) - https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
32,000 (tokens) * 4 = 128,000 (characters)
> While a general guideline is one page is 500 words (single spaced) or 250 words (double spaced), this is a ballpark figure - https://wordcounter.net/words-per-page https://wordcounter.net/words-per-page
Assuming (on average) one word = 5 letters, context ends up being ~50 pages (128000 / (500 * 5)).
Just to put the number of "32k tokens" into somewhat estimated context.
- dmix 3y agoI wonder how many LoC that is on average. Is there an average for LoC? It’s probably based on the language…
- ethbr0 3y agoFinally, APL's day is here! https://en.m.wikipedia.org/wiki/APL_(programming_language) https://en.m.wikipedia.org/wiki/APL_(programming_language)
- wongarsu 3y agoTesting some random rust code, it's about 15-20 tokens per LoC, so about 1500-2000 LoC in a 32k context. Interestingly, using 2-space indentation, as soon as you are about 3-4 indentation levels deep you spend as many tokens on indentation as on the actual code. For example, "log::LevelFilter::Info" is 6 tokens, same as 6 consecutive spaces. There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all.
- ec109685 3y agoMake sure you are testing with the tiktokenizer: https://news.ycombinator.com/item?id=35460648 https://news.ycombinator.com/item?id=35460648
- wongarsu 3y agoAh, good catch, it's actually closer to 8 tokens per LoC with GPT4's toktoken, so about twice as good. Some quick testing suggests that's mostly down to better whitespace handling.
- JimDabell 3y ago> There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all. Using tabs for indentation needs fewer tokens than using multiple spaces for indentation.
- mafuy 3y agoCurious how this point in the tabs vs. spaces debate re-emerges
- omega3 3y agoHow do this even work if the code also uses external libraries?
- MacsHeadroom 3y agoA LoC is ~7 tokens thanks to the new toktoken tokenization in GPT-4. 32k is ~4.5k LoC
- user_named 3y ago0.75 * 32,000 = 24,000 words is faster and more direct
- capableweb 3y agoThanks, math was never my strong suit and I was writing the comment as I was calculating towards the results, never refined and raw, as it should :)
- s-macke 3y agoThis is a better calculator, because it also supports GPT-4: https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/ Not sure, why they don't support GPT-4 on their own website.
- modernpink 3y agoAt $0.60 for 20k prompt tokens, it's not going to be cheap so they will need to bring the price down to get broader adoption. As far as I can tell, the initial reading in of the document (let's say a 20k token one), will be a repeated cost for each subsequent query over the document. If I have a 20k token document, and ask 10 follow-up prompts consisting of 100 tokens, that would take me to a total spend of 20k * 10 + (10 * 11)/2 * 100 = 205,500 prompt tokens, or over $6. This does not include completion tokens or the response history which would edge us closer to $8 for the chat session.
- smodo 3y agoWhat I’ve read is that people let a kind of meta-chat run alongside the client interaction. The meta channel decides what parts of the history to retain so the primary channel doesn’t use as many resources. You could let GPT decide when it needs to see the whole context again etc. There’s a lot of interesting ways to let GPT handle it’s own scope, I think.
- bravura 3y agoIs this a native feature? Or something home brewed? If the second, how to use it?
- happycube 3y agoI forget if Langchain can do that, but something along those lines will exist if it doesn't already, it's too obvious not to, and too important for the free ChatGPT equivalents which will be popping up over a matter of days/weeks now that truly free versions of llama are coming out. TL;DR the puck should get there very soon, not just Really Soon Now.
- sdo72 3y ago32k tokens, 3/4 of 32k is 24k words, each page average is 500 or 0.5k words, so that's basically 24k / .5k = 24 x 2 =~48 pages.
- capableweb 3y agoThat's great, imagine if more researchers were as excited to reproduce results as you are? The world would be much better.
- elashri 3y agoprobably off topic, but part of this is "Good luck writing a grant proposal that says I want to reproduce the work of this group and get it accepted". Unless of course you are claiming ground breaking paradigm shift like evidence of unified theory or super-conductivity at room temperature.
- cmonnow 3y ago[dead]
- tysam_and 3y agoForgive my ignorance, but what is the relevance of this to the above comment?
- capableweb 3y agoWe both arrived at mostly the same result from the same question of "how many pages of text would 32k tokens be?", they basically did the calculation again albeit in a slightly different way. Just like when researchers try to reproduce the results of other's studies.
- sdwr 3y agoThis comes across as patronizing to me. "Who's a good little verifier. You are!"
- 3y ago
- skybrian 3y agoIf it’s not any faster, I’m thinking that how long you’re willing to wait for an answer will be the practical bottleneck? 38 seconds for that example.
- BaculumMeumEst 3y agoif you wanted to send 32k tokens of code, are you able to do that using a model with a 4k context limit by spreading those tokens out across multiple messages? or does it not work that way?
- dragonwriter 3y agoThe context limit is for request + response, and there is no storage in between requests (ongoing chat interactions are done by adding prior interactions to the prompt, so the whole chat – before things start falling out of history – is limited to the context window.)
- simonbw 3y agoNot really. The API is stateless, you pass it in a whole conversation and it responds with the next message. The entire conversation including its response is limited to 32k tokens.
- BaculumMeumEst 3y agoi’m just confused because i thought i remembered sending long chunks of code using the api, the request will fail, but then i would split it up and then it would work okay. i guess i’m running into a different limit (not context length), or maybe i’m misremembering
- glandium 3y agoUnfortunately, in CJK languages, it averages to slightly more than one token per character.
- TeMPOraL 3y agoI suppose we have a quite objective way of measuring and comparing semantic density of natural languages.