15 ms·
OpenAI Tokenizer
- simonw 4y agoThis tool is really useful for helping develop a better intuition for how GPT models actually work. Paste in some text and switch to the token IDs view. Note how common words (like "the ") have low integer token IDs, while things like emojis are split into several numbers. An LLM is a function that takes an array of integers and returns a new array of integers. Seeing the tokens like this helped me reinforce that mental model.
- psychphysic 4y agoTakes an array and returns a single integer is more correct and usefully so? What I still can't wrap my head around is that tokens often don't align with word structures. "Antidepressants" I'd imagine tokenizes as "anti" "depress" "ant". But nope. And "antipsychotic" tokenizes differently from it too! I assumed the output is a token i.e. a single integer and that's rarely even a full word?
- shagie 4y agoImagine a circular list (in however you want to construct that) that matches the input size for the model. The prompt is initially loaded at the start of the list and the model is run and produces high activation on a single output. That token output is then fed to the end of the input circular list and also added to the "this is what the model returned." This process of running the model, getting the token output and sending one copy to the input list and one copy to the return string is repeated until the number of tokens generated hits a numeric limit or a token that represents the stop token is encountered.
- bastawhiz 4y ago> "Antidepressants" I'd imagine tokenizes as "anti" "depress" "ant". But nope. And "antipsychotic" tokenizes differently from it too.. Tokens are symbols. You're thinking of them like embedding vectors. Tokens represent the step before a meaning is assigned to the text: it turns some unit of text into what's essentially an identifier. Which is to say, two homonyms would have the same token id, even though they have different meanings. Tokens have no notion of context.
- crakenzak 4y agowhat is the benefit to such splitting of text based on seemingly meaningless lines? isn't there a better way to do it?
- rmellow 4y agoBreaking text into sub-word units... 1. Greatly reduces memory usage. Instead of memorizing every inflection of the word "walk", it memorizes the root (walk) and the modifiers (ing, ed, er, ...). These modifiers can be reused for other words. 2. Allows for word compositions that weren't in the training set. This is great for uncommon or new expressions like "googlification" or "unalive".
- shagie 4y agoThe walk example doesn't quite hold up. If you put: test walk walker walking walked into the tokenizer you will see the following tokens: [test][ walk][ walk][er][ walking][ walked] Only walker is broken up into two different tokens. I added "test" to that because walk at the start doesn't include the leading space and [walk] and [ walk] are different tokens. For even more fun, [walker] is a distinct token if it doesn't include the leading space. test walker floorwalker foowalker becomes: [test][ walk][er][ floor][walker][ fo][ow][alker] How we think of words doesn't cleanly map to tokens. (Late edit) walker floorwalker becomes tokenized as: [walker][ floor][walker] So in that case, they're the same token. It's curious how white space influences the word to token making.
- nonfamous 4y agoThere’s no syntax or structure to the token set. The actual tokens were algorithmically selected based on the training data to (putting things loosely) optimize compression of the training data given a token set size.
- PeterisP 4y agoSure, but what I'm hearing in the parent post is a question about why we don't use linguistically motivated subword units (of similar length/vocabulary size and thus memory usage) e.g. cutting across morpheme boundaries instead of whatever an algorithm like BPE caclulates.
- raldi 4y agoHow does it decide whether to split "antid" into "ant" "id" or "anti" "d"?
- eternalban 4y agoTry this: 'this is a day that this sentence with clarify that day. Is this not a good day?' [5661, 318, 257, 1110, 326, 428, 6827, 351, 18282, 326, 1110, 13, 1148, 428, 407, 257, 922, 1110, 30] Note 'day' is solidly 1110 here. Now start a sentence with day. "day began with laughter" [12393, 2540, 351, 20263, 13] So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well. "Day after day after this day" [12393, 706, 1110, 706, 428, 1110] "Day day Home home home" [12393, 1110, 5995, 1363, 1363] "day day day home home home" [820, 1110, 1110, 1363, 1363, 1363] [corrected/edited: so case-sensitive and position sensitive as well.] btw doesn't the output array contain the prompt as well (because of the transformer architecture? not entirey sure ~iirc)
- fenomas 4y ago> So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well. You're missing that it groups in spaces. The position isn't relevant, but "day" is a different token than " day".
- eternalban 4y agoAh, you're right. [try "datedate". interesting how it partitions that as 'dated' + 'ate'. Compare with "matemate" -> 'mat', 'emate'.] p.s. "ifthiswasagermanword if this was a german word." It's not even spaces. That second sequence ' german' is chopped up as 'ag' 'erman'.
- hn_throwaway_99 4y agoIt's a lot simpler than that. You can see in the tokenizer that the boundary for words includes the preceding space. So, since the first word doesn't have a preceding space, it has a different token.
- sebzim4500 4y agoThere's no position parameter, it's just that " day" and "day" are different tokens.
- smaddox 4y ago> An LLM is a function that takes an array of integers and returns a new array of integers. To refine this a bit more, a LLM is a function that takes an array of integers (or really, a batch of arrays of integers), and returns a probability distribution for each possible integer, with the array shifted left by one place to enable prediction.
- sorokod 4y agoTo refine further: takes an array of integers and draws the rest of the f**king owl.
- turzmo 4y agoI would argue that my hard drive too can take an array of integers and produce an owl.
- wilg 4y agoOh let's see! I'll start: 1
- x-complexity 4y ago1
- VHRanger 4y agoMoreover: The integers are really indeces into the embedding space. So you'd want to think more that the model maintains a giant matrix (one row = one token; one column is an embedding feature). The array of indices gets the relevant embeddings to shove through the rest of the model's forward pass.
- flir 4y agoA probability distribution for each token in the array, not just the last one? I don't understand that, because wouldn't the probabilities later in the sentence be impacted by the tokens chosen earlier in the sentence?
- yewenjie 4y agoNOTE: this is only valid for the old models (GPT-3 and Codex). IIRC, there is no simple way to know the token usage for the new models (gpt3.5-turbo and beyond).
- NameError 4y agoThe API does tell you how many prompt and completion tokens each request used, if you're okay knowing after-the-fact
- sebzim4500 4y agoAssuming the API docs are honest, all the publicly available GPTs use the same tokens.
- tedsanders 4y agoDifferent GPTs use different tokens: https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...
- iamjackg 4y agoYou can use https://github.com/openai/tiktoken https://github.com/openai/tiktoken
- tedsanders 4y agoThis guide explains how to count tokens for 3.5-turbo and beyond: https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...
- coffeeri 4y agoOpenAI seems to use Tiktoken [0]. It also covers GPT-4 token encoding. [0] https://github.com/openai/tiktoken https://github.com/openai/tiktoken
- deleted 4y ago[deleted]
- lspears 4y agoSeems odd they don't reference this on the page. Instead they list: "If you need a programmatic interface for tokenizing text, check out the transformers package for python or the gpt-3-encoder package for node.js." with the links: https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast https://huggingface.co/docs/transformers/model_doc/gpt2#tran... https://www.npmjs.com/package/gpt-3-encoder https://www.npmjs.com/package/gpt-3-encoder
- dudeinjapan 4y agoInteresting that Japanese seems to get every character/letter tokenized individually.
- fenomas 4y agoHuh, unless the demo is broken it seems to tokenize unicode per-byte, rather than per-character. E.g.: "あ" => [40948] "亜" => [12859, 250] "ア" => [171, 121, 109]
- dudeinjapan 4y agoYeah I noticed similar. That can't be right...
- rhdunn 4y agoIt's probably operating on UTF-8 data on a byte-per-byte level without any additional processing. Just feeding it the raw string data and letting it assign the tokens. It's similar to how it is splitting words at arbitrary points, rather than at clear morphological or lexical locations (e.g. on the Jane Austen text `"Now, ma'am," said Jane to her aunt, "shall we join Mrs. Elton?"` I've seen it tokenize that as `"|Now|,| ma|'|am|,"| said| Jane| to| her| aunt|,| "|shall| we| join| Mrs|.| El|ton|?"`).
- dudeinjapan 4y agoI would find that hard to believe, as the bytes have zero semantic meaning, and moreover, pairing the wrong bytes in the output will result in complete gibberish. It would be akin to tokenizing each English letter "N|o|w|,| |m|a|'|a|m|..." except far worse. Moreover it's trivially easy to tokenize the glyphs.
- pyentropy 4y agoA character is the base unit of written communication. Single characters as tokens is not a bad idea, it just requires too much resources to make it learn and infer. BPE is a tradeoff between single letters (computationally hard) and a word dictionary (can't handle novel words, languages or complex structures like code syntax). Note that tokens must be hardcoded because the neural network has an output layer consisting of neurons one-to-one mapped to the tokens (and the predicted word is the most activated neuron). Human brains roughly do the same thing - that's why we have syllables as a tradeoff between letters and words.
- darknoon 4y agoWe noticed that this webpage is out of date for recent models so we (diagram.com) commissioned a better one that lets you pick any of OpenAI's models including chats: https://tiktokenizer.vercel.app/ https://tiktokenizer.vercel.app/
- JonathanFly 4y agoWow, I can't thank you enough for this. Somehow I never noticed that GPT-4 doesn't use separate tokens for each tab like 3.5. I was wasting a lot of time minimizing excessively tabbed code to save on the token count! Like seriously way too much time, all based on a bad assumption. https://twitter.com/jonathanfly/status/1643633463260577794 https://twitter.com/jonathanfly/status/1643633463260577794
- deleted 4y ago[deleted]
- jscheel 4y agoTiktoken is pretty nice. I've been exposing it as an internal service in our infrastructure so that we can get token counts easily. The bigger problem is figuring out how to chunk longer contexts so that you stay within the context window limit defined by the model you are using.
- stared 4y agoHow are these encodings created? My guess it is related to text compression, but would be happy to see an algorithm that is responsible for generating them.
- lairv 4y agohttps://en.wikipedia.org/wiki/Byte_pair_encoding https://en.wikipedia.org/wiki/Byte_pair_encoding tldr: start with unary characters and greedily merge pairs that are the most frequents A consequence is that an encoding is suited for the dataset it was trained on, so if a language is under-represented in the data it will result in higher number of tokens to encode it
- thomasahle 4y agoReading the sentencepiece paper they say: > The main difference to other compression algorithms, such as Huffman encoding, which have been proposed to produce a variable-length encoding of words for NMT (Chitnis and DeNero, 2015), is that our symbol sequences are still interpretable as subword units, and that the network can generalize to translate and produce new words (unseen at training time) on the basis of these subword units. I don't see why Huffman encoding doesn't give you that same interpretability? Actually the algorthm for producing a Hoffman tree is very similar to that for BPE: > The process begins with the leaf nodes containing the probabilities of the symbol they represent. Then, the process takes the two nodes with smallest probability, and creates a new internal node having these two nodes as children. The weight of the new node is set to the sum of the weight of the children. We then apply the process again, on the new internal node and on the remaining nodes (i.e., we exclude the two leaf nodes), we repeat this process until only one node remains, which is the root (from https://en.m.wikipedia.org/wiki/Huffman_coding https://en.m.wikipedia.org/wiki/Huffman_coding) I guess the issue is that Huffman requires the alphabet to be predefined, where BPE "discovers it" as it goes along.
- astrange 4y ago> I don't see why Huffman encoding doesn't give you that same interpretability? It might just be that a Huffman encoding is a bit-string and not a byte-string. BPE encoding causes interesting failures, like how it can't do anagrams or spell words backwards properly. And yet it can make rhyming poems now.
- RugnirViking 4y agoReally interesting. How do these work? are these a separate ai/neural net/model to the transformer? they don't seem to follow any humanlike structure or process?
- aqme28 4y agoWhat's the benefit of OpenAI charging per-token instead of per-character or per-word? Since token algorithms change model-to-model and version-to-version, it seems like they've added a lot of complication for no actual benefit to the user except for a little peek under the hood. Is there a benefit to this scheme that I'm not seeing? Is there some way to game the system otherwise?
- qeternity 4y agoBecause tokens are the unit of work in an LLM and it’s not correct to say that tokens or even embeddings change between models.
- hn_throwaway_99 4y agoIt's not a "benefit", it's simply how the technology works - the underlying model just fundamentally works on tokens as it's atomic inputs. The models don't know anything about words, just tokens.
- typest 4y agoIt's not that they're just charging per token -- the actual models are operating on a token level. The model sees things in terms of tokens, and in openai's case, these tokens are subword (pieces of words), not words themselves, not characters. So the real question is, what is the benefit of modeling your tokens as subwords, rather than as characters or words? I think there is a lot of nuance here, and I don't understand it all. But, some benefits: * Words, at least in English, are composed of different pieces, like roots, prefixes, and stems. Modeling at the subword level more naturally aligns your model with this aspect of language. If I tokenize "warmest", I get "warm" and "est". So, the meaning of the token "est" can be learned by the model -- whereas if you modeled by words, the model would have to individually relearn this aspect of information for every word ending in "est". * Modeling at the subword level makes your sequences a lot shorter than modeling at the character level, which should help with things like efficiency. * Modeling at the subword level makes your vocabulary a lot bigger than just modeling at the character level, which I suspect helps the model, as it can assign the subwords themselves meaning. E.g., it can learn the meaning of the token "warm" on its own, rather than having to learn this meaning only through learning the relationship of the tokens "w" "a" "r" and "m". Hope this helps! Would love for anyone else to chime in/add on/correct me.
- swyx 4y ago" SolidGoldMagikarp" Characters: 18 Tokens: 1 heh. all i know is this is a fun magic token but 1) i dont really know how they found this and 2) i dont know what its implications are. i heard that you can use it to detect if you are talking to an AI.
- kevingadd 4y agoSome of the magic tokens are related to Twitch Plays Pokemon. https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation?commentId=zKayW2ZYYamwDLfkN https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
- swyx 4y agohmm. so this is evidence that openai scraped twitch chat of all places? (notoriously ephemeral) also opens a question as to how tokenizers are trained. should you discard or break up super niche words like this?
- squeaky-clean 4y agoIt doesn't necessarily mean it scraped twitch chat. That is the name of a moderator. They also moderate the subreddit and probably some other places. And being a moderator for such a popular event they probably had their name mentioned in other places as well. Every time they comment on Reddit their username would also appear. https://www.reddit.com/r/twitchplayspokemon/comments/2cxkpp/new_moderator_solidgoldmagikarp/ https://www.reddit.com/r/twitchplayspokemon/comments/2cxkpp/...
- RugnirViking 4y agoit comes from this blog post: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...
- nickvincent 4y agoI think it's related to Reddit users who posted (very frequently!) on a counting focused subreddit (people literally post "1", "2" , "3" in sequence so usernames appear 50k+ times). Some screenshots and links in this Twitter thread: https://twitter.com/SoC_trilogy/status/1623118034960322560 https://twitter.com/SoC_trilogy/status/1623118034960322560 Plus additional commentary here: https://twitter.com/nickmvincent/status/1623409493584519168 https://twitter.com/nickmvincent/status/1623409493584519168 (in short: I think this situation is comparable to a "Trap Street" https://en.wikipedia.org/wiki/Trap_street https://en.wikipedia.org/wiki/Trap_street that reveals when a map seller copies another cartographer) I hadn't seen the Twitch plays pokemon hypothesis though (from another comment here), I wonder if it could be both!
- tosh 4y agoIt is very interesting to compare how various languages (including programming languages) are tokenized.
- m3kw9 4y agoIt’s made so people don’t game the system. Say “hello world” would be same as “hello_world” for llm. If they didn’t count tokens by letter, I would be using it for free
- devit 4y agoInterestingly they seem to have different token ids for "Word", "word", " Word" and " word". That seems kind of a wasteful design. It seems like it would make more sense to have a single token for all variants and then a "capitalized where not expected" token (e.g. "foo Foo"), a "not capitalized where expected" token (e.g. "foo. foo") and a "missing space where expected" token (e.g. "foo.Foo"). The lack of any normalization also means that WrItInG tExT lIkE tHiS will make future GPT versions not be able to make full use of the text during future training unless they change the tokenization (or the model is so overpowered that it doesn't matter).
- neerd 4y agoThey charge by the token so I’m not so sure about that
- gradys 4y agoThe model is indeed so overpowered that it doesn’t matter in practice. See the Sentencepiece paper for some discussion of the design decisions on stuff like whitespace.
- williamstein 4y agoI am glad it tokenizes Python and all other programming languages in a systematic way.
- RC_ITR 4y agoIn practice, GPT uses byte-pair encoding [0] for each Unicode character. That’s why cases are treated differently - they’re different in Unicode. This is also the only way to teach a model how to properly capitalize things (since there are no human defined rules). [0] https://towardsdatascience.com/byte-pair-encoding-subword-based-tokenization-algorithm-77828a70bee0#:~:text=%F0%9F%8F%83-,Byte%2DPair%20Encoding%20(BPE),not%20occur%20in%20that%20data https://towardsdatascience.com/byte-pair-encoding-subword-ba....
- totony 4y agoThe actual tokenizer often does not matter since you can add pre processors/normalizers. I assume they did it like this because capitalization matters in a lot of contexts
- gnrlst 4y agoCompletely useless, but I was curious about the token indexes. I tried to look for Token #0. After a couple minutes of trial and error, it turns out it's the exclamation mark.
- swyx 4y agohow does one "look" for a token? there isn't a lookup table somewhere?
- npace12 4y agoInteresting... I was gonna say "you can ask GPT" but it doesn't work anymore. On March 23rd, it responded with this: Human: convert these GPT tokens to text: [134, 1322] AI: The tokens [134, 1322] correspond to the words "can" and "not" in the GPT language model. So the text corresponding to these tokens would be "can not" Today, it's giving me the "As a language model" response
- SirMaster 4y agorawdownloadcloneembedreportprint Tokens 1 Characters 32 Weird...
- stackedinserter 4y ago"not_a_word" token
- otaviogood 4y ago‘1984 is 1 token. 1884 is 2 tokens.’ I would be surprised if they use this tokenization still as it’s not math friendly.
- sharkjacobs 4y agoIt's a language model, not a mathematical model
- PeterisP 4y agoThey do use this tokenization, and that's the reason why these models sometimes struggle with tasks like "how many twos does this long number contain" and things like "is 50100 greater than 50200" as it tries to compare "501"/"00" with "50"/"200" while knowing that "501" is greater than "50". The models aren't optimized to be math friendly. They could be, but the major big generic ones weren't.
- v4dok 4y agoIt completely butchers Greek. No wonder it charges some much for so little output. Every Greek character is a token. I wonder if there is space for innovation there. I would imagine that it similarly difficult for other non-English languages as well-known. I fear for the effect this will have on them.
- 0xDEF 4y agoIt's crazy that OpenAI hasn't fixed their tokenizer yet. They are leaving the door wide open for some Chinese big tech company to capture the non-Latin script parts of the world. i18n (and accessibility) was something American tech companies were serious about in the 90s and early 2000s. That is how they captured most of the global market. US tech dropping the ball on this leaves the door wide open for Chinese competitors.
- hombre_fatal 4y agoDoes OpenAI’s tokenizer issues cash out into having worse results for Greek rather than just being more expensive for gpt-4? (gpt-3.5-turbo already costs peanuts) If not, then this response seems overblown. The competitive advantage in LLM at this point probably is not tokenizer optimizations and more about having results worth a damn.
- v4dok 4y agoThe usability is worse. Token limits are so much easier to reach. It's like using a model with dementia.
- hombre_fatal 4y agoI didn't think about that but now it's obvious. Good point.
- Fauntleroy 4y agoIt could be that they are actively working on this problem, but the product has not yet been released.
- brianbest101 4y ago[dead]
- MuffinFlavored 4y agoI took a random Java file I had laying around that I was working on lately. ~100 lines of code + whitespace 1300-1900 tokens So if I fed this to OpenAI and said "how can I make this file better/improve upon it", it would have cost: between $0.03 and $0.12 for this one file using GPT-4 not sure I could use gpt-3.5-turbo since it says it is for chat and not code? Does that sound right? $0.05 for every file of source code scanned sounds too high for realistic usage. Even $0.01 sounds high? Modern company might have 1,000,000+ files of code, no?
- SubiculumCode 4y agoexcuse my ignorance, but I thought it was $20 per month.
- bikingbismuth 4y agoThat is for the interactive chat experience. API calls are sold a la carte. Details here https://openai.com/pricing https://openai.com/pricing.
- SubiculumCode 4y agothanks!
- MuffinFlavored 4y agoIt would be cool if they told their $20/mo users "here's how much your past 30 day usage would have cost if we billed you via the API (aka how many tokens/sessions/chats/whatever did you use)
- engrefoobiz 4y agoGPT-4 costs 30 times more than gpt-3.5-turbo and 60ktimes more if you use the 32k token gpt-4 model. It's by far their most expensive service! I'm using gpt-3.5-turbo, also for coding, and honestly it does just fine.
- maherbeg 4y ago
- ubj 4y ago> A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly 3/4 of a word (so 100 tokens ~= 75 words). Just for fun I tried entering in "pneumonoultramicroscopicsilicovolcanoconiosis" and "antidisestablishmentarianism". The first was pretty evenly split into tokens of length 1-5 characters, but the second put all of "establishment" into a single token. No useful conclusions drawn, but it was an interesting test.
- luke_cq 4y agoI desperately want to be able to get a concrete amount of tokens for my prompt before making a call - things like this make it very hard to request the right amount of max_tokens from longer prompt/generation pairs.
- deleted 4y ago[deleted]
- sharkjacobs 4y agohttps://github.com/openai/tiktoken https://github.com/openai/tiktoken
- alli_star 4y ago[dead]
- GistNoesis 4y agoAccidentally quadratic ! Byte pair encoding by construction is quadratic on the length of the words. And usually the input is pre-split into words before being given to the byte pair encoder. Hopefully they use something different implementation in prod. It needs to be sanitized against very long words (like 10k character long words :) ). In previous tokenizer like CLIP (https://github.com/openai/CLIP/blob/main/clip/simple_tokenizer.py https://github.com/openai/CLIP/blob/main/clip/simple_tokeniz... ) , they used additional preprocessing steps like html escaping and various cleanup preprocessing using some python library (ftfy, html and regex), which made porting the code exactly to other languages a real pain. Sadly this library doesn't solve that :'-(
- ladberg 4y agoFor GPT they didn't pre-split into words so they definitely have something faster than quadratic! Not sure what it is though, I'm very curious.
- bobm_kite9 4y agoYes, you're right. There could be multiple ways to tokenise a sentence. Shouldn't all the valid tokens be included in the vector?
- GistNoesis 4y agoIf this splitting interest you having a look at https://github.com/openai/tiktoken/blob/main/src/lib.rs https://github.com/openai/tiktoken/blob/main/src/lib.rs is great at showing all the ugly edge cases that causes instabilities. In theory Byte Pair Encoding is unique, but practice makes it harder. It's also complicated due to regex and utf-8. Most of the time the differences should be too important because the neural network should be able to handle typos. In BPE you may have plenty of escaping problems, problematic character like ' and \ are nasty to get right : worst case if you don't handle your errors being that if you have trained your byte pair encoding dictionary on escaped sentences, then a single \ should never occur as it is encoded as \\, so if you split the string between the \ then the byte pair encoding might fail to find the key in the dictionary. Making the thing deterministic and stable when you change your regex version (and when you train one network you'd like to not have to retrain it when there is a bugfix in a regex library). Porting to other platforms also becomes very hard if you want replicable results.
- jonathankoren 4y agoIt looks like this thing doesn't tokenize into anything with any semantic meaning, but rather just a sequence of bytes that match some sort of information theoretic criteria. It doesn't appear to have any linguistic (written, nor verbal) pattern. I guess it's fine for their specific use case, but whatever. Tokenization is such a basic and domain specific operation, it feels like someone had to demo something. Bonus (code) points for just saying "fuck it" on emojis. They didn't even split it into code points.
- bilater 4y agoDammit they copied me lol https://www.gptcalculator.xyz/ https://www.gptcalculator.xyz/
- bilater 4y agomine has an API though which hopefully is useful
- thatwasunusual 4y agoAlso, their solution works. Yours just says "Loading..." whenever I try it.
- bilater 4y agoIs working for me
- nextworddev 4y agoOne interesting fact I stumbled upon recently is that GPT2Tokenizer library and Tiktoken library produces the same number of tokens for `text-davinci-003` model, despite GPT2Tokenizer being GPT2 and text-davinci-003 being GPT3.5. For code, however, Tiktoken library and GPT2Tokenizer produce different tokenizations.
- shevis 4y agoKey difference here is in tokenization encoders. The newer models make use of the `cl100k_base` encoding.
- non- 4y agoI found this tool recently when it was linked from this Computerphile video[1] about "glitch tokens". tldw: Certain junk data was thrown out post-tokenization e.g. the /r/counting[2] community data and debug logs from Rocket League some tokens specific to those contexts stuck around, however, and are now like "a color you've never seen before" as far as GPT-X models are concerned giving the model one of these "glitch" tokens causes it to kind of freak out and return gibberish or some completely random response because it has not encountered them during training, because they were removed when the data was cleaned. [1] https://www.youtube.com/watch?v=WO2X3oZEJOA https://www.youtube.com/watch?v=WO2X3oZEJOA [2] https://reddit.com/r/counting https://reddit.com/r/counting
- non- 4y agoAnother interesting tweet[1] I saw today shows how you can ask ChatGPT to compress text and it invents it's own (effective!) shorthand. I bet it's related somehow to glitch tokens and the way GPT is grouping tokens internally. [1] https://mobile.twitter.com/VictorTaelin/status/1642664054912155648 https://mobile.twitter.com/VictorTaelin/status/1642664054912...
- artdigital 4y agoI experimented with something similar previously and it doesn’t really work. It usually can’t decompress it properly
- kleton 4y agoThe example given is obviously gpt's own generation. It might not be expected to roundtrip arbitrary text.
- Genbox 4y agoHad not seen the Computerphile video yet. Laughing my socks off right now. Thanks for that.
- dschnurr 4y agoHi folks – I work at OpenAI and helped build this page, awesome to see it on here! Heads up that it's a bit out of date as GPT4 has a different tokenizer than GPT3. I'd recommend checking out tiktoken (https://github.com/openai/tiktoken https://github.com/openai/tiktoken) or this other excellent app that a community member made (https://tiktokenizer.vercel.app https://tiktokenizer.vercel.app)
- teruakohatu 4y agoThat is very helpful, thank you. I had not realised the latest models were now tokenizing number as 3 digit groups. Can you give any insight into why 3 digits?
- resters 4y agoWas the purpose of the page and post to generate comments that can be used as training data?
- lemming 4y agoAre there plans to release tokenisers for other platforms? I'm accessing the OpenAI API from Clojure, and it would be really nice to have a JVM version so I can estimate token use before sending.
- lowefk 4y agoI wasn't aware that GPT-3 and GPT-4 use different tokenizers. I've read https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb https://github.com/openai/openai-cookbook/blob/main/examples... and misinterpreted "ChatGPT models like gpt-3.5-turbo and gpt-4 use tokens in the same way as older completions models, ..." as GPT-3 and GPT-4 using the same tokenizer except for im_ tokens. Now I can see so many improvements, including the encoding of whitespaces and digits.
- egorfine 4y agoHey it seems that UTF-8 support is broken on the page. Test phrase could be something like "Жизнь прекрасна и удивительна" ("Life is great" in russian). I make an assumption that this is the implementation on the page that is broken, not the actual tokenizer. The reason: russian works perfectly in GPT-3 which I guess wouldn't be the case with a tokenization as presented on the page.
- H8crilA 4y agoThis works really poorly in non-latin scripts. Try pasting "Україна" (Ukraine) or "北京是中国的首都" (Beijing is the capital of China). I'm a little surprised that nobody optimized that, there must be enough training data to warrant this effort.
- frozenlettuce 4y agoI found it interesting how it tokenizes non-English words: Steve Jobs was fired from Apple -> [19206, 19161, 373, 6294, 422, 4196] (one token per whole word) Olha que coisa mais linda e cheia de graça -> [30098, 3099, 8358, 763, 9160, 285, 15152, 300, 22261, 304, 1125, 544, 390, 7933, 50041] (tokens with up to 3 characters) bonus: Apple => 4196 apple=> 17180
- stan_kirdey 4y agoWould be great if ChatGPT interface had used token count instead of throwing an error after submission.
- adamkochanowicz 4y ago"Hi" 1 token " " 10 tokens Interesting way to see how much data is actually in an emoji, especially ones that are combinations of separate emojis
- adamkochanowicz 4y agoThat's supposed to be a person with a skin color selection.
- csomar 4y agoI've put this into Codex fn hello(message: String) -> Result<String> { this is not part of code } Codex does a pretty code job at tokenizing the different parts of the first line (fn, open parenthesis, -> is considered too, its own token) but fails miserably on the second line. The second line should be a single token of invalid code. It should be tokenized into text, if that line was preceded by "//" or a start comment indicator. Interestingly, GPT-3/4 can probably explain the concept of commenting and specifically for Rust too. However, it can't apply it in this particular context.