6 ms·
Tiktoken: OpenAI’s Tokenizer
- phneutral26 4y agoMaybe the name is a bit misleading at first sight... But the project is great!
- echelon 4y agoThis is so great. 1. Name 2. OpenAI is releasing useful stuff 3. Rust in AI!
- Vetch 4y agoNB: huggingface has long had a rust based tokenizer.
- ianai 4y agoI genuinely hope they consider a rename. I wouldn’t want to use software with such a sleazy connection. It could have been “OAITokenizer”.
- intelVISA 4y agoFitting for a company named OpenAI with closed models.
- Workaccount2 4y agoThey investigated themselves and found that their products are too powerful for the public, so in the best interest of everyone they had to make the incredibly difficult, but right thing to do, choice of.... ....going closed source and monetizing.
- voz_ 4y agoI do not like the name.
- bartread 4y agoThe comparison is with an alternative called "huggingface". At least "Tiktoken" gives some clue as to what the component is for.
- rob74 4y agoWhen I first encountered huggingface (without having seen its logo), I had to think of these guys: https://en.wikipedia.org/wiki/Alien_(creature_in_Alien_franchise)#Facehugger https://en.wikipedia.org/wiki/Alien_(creature_in_Alien_franc... - so, by trying to pick a domain that sounded welcoming (and was still available), they achieved the exact opposite (at least in my case)...
- charcircuit 4y agoThis project has nothing to do with TikTok though
- ianai 4y agoAnd the name implies there’s a connection. Which is problematic.
- jraph 4y agoOf course, it's because you are not tokenizing it right. It's tik-token. (Could we say it is kind of a garden-path word?)
- _Algernon_ 4y agoIt really doesn't. Does every name that has "book" or "face" in the name, imply a connection to Facebook?
- fragmede 4y agoMaybe not but Facebooker, Facebooked, Facebocker, and Facebooken all sound like they do.
- worldsavior 4y agoWhy regex is not in rust stdlib? It's one of the fundamental libraries out there.
- nneonneo 4y agoRust's philosophy is "batteries not included". It's weird coming from Python or Go, where the standard libraries are much larger, but it makes sense for Rust: keep the standard library small and agile (especially as it is statically linked into every binary!) so that it can be backwards-compatible forever and easily ported to different platforms, etc. For regex specifically, the usual "regex" crate is fantastic and provides very good performance guarantees (no exponential behaviour, ever!), unlike a lot of regex libraries. However, it also comes with the tradeoff of not supporting arbitrary lookahead/lookbehind or many regex extensions, so people needing those features might need a different crate. There's a nice discussion on another thread: https://news.ycombinator.com/item?id=33510976 https://news.ycombinator.com/item?id=33510976
- dmitriid 4y ago> Rust's philosophy is "batteries not included". In Javascript this led to, well, npm. Because there's no standard library, and everyone ended up reinventing everything.
- sbergot 4y agoI am not sure that python is a shining example for library management. Cargo seems well designed. And the package size is more a cultural thing.
- deleted 4y ago[deleted]
- wongarsu 4y agoAnd Python's philosophy leads to three different HTTP libraries in the standard library, but everyone still just does "pip install requests". It's easy to find drawbacks of either extreme. I find Rust's approach of favoring competing, versioned libraries over standard library inclusion generally works well. Many of the libraries managed to significantly improve performance and ergonomics because they can have space to improve their API, and explore which behaviors and guarantees are useful. The biggest downside is that it's more difficult to protect against supply chain attacks.
- nulltype 4y agoWhat a great name!
- stabbles 4y agoSounds like it could be optimized to run 10x faster on a single thread. ~7MB/s is not that fast
- moss2 4y agoWhat's a tokenizer?
- sbergot 4y agoIt is a processing step in natural language processing: https://www.analyticsvidhya.com/blog/2020/05/what-is-tokenization-nlp/ https://www.analyticsvidhya.com/blog/2020/05/what-is-tokeniz...
- teruakohatu 4y agoExample: Split an input into tokens. Tokens: [split, an, input, into, tokens, .]
- deelly 4y agoExample: umbrella Tokens: [umb, rella] ?
- artifabrian 4y agoCould be as well, given there's many varieties of tokenisers, each with different pros and cons. This particular tokenizer is very interesting given that it tries to be best of both worlds (word-level tokenizer and character-level).
- Omie6541 4y agocan we please have some good example input/outputs in the readme itself? what is the expected output of print(enc.encode("hello world")) ?
- stabbles 4y agohttps://github.com/openai/tiktoken/blob/main/tests/test_simple_public.py https://github.com/openai/tiktoken/blob/main/tests/test_simp...
- mnks 4y agoHave a look at https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer which uses javascript reimplementation of the GPT-2 / GPT-3 BPE tokenizer. In this case it's [31373, 995].
- chaoz_ 4y agoAre there any performance benchmarks or is it not applicable for tokenization?
- forgingahead 4y agoI know OpenAI has been getting a lot of flack about their seemingly extreme measures of "safety" (and I agree to an extent, although it's more nuanced from my perspective), full kudos to them for open sourcing many useful projects that can serve as the building block for many other projects. From CLIP, to Whisper, and now this project, I do appreciate that effort from their team. So thanks, if you're reading this!
- mellosouls 4y agoThe main complaints against OpenAI have been it's lack of openness and betrayal of it's claimed founding principles. In that context "open sourcing many useful projects" seems the bare minimum it should be doing, because that was the promise still enshrined in it's name - this is not a bog-standard commercial organisation where that would not be expected. To be clear OpenAI does deserve massive kudos for it's achievements but openness is not among them.
- krageon 4y agoIt's not for safety, it is to make money. The "open" storyline is to get relatively cheap goodwill off things that market their paid products.
- est 4y agoRequires az:// blob download. I hope pypi libraries can provide complete standalone offline versions instead of requests+urllib3+some_object_storage shenanigans. If these blobs are too large to host it on pypi, maybe give us an alternative way to download it altogether so we can deploy the full lib to a server without network access?
- capableweb 4y agoLots of ML/AI stuff wants to your sign some sort of license before you can use it, being able to deploy stuff offline kind of defeats that. Although of course there are ways around it (download once, store yourself, put in right directory), but they are unlikely to help you work around that.
- yencabulator 4y ago> Lots of ML/AI stuff wants to your sign some sort of license What's the point of this MIT license, then: https://github.com/openai/tiktoken/blob/main/LICENSE https://github.com/openai/tiktoken/blob/main/LICENSE
- capableweb 4y agoMaybe the license is for the code, not the model itself?
- yencabulator 4y agoWhat's the point of an "open source project" that requires a proprietary data set with seemingly no license? It might as well just the fetch & execute a random executable.
- seemethere 4y agoThis is most likely because pypi has size restrictions on what you can upload and most users won't go the extra mile to actually pip install from your bespoke download site.
- obert 4y ago- python: https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast https://huggingface.co/docs/transformers/model_doc/gpt2#tran... - javascript: https://www.npmjs.com/package/gpt-3-encoder https://www.npmjs.com/package/gpt-3-encoder - c# https://github.com/dluc/openai-tools https://github.com/dluc/openai-tools - java: https://www.reddit.com/r/MachineLearning/comments/upej7e/p_java_implementation_of_gpt2_tokenizer/ https://www.reddit.com/r/MachineLearning/comments/upej7e/p_j... - php: https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP
- pifm_guy 4y agoI don't understand why you'd want a tokenizer to be fast... Surely the output of said tokenizer you're going to stick through a machine learning model that is orders of magnitude slower? So it really doesn't matter if the tokenizer takes microseconds or milliseconds, when the main model takes seconds for the same input/output.
- manholio 4y agoSurely there can't be a microsecond to second ratio between the two steps, because then they would never be able to crawl the hundreds of billions of documents the model is trained on. Once the model is built, sure, the front end is irrelevant.
- mirker 4y agoInference is latency sensitive so the front end is still relevant.
- pifm_guy 4y agoThe ratio between the tokenizer and the model is constant for both training and inference. And I believe the ratio is that big - you just use a lot of compute for training, and that's why it's so expensive.
- black_puppydog 4y agoWould be curious to know what hardware the benchmark was run on. That drop-off beyond 16 threads is steep...
- ipsum2 4y agoLooks like they have 4 different tokenizers. Besides gpt2, does anyone know which models correspond to which tokenizers? One of them is probably Codex. "gpt2": gpt2, "r50k_base": r50k_base, "p50k_base": p50k_base, "cl100k_base": cl100k_base,
- minimaxir 4y agocl100k_base is a new tokenizer that is apparently being used in their new Embeddings project.
- make3 4y agotoo bad it's not compatible with huggingface tokenizer configs and need its own
- Llamamoe 4y agoAre GPT models even bottlenecked by input tokenization? What real world speedup does this actually translate into?
- Jensson 4y agoCould be used for their wrapper API where they check and alter your input and the models output. This way they could quickly detect "bad words" and so on and return their standard platitudes about how the model can't answer. Or maybe they do a quick pass on websites and filter out websites with bad words quickly, just leaving a small fraction for the model to train on. Parsing to get the standardized tokens quickly would make that easier.
- mixeden 4y agoAt first I thought this is about TikTok
- deleted 4y ago[deleted]
- mark_l_watson 4y agoAny information on which human languages it works with? Some languages like Thai can be challenging so I am wondering how general this is.
- joelburget 4y agoIt works on all human languages, just inefficiently. I ran it over a sample I found on wikipedia: sample = "ฟองมันฟันหนู, ฟันหนูฟองมัน, ฝนทองฟองมัน" len(sample), len(enc.encode(sample)) This returns `39, 40` so it's just encoding one character at a time. It's probably like this for almost all non-English text.
- theragra 4y agoYeah, at least it does it with Russian
- joelburget 4y agoA few interesting findings: * the cl100k_base tokenizer has ~100k tokens -- previous tokenizers had ~50k. (enc.n_vocab gives 100277 but some numbers in that range don't work, starting at 100256) * it has exactly 1110 tokens which are just digits. 10 1 digit tokens, 100 2 digit tokens and 1000 3 digit tokens! (none have preceding spaces). this is a huge improvement from GPT2's tokenizer, which was a huge mess. * there are <|fim_prefix|>, <|fim_middle|>, and <|fim_suffix|> tokens (see Efficient Training of Language Models to Fill in the Middle) The biggest news to me is the improved handling of numbers. This could explain some improved performance on arithmetic. One disappointment is that it tokenizes from the front, e.g. "1000000" -> 100|000|0. This is one of those "so close!" moments -- I would work for free to fix this.
- FL33TW00D 4y agoCan someone optimize this further? Seems like there is significant low hanging fruit, as evidenced by this line: https://github.com/openai/tiktoken/blob/main/src/lib.rs#L419 https://github.com/openai/tiktoken/blob/main/src/lib.rs#L419