9 ms·
The bitter lesson is coming for tokenization
- Scene_Cast2 1y agoI realized that with tokenization, there's a theoretical bottleneck when predicting the next token. Let's say that we have 15k unique tokens (going by modern open models). Let's also say that we have an embedding dimensionality of 1k. This implies that we have a maximum 1k degrees of freedom (or rank) on our output. The model is able to pick any single of the 15k tokens as the top token, but the expressivity of the _probability distribution_ is inherently limited to 1k unique linear components.
- unoti 1y agoI imagine there’s actually combinatorial power in there though. If we imagine embedding something with only 2 dimensions x and y, we can actually encode an unlimited number of concepts because we can imagine distinct separate clusters or neighborhoods spread out over a large 2d map. It’s of course much more possible with more dimensions.
- blackbear_ 1y agoWhile the theoretical bottleneck is there, it is far less restrictive than what you are describing, because the number of almost orthogonal vectors grows exponentially with ambient dimensionality. And orthogonality is what matters to differentiate between different vectors: since any distribution can be expressed as a mixture of Gaussians, the number of separate concepts that you can encode with such a mixture also grows exponentially
- Scene_Cast2 1y agoI agree that you can encode any single concept and that the encoding space of a single top pick grows exponentially. However, I'm talking about the probability distribution of tokens.
- anonymoushn 1y agoI think within the framework of "almost-orthogonal axes" you can still create a vector that has the desired mix of projections onto any combination of these axes?
- yorwba 1y agoNo. You can fit an exponential number of almost-orthogonal vectors into the input space, but the number of not-too-similar probability distributions over output tokens is also exponential in the output dimension. This is fine if you only care about a small subset of distributions (e.g. those that only assign significant probability to at most k tokens), but if you pick any random distribution, it's unlikely to be represented well. Fortunately, this doesn't seem to be much of an issue in practice and people even do top-k sampling intentionally.
- anonymoushn 1y agoI see. You're right. I was either badly mistaken or only thinking about small k.
- kevingadd 1y agoIt seems like you're assuming that models are trying to predict the next token. Is that really how they work? I would have assumed that tokenization is an input-only measure, so you have perhaps up to 50k unique input tokens available, but output is raw text or synthesized speech or an image. The output is not tokens so there are no limitations on the output.
- anonymoushn 1y agoyes, in typical architectures for models dealing with text, the output is a token from the same vocabulary as the input.
- molf 1y agoThe key insight is that you can represent different features by vectors that aren't exactly perpendicular, just nearly perpendicular (for example between 85 and 95 degrees apart). If you tolerate such noise then the number of vectors you can fit grows exponentially relative to the number of dimensions. 12288 dimensions (GPT3 size) can fit more than 40 billion nearly perpendicular vectors. [1]: https://www.3blue1brown.com/lessons/mlp#superposition https://www.3blue1brown.com/lessons/mlp#superposition
- bravesoul2 1y agoHigh dimensions are weird!
- imurray 1y agoA PhD thesis that explores some aspects of the limitation: https://era.ed.ac.uk/handle/1842/42931 https://era.ed.ac.uk/handle/1842/42931 Detecting and preventing unargmaxable outputs in bottlenecked neural networks, Andreas Grivas (2024)
- incognito124 1y ago(I left academia a while ago, this might be nonsense) If I remember correctly, that's not true because of the nonlinearities which provide the model with more expressivity. Transformation from 15k to 1k is rarely an affine map, it's usually highly non-linear.
- deleted 1y ago[deleted]
- cheesecompiler 1y agoThe reverse is possible too: throwing massive compute at a problem can mask the existence of a simpler, more general solution. General-purpose methods tend to win out over time—but how can we be sure they’re truly the most general if we commit so hard to one paradigm (e.g. LLMs) that we stop exploring the underlying structure?
- logicchains 1y agoWe can be sure via analysis based on computational theory, e.g. https://arxiv.org/abs/2503.03961 https://arxiv.org/abs/2503.03961 and https://arxiv.org/abs/2310.07923 https://arxiv.org/abs/2310.07923 . This lets us know what classes of problems a model is able to solve, and sufficiently deep transformers with chain of thought have been shown to be theoretically capable of solving a very large class of problems.
- dsr_ 1y agoA random number generator is guaranteed to produce a correct solution to any problem, but runtime usually does not meet usability standards. Also, solution testing is mandatory. Luckily, you can ask an RNG for that, too, as long as you have tests for the testers already written.
- cheesecompiler 1y agoBut this uses the transformers model to justify its own reasoning strength which might be a blindspot, which is my original point. All the above shows is that transformers can simulate solving a certain set of problems. It doesn't show that they are the best tool for the job.
- yorwba 1y agoKeep in mind that proofs of transformers being able to solve all problems in some complexity class work by taking a known universal algorithm for that complexity class and encoding it as a transformer. In every such case, you'd be better off using the universal algorithm you started with in the first place. Maybe the hope is that you won't have to manually map the universal algorithm to your specific problem and can just train the transformer to figure it out instead, but there are few proofs that transformers can solve all problems in some complexity class through training instead of manual construction.
- marcosdumay 1y agoYeah, make the network deeper. When all you have is a hammer... It makes a lot of sense that a transformation layer that makes the tokens more semantically relevant will help optimize the entire network after it and increase the effective size of your context window. And one of the main immediate obstacle stopping those models from being intelligent is context window size. On the other hand, the current models already cost something on the line of the median country GDP to train, and they are nowhere close to that in value. The saying that "if brute force didn't solve your problem, you didn't apply enough force" is intended to be listened as a joke.
- jagraff 1y agoI think the median country GDP is something like $100 Billion https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP) https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP) Models are expensive, but they're not that expensive.
- telotortium 1y agoLLM model training costs arise primarily from commodity costs (GPUs and other compute as well as electricity), not locally-provided services, so PPP is not the right statistic to use here. You should use nominal GDP for this instead. According to Wikipedia[0], the median country's nominal GDP (Cyprus) is more like $39B. Still much larger than training costs, but much lower than your PPP GDP number. [0] https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal) https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nomi...
- amelius 1y agoMaybe it checks out if you don't use 1 year as your timeframe for GDP but the number of days required for training.
- kordlessagain 1y agoThe median country GDP is approximately $48.8 billion, which corresponds to Uganda at position 90 with $48.769 billion. The largest economy (US) has a GDP of $27.7 trillion. The smallest economy (Tuvalu) has a GDP of $62.3 million. The 48 billion number represents the middle point where half of all countries have larger GDPs and half have smaller GDPs.
- qoez 1y agoThe counter argument is that the theoretical minimum is a few mcdonalds meals a day worth of energy even for the highest ranked human pure mathematician.
- tempodox 1y agoIt's just that no human would live long on McDonalds meals.
- bravetraveler 1y agoPresident in the distance, cursing
- floxy 1y agohttps://www.today.com/health/man-eating-only-mcdonalds-100-days-final-weight-loss-rcna87220 https://www.today.com/health/man-eating-only-mcdonalds-100-d...
- pfdietz 1y agoJust don't drink the sugary soda.
- pylotlight 1y agoye coke zero solves that right :P
- imtringued 1y agoThis makes me wonder if there is a market for weight loss restaurants that automatically size your portions around your diet plan.
- handsclean 1y ago“Somebody survived this for about as long as one can survive starvation, and it made the news” is not ringing endorsement you think it is.
- 1y ago
- andy99 1y ago> inability to detect the number of r's in:strawberry: meme Can someone (who know about LLMs) explain why the r's in strawberry thing is related to tokenization? I have no reason to believe an LLM would be better at counting letters if each was one token. It's not like they "see" any of it. Are they better at counting tokens than letters for some reason? Or is this just one of those things someone misinformed said to sound smart to even less informed people, that got picked up?
- ijk 1y agoWell, which is easier: Count the number of Rs in this sequence: [496, 675, 15717] Count the number of 18s in this sequence: 19 20 18 1 23 2 5 18 18 25
- ASalazarMX 1y agoFor a LLM? No idea. Human: Which is the easier of these formulas 1. x = SQRT(4) 2. x = SQRT(123567889.987654321) Computer: They're both the same.
- drdeca 1y agoDepending on the data types and what the hardware supports, the latter may be harder (in the sense of requiring more operations)? And for a general algorithm bigger numbers would take more steps.
- ijk 1y agoYou can view the tokenization for yourself: https://huggingface.co/spaces/Xenova/the-tokenizer-playground https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun... [496, 675, 15717] is the GPT-4 representation of the tokens. In order to determine which letters the token represents, it needs to learn the relationship between "str" and [496]. It can learn the representation (since it can spell it out as "S-T-R" or "1. S, 2. T, 3. R" or whatever) but it adds an extra step. The question is whether the extra step adds enough extra processing to degrade performance. Does the more compact representation buy enough extra context to make the tokenized version more effective for more problems? It seems like the longer context length makes the trade off worth it, since spelling problems are a relatively minor subset. On the other hand, for numbers it does appear that math is significantly worse when it doesn't have access to individual digits (early Llama math results, for example). Once they changed the digit tokenization, the math performance improved.
- smeeth 1y agoThe main limitation of tokenization is actually logical operations, including arithmetic. IIRC most of the poor performance of LLMs for math problems can be attributed to some very strange things that happen when you do math with tokens. I'd like to see a math/logic bench appear for tokenization schemes that captures this. BPB/perplexity is fine, but its not everything.
- calibas 1y agoIt's a non-deterministic language model, shouldn't we expect mediocre performance in math? It seems like the wrong tool for the job...
- drdeca 1y agoDeterministic is a special case of not-necessarily-deterministic.
- CamperBob2 1y agoWe passed 'mediocre' a long time ago, but yes, it would be surprising if the same vocabulary representation is optimal for both verbal language and mathematical reasoning and computing. To the extent we've already found that to be the case, it's perhaps the weirdest part of this whole "paradigm shift."
- rictic 1y agoModels are deterministic, they're a mathematical function from sequences of tokens to probability distributions over the next token. Then a system samples from that distribution, typically with randomness, and there are some optimizations in running them that introduce randomness, but it's important to understand that the models themselves are not random.
- mgraczyk 1y agoThis is only ideally true. From the perspective of the user of a large closed LLM, this isn't quite right because of non-associativity, experiments, unversioned changes, etc. It's best to assume that the relationship between input and output of an LLM is not deterministic, similar to something like using a Google search API.
- deleted 1y ago[deleted]
- pona-a 1y agoDidn't tokenization already have one bitter lesson: that it's better to let simple statistics guide the splitting, rather than expert morphology models? Would this technically be a more bitter lesson?
- matusp 1y agoAgreed completely. There is a ton of research into how to represent text, and these simple tokenizers are consistently performing on SOTA levels. The bitter lesson is that you should not worry about it that much.
- kingstnap 1y agoSimple statistics aren't some be all. There was a huge improvement in Python coding by fixing the tokenization of indents in Python code. Specifically they made tokens for 4,8,12,16 or something spaces.
- citizenpaul 1y agoThe best general argument I've heard against the bitter lesson is. If the bitter lesson is true? How come we spend so many million man hours a year of tweaking and optimizing software systems all day long? Surely its easier and cheaper to just buy a rack of servers. Maybe if you have infinite compute you don't worry about software design. Meanwhile in the real world... Not only that but where did all these compute optimized solutions come from? Oh yeah millions of man hours of optimizing and testing algorithmic solutions. So unless you are some head in the clouds tenured professor just keep on doing your optimizations and job as usual.
- Uehreka 1y agoBecause the Even Bitterer Lesson is that The Bitter Lesson is true but not actionable. You still have to build the inefficient ”clever” system today because The Bitter Lesson only tells you that your system will be obliterated, it doesn’t tell you when. Some systems built today will last for years, others will last for weeks, others will be obsoleted before release, and we don’t know which are which. I’m hoping someday that dude releases an essay called The Cold Comfort. But it’s impossible to predict when or who it will help, so don’t wait for it.
- citizenpaul 1y agoYeah I get it. I just don't like that is always sorta framed as a can't win don't try message.
- Jensson 1y ago> I just don't like that is always sorta framed as a can't win don't try message. That is why its called bitter, it isn't a fun realization.
- nullc 1y agoThe principal of optimal slack tells you that if your training will take N months on current computing hardware that you should go spend Y months at the beach before buying the computer, and you will complete your task in better than N-Y months thanks to improvements in computing power. Of course, instead of the beach one could spend those Y months improving the algorithms... but it's never wise to bid against yourself if you don't have to. A colloquially is that to maximize your beach time you should work on the biggest N possible, neatly explaining the popularity of AI startups.
- blixt 1y agoI’m starting to think “The Bitter Lesson” is a clever sounding way to give shade to people that failed to nail it on their first attempt. Usually engineers build much more technology than they actually end up needing, then the extras shed off with time and experience (and often you end up building it again from scratch). It’s not clear to me that starting with “just build something that scales with compute” would get you closer to the perfect solution, even if as you get closer to it you do indeed make it possible to throw more compute at it. That said the hand coded nature of tokenization certainly seems in dire need of a better solution, something that can be learned end to end. And It looks like we are getting closer with every iteration.
- RodgerTheGreat 1y agoThe bitter lesson says more about medium-term success at publishable results than it does about genuine scientific progress or even success in the market.
- QuesnayJr 1y agoI'm starting to think that half the commenters here don't actually know what "The Bitter Lesson" is. It's purely a statement about the history of AI research, in a very short essay by Rich Sutton: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html It's not some general statement about software engineering for all domains, but a very specific statement about AI applications. It's an observation that the previous generation's careful algorithmic work to solve an AI problem ends up being obsoleted by this generation's brute force approach using more computing power. It's something that's happened over and over again in AI, and has happened several times even since 2019 when Sutton wrote the essay.
- tantalor 1y agoThat essay is actually linked in the lead: > As it's been pointed out countless times - if the trend of ML research could be summarised, it'd be the adherence to The Bitter Lesson - opt for general-purpose methods that leverage large amounts of compute and data over crafted methods by domain experts But we're only 1 sentence in, and this is already a failure of science communication at several levels. 1. The sentence structure and grammar is simply horrible 2. This is condescending: "pointed out countless times" - has it? 3. The reference to Sutton's essay is oblique, easy to miss 4. Outside of AI circles, "Bitter Lesson" is not very well known. If you didn't already know about it, this doesn't help.
- perching_aix 1y agoCan't wait for models to struggle with adhering to UTF-8.
- resters 1y agoTokenization as a form of preprocessing has the problems the authors mention. But it is also a useful way to think about data vs metadata and moving beyond text/image io into other domains. Ultimately we need symbolic representations of things, sure they are all ultimately bytes which the model could learn to self-organize, but things like that can be useful when humans interact with the data directly, in a sense, tokens make more aspects of LLM internals "human readable", and models should also be able to learn to overcome the limitations of a particular tokenization scheme.
- fooker 1y ago‘Bytes’ is tokenization. There’s no reason to assume it’s the best solution. It might be the case that a better tokenization scheme is needed for math, reasoning, video, etc models.
- kgeist 1y ago>From a domain point of view, some are skeptical that bytes are adequate for modelling natural language If I remember correctly, GPT3.5's tokenizer treated Cyrillic as individual characters, and GPT3.5 was pretty good at Russian.
- dgfitz 1y agoI wonder if they treat each letter as a Unicode code point, and each of those is a token? I could see the same being true of other languages.
- broses 1y agoThis gave me an idea: we can take a mixture of tokenizations with learned weights, just like taking a mixture of experts with learned weights. BLT is optimized for compression, but an approach like this could be optimized directly for model performance, and really learn to skim. Concretely: we learn a medium sized model that takes a partial tokenization and outputs a probability distribution over the endpoints of the next token (say we let the token lengths range from 1 to 64 bytes, the model outputs 64 logits). Then we do a beam search to find the, say, 4 most likely tokenizations. Then we run the transformer on all four tokenizations, and we take the expected value of the loss to be the final loss. If we train this on prompt-response pairs, so that it only has to learn what to say and doesn't have to predict the context, then it could learn to skim boring stuff by patching it into ~64 byte tokens. Or more if we want. And ofc we'd use a short context byte level transformer to encode/decode tokens to vectors. Idk this idea is kinda half baked.
- physix 1y agoI think that's what evolution did when developing the brain! :-) I'm a total noob in ML. I just had to vent something for not understanding this stuff and realizing that knowing physics doesn't mean you can grok ML mechanics.
- curtisszmania 1y ago[dead]
- rryan 1y agoDon't make me tap the sign: There is no such thing as "bytes". There are only encodings. UTF-8 is the encoding most people are using when they talk about modeling "raw bytes" of text. UTF-8 is just a shitty (biased) human-designed tokenizer of the unicode codepoints.
- cschmidt 1y agoVirtually all current tokenization schemes do work at the raw byte level, not the utf-8 character. They do this to avoid the Out of Vocabulary (OOV) or unknown token problem. In older models, if you came across something in the data you can't tokenize, you add a <UNK>. But tokenization should be exactly reversible, so now people use subword tokenizers including all 256 single bytes in the vocab. That way you can always represent any text by dropping down to the single byte level. The other alternative would be to add all utf-8 code points to the vocabulary, but there are more than 150k of those, and enough are rare, that many would be undertrained. You'd have a lot of glitch tokens (https://arxiv.org/abs/2405.05417 https://arxiv.org/abs/2405.05417). That does mean an LLM isn't 100% guaranteed to output well formed utf-8.
- cschmidt 1y agoAnd in regard to utf-8 being a shitty biased tokenizer, here is recent paper trying to design a better style of encoding https://arxiv.org/abs/2505.24689 https://arxiv.org/abs/2505.24689
- hiddencost 1y agoWell akshually... I assume you started programming some time this millennia? That's the only way I can explain this "take".
- roflcopter69 1y agoCare to elaborate?
- vaxman 1y agoRoger, who spoke only Chinglish and never paused between words, was working on a VAX FORTRAN program that exchanged tapes with IBM mainframes and a memory mapped section, inventing a new word in the process that still has me rolling decades later: ebsah-dicky-asky-codah
- ofou 1y agoThis is stupid because a UTF8 is a tokenizer that covers all Unicode with a vocab of only 256 (yes, without a K). This is the only way of scaling the bitter lesson with tokenizers. Also, with architectures that span +1M context windows, it’s no longer an argument/issue the reduced context windows.
- hoseja 1y agoThere are no r's in strawberry, there are two ɹ's and several dozen achenes. It's a member of Rosaceae by any other name. Fighting with nonsensical english orthography seems kinda pointless to me. Stop trying to make intelligent entities composed of written text.
- ilaksh 1y agoI've always felt that the ideal would be to somehow create many different tokenizations for different use cases. And allow them to sometimes build on each other. Like a lot of domain-specific languages. Maybe there could be something like a mixture-of-experts but with a thousand experts and each has its own tokenization. Fortunately I don't actually understand this stuff, so I am going to go ahead and congratulate myself on my brilliant ideas and let the geniuses work out the details. :P