6 ms·
Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking. I.e. whatever feat of computation needs to happen to produce results you seek,
by TeMPOraL 7mo ago
Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking. I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you can guarantee not to get a good answer because it requires more computation than possible with the tokens produced.
I.e. by demanding the model to be concise, you're literally making it dumber.
(Separating out "chain of thought" into "thinking mode" and removing user control over it definitely helped with this problem.)
- andai 7mo agoI remember a while back they found that replacing reasoning tokens with placeholders ("....") also boosted results on benchies. But does talk like caveman make number go down? Less token = less think? I also wondered, due to the way LLMs work, if I ask AI a question using fancy language, does that make it pattern match to scientific literature, and therefore increase the probability that the output will be true?
- baq 7mo agoDo you know of evals with default Claude vs caveman Claude vs politician Claude solving the same tasks? Hypothesis is plausible, but I wouldn’t take it for granted
- Rexxar 7mo ago> Someone didn't get the memo that for LLMs, tokens are units of thinking. Where do you get this memo ? Seems completely wrong to me. More computation does not translate to more "thinking" if you compute the wrong things (ie things that contribute significantly to the final sentence meaning).
- staminade 7mo agoThat’s why you need filler words that contribute little to the sentence meaning but give it a chance to compute/think. This is part of why humans do the same when speaking.
- jaccola 7mo agoDo you have any evidence at all of this? I know how LLMs are trained and this makes no sense to me. Otherwise you'd just put filler words in every input e.g. instead of: "The square root of 256 is" you'd enter "errr The er square um root errr of 256 errr is" and it would miraculously get better? The model can't differentiate between words you entered and words it generated its self...
- lijok 7mo agoYou’re conflating training and inference
- staminade 7mo agoWhat do you think chain of thought reasoning is doing exactly?
- muzani 7mo agoIt's why it starts with "You're absolutely right!" It's not to flatter the user. It's a cheap way to guide the response in a space where it's utilizing the correction.
- mike_hearn 7mo agoPeople have researched pause tokens for this exact reason.
- dTal 7mo agoThe LLM has no accessible state beyond its own output tokens; each pass generates a single token and does not otherwise communicate with subsequent passes. Therefore all information calculated in a pass must be encoded into the entropy of the output token. If the only output of a thinking pass is a dumb filler word with hardly any entropy, then all the thinking for that filler word is forgotten and cannot be reconstructed.
- NiloCK 7mo agoI agree with this take in general, but I think we need to be prepared for nuance when thinking about these things. Tokens are how an LLM works things out, but I think it's just as likely as not that LLMs (like people) are capable of overthinking things to the point of coming to a wrong answer when their "gut" response would have been better. I do not content that this is the default mode, but that it is both possible, and that it's more or less likely on one kind of problem than another, problem categories to be determined. A specific example of this was the era of chat interfaces that leaned too far in the direction of web search when responding to user queries. No, claude, I don't want a recipe blogspam link or summary - just listen to your heart and tell me how to mix pancakes. More abstractly: LLMs give the running context window a lot of credit, and will work hard to post-hoc rationalize whatever is in there, including any prior low-likelihood tokens. I expect many problematic 'hallucinations' are the result of an unlucky run of two or more low probability tokens running together, and the likelihood of that happening in a given response scales ~linearly with the length of response.
- samus 7mo agoThe solution to that is turning off thinking mode or reducing thinking budget.
- avaer 7mo agoThat was my first thought too -- instead of talk like a caveman you could turn off reasoning, with probably better results. Additionally, LLMs do not actually operate in text; much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. So unless the LLM was trained otherwise, making it talk like a caveman is more than just theoretically turning it into a caveman.
- DrewADesign 7mo ago> much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. What do you mean by that? It’s literally text prediction, isn’t it?
- cyanydeez 7mo agoThere was a paper recently that demonstrated that you can input different human languages and the middle layers of the model end up operating on the same probabilistic vectors. It's just the encoding/decoding layers that appear to do the language management. So the conclusion was that these middle layers have their own language and it's converting the text into this language and this decoding it. It explains why sometime the models switch to chinese when they have a lot of chinese language inputs, etc.
- DrewADesign 7mo agoOk — that sounds more like a theory rather than an open-and-shut causal explanation, but I’ll read the paper.
- trenchgun 7mo agoYou’re a literature cycle behind. ‘Middle-layer shared representations exist’ is the observed phenomenon; ‘why exactly they form’ is the theory. You are also confusing ‘mechanistic explanation still incomplete’ with ‘empirical phenomenon unestablished.’ Those are not the same thing. PS. Em dash? So you are some LLM bot trying to bait mine HN for reasoning traces? :D
- agumonkey 7mo agoHow do we know if a token sits at an abstract level or just the textual level ?
- raincole 7mo agoWhen it comes to LLM you really cannot draw conclusions from first principles like this. Yes, it sounds reasonable. And things in reality aren't always reasonable. Benchmark or nothing.
- samus 7mo agoThere have been papers about introducing thinking tokens in intermediary layers that get stripped from the output.
- kubb 7mo agoThis is condescending and wrong at the same time (best combo). LLMs do stumble into long prediction chains that don’t lead the inference in any useful direction, wasting tokens and compute.
- prodigycorp 7mo agoAre you sure about that? Chain of thought does not need to be semantically useful to improve LLM performance. https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758
- davidguetta 7mo agostill doesn't mean all tokens are useful. it's the point of benchmarks
- prodigycorp 7mo agoCare to share the benchmarks backing the claims in this repo?
- kubb 7mo agoIf you're misusing LLMs to solve TC^0 problems, which is what the paper is about, then... you also don't need the slop lavine. You can just inject a bunch of filler tokens yourself.
- otabdeveloper4 7mo agoLLMs don't think at all. Forcing it to be concise doesn't work because it wasn't trained on token strings that short.
- HumanOstrich 7mo ago> Forcing it to be concise doesn't work because it wasn't trained on token strings that short. This is a 2023-era comment and is incorrect.
- otabdeveloper4 7mo agoLLMs architectures have not changed at all since 2023. > but mmuh latest SOTA from CloudCorp (c)! You don't know how these things work and all you have to go on is marketing copy.
- HumanOstrich 7mo agoYea you don't know anything about LLM architectures. They often change with each model release. You also aren't aware that there's more to it than "LLM architecture". And you're rather confident despite your lack of knowledge. You're like the old LLMs before ChatGPT was released that were kinda neat, but usually wrong and overconfident about it.
- otabdeveloper4 7mo agoIt's still attention and next-token-prediction and nothing else. The only new innovation is MoE, something that's used to optimize local models and not for the "SOTA" cloud offerings you're so fond of.
- HumanOstrich 7mo agoYou no listen. Me give up. Go learn on fruit phone.
- deleted 7mo ago[deleted]
- cyanydeez 7mo agoIt's not "units of thinking" its "units of reference"; as long as what it produces references the necessary probabilistic algorithms, itll do just fine.
- jstummbillig 7mo agoWhat do you mean? The page explicitly states: > cutting ~75% of tokens while keeping full technical accuracy. I have no clue if this claim holds, but alas, just pretending they did not address the obvious criticism, while they did, is at the very least pretty lazy. An explanation that explains nothing is not very interesting.
- getpokedagain 7mo agoIn the age of vibe coding and that we are literally talking about a single markdown file I am sure this has been well tested and achieves all of its goals with statistical accuracy, no side effects with no issues.
- systoll 7mo agoThe author pretended they addressed the obvious criticism. You can read the skill. They didn't do anything to mitigate the issue, so the criticism is valid.
- prodigycorp 7mo agoThe burden of proof is on the author to provide at least one type of eval for making that claim.
- jstummbillig 7mo agoI notice that the number of people confidently talking about "burden of proof" and whose it allegedly is in the context of AI has gone up sharply. Nobody has to proof anything. It can give your claim credibility. If you don't provide any, an opposing claim without proof does not get any better.
- prodigycorp 7mo agoSorry I don't know how engaging in this could lead to anything productive. There's already literature out there that gives credence to TeMPOraL claim. And, after a certain point, gravity being the reason that things fall becomes so self evident that every re-statements doesnt not require proof.
- afro88 7mo agoIIUC this doesn't make the LLM think in caveman (thinking tokens). It just makes the final output show in caveman.
- vova_hn2 7mo agoYeah, I don't think that "I'd be happy to help you with that" or "Sure, let me take a look at that for you" carries much useful signal that can be used for the next tokens.
- jerf 7mo agoThere is a study that shows that what the model is doing behind the scenes in those cases is a lot more than just outputting those tokens. For an LLM, tokens are thought. They have no ability to think, by whatever definition of that word you like, without outputting something. The token only represents a tiny fraction of the internal state changes made when a token is output. Clearly there is an optimal for each task (not necessarily a global one) and a concrete model for a given task can be arbitrarily far from it. But you'd need to test it out for each case, not just assume that "less tokens = more better". You can be forcing your model to be dumber without realizing it if you're not testing.
- DonHopkins 7mo agoHigh dimensional vectors are thought (insofar as you can define what that even means). Tokens are one dimensional input that navigates the thought, and output that renders the thought. The "thinking" takes place in the high dimension space, not the one dimensional stream of tokens.
- gchamonlive 7mo agoBut isn't the one dimensional tokens a reflex of high dimensional space? What you see is "sure let's take a look at that" but behind the curtains it's actually an indication that it's searching a very specific latent space which might be radically different if those tokens didn't exist. Or not. In any case, you can't just make that claim and isolate those two processes. They might be totally unrelated but they also might be tightly interconnected.
- sheiyei 7mo agoI assume in practice, filler words do nothing of value. When words add or mean nothing (their weights are basically 0 in relation to the subject), I don't see why they'd affect what the model outputs (except cause more filler words)?
- kogold 7mo ago[flagged]
- ShowalkKama 7mo agothe fact that more tokens = more smart should be expected given cot / thinking / other techniques that increase the model accuracy by using more tokens. Did you test that ""caveman mode"" has similar performance to the ""normal"" model?
- Garlef 7mo agoYes but: If the amount is fixed, then the density matters. A lot of communication is just mentioning the concepts.
- bitexploder 7mo agoThat is part of it. They are also trained to think in very well mapped areas of their model. All the RHLF, etc. tuned on their CoT and user feedback of responses.
- mynegation 7mo agoNo, let me rephrase it for you. “tokens used for think. Short makes model dumb”
- zozbot234 7mo agoGrug says you quite right, token unit thinking, but empty words not real thinking and should avoid. Instead must think problem step by step with good impactful words.
- PufPufPuf 7mo agoYou mention thinking tokens as a side note, but their existence invalidates your whole point. Virtually all modern LLMs use thinking tokens.
- hackerInnen 7mo agoYou are absolutely right! That is exactly the reason why more lines of code always produce a better program. Straight on, m8!
- ZoomZoomZoom 7mo agoThis might be not so far from the truth, if you count total loc written and rewritten during the development cycle, not just the final number. Not everybody is Dijkstra.
- andy99 7mo agoI’ve heard this, I don’t automatically believe it nor do I understand why it would need to be true, I’m still caught on the old fashioned idea that the only “thinking” for autoregressive modes happens during training. But I assume this has been studied? Can anyone point to papers that show it? I’d particularly like to know what the curves look like, it’s clearly not linear, so if you cut out 75% or tokens what do you expect to lose? I do imagine there is not a lot of caveman speak in the training data so results may be worse because they don’t fit the same patterns that have been reinforcement learned in.
- conception 7mo agoI have seen a paper though I can’t find it right now on asking your prompt and expert language produces better results than layman language. The idea of being that the answers that are actually correct will probably be closer to where people who are expert are speaking about it so the training data will associate those two things closer to each other versus Lyman talking about stuff and getting it wrong.
- therealdrag0 7mo agoWe’re years into the industry leaning into “chain of thought” and then “thinking models” that are based on this premise, forcing more token usage to avoid premature conclusions and notice contradictions (I sometimes see this leak into final output). You may remember in the early days users themselves would have to say “think deeply” or after a response “now check your work” and it would find its own “one shot” mistakes often. So it must be studied and at least be proven effective in practice to be so universally used now. Someone else posted a few articles like this in the thread above but there’s probably more and better ones if you search. https://news.ycombinator.com/item?id=47647907 https://news.ycombinator.com/item?id=47647907
- xgulfie 7mo agoAh so obviously making the LLM repeat itself three times for every response it will get smarter
- TeMPOraL 7mo agoYes, and observe that people do that too. It gives them more time to notice their own confusion and go "but wait, that's not right" on you.
- xgulfie 7mo agoYou're absolutely correct! Having the LLM using more tokens does improve its output. Here's why this works: ## More tokens = smarter outputs When an LLM uses tokens, it is putting more information into its context ## Better context, better results The more information the LLM has in its context, the more complete and well thought-through the outputs will be ## More complete thinking When an LLM is able to iterate on itself, results improve ## Better shareholder value Numbers need to go up in order for us to maintain our shareholder value. This means instead of focusing on results that are qualitative, instead the brand should focus on quantitative, hard results
- marginalia_nu 7mo agoI wonder if a language like Latin would be useful. It's a significantly much succinct semantic encoding than English while being able to express all the same concepts, since it encodes a lot of glue words into the grammar of the language, and conventionally lets you drop many pronouns. e.g. "I would have walked home, but it seemed like it was going to rain" (14 words) -> "Domum ambulavissem, sed pluiturum esse videbatur" (6 words).
- dmboyd 7mo agoWords <> tokens
- mike_hearn 7mo agoI think speculative decoding eliminates a lot of the savings people imagine they're getting from making LLMs use strange languages.
- deleted 7mo ago[deleted]
- HarHarVeryFunny 7mo agoThat's going to depend on what model you're using with Claude Code. All of the more recent Anthropic models (4.5 and 4.6) support thinking, so the number of tokens generated ("units of thought") isn't directly tied to the verbosity of input and non-thought output. However, another potential issue is that LLMs are continuation engines, and I'd have thought that talking like a caveman may be "interpreted" as meaning you want a dumbed down response, not just a smart response in caveman-speak. It's a bit like asking an LLM to predict next move in a chess game - it's not going to predict the best move that it can, but rather predict the next move that would be played given what it can infer about the ELO rating of the player whose moves it is continuing. If you ask it to continue the move sequence of a poor player, it'll generate a poor move since that's the best prediction. Of course there's not going to be a lot of caveman speak on stack overflow, so who knows what the impact is. Program go boom. Me stomp on bugs.
- strogonoff 7mo agoA fundamental (but sadly common) error behind “tokens are units of thinking” is antropomorphising the model as a thinking being. That’s a pretty wild claim that requires a lot of proof, and possibly solving the hard problem, before it can be taken seriously. There’s a less magical model of how LLMs work: they are essentially fancy autocomplete engines. Most of us probably have an intuition that the more you give an autocomplete, the better results it will yield. However, does this extend to output of the autocomplete—i.e. the more tokens it uses for the result, the better? It could well be true in context of chain of thought[0] models, in the sense that the output of a preceding autocomplete step is then fed as input to the next autocomplete step, and therefore would yield better results in the end. In other words, with this intuition, if caveman speak is applied early enough in the chain, it would indeed hamper the quality of the end result; and if it is applied later, it would not really save that many tokens. Willing to be corrected by someone more familiar with NN architecture, of course. [0] I can see “thinking” used as a term of art, distinct from its regular meaning, when discussing “chain of thought” models; sort of like what “learning” is in “machine learning”.
- ForceBru 7mo agoIMO "thinking" here means "computation", like running matrix multiplications. Another view could be: "thinking" means "producing tokens". This doesn't require any proof because it's literally what the models do. As I understand it, the claim is: more tokens = more computation = more "thinking" => answer probably better.
- TeMPOraL 7mo agoI don't agree with GP's take on anthropomorphising[0], but in this particular discussion, I meant something even simpler by "thinking" - imagine it more like manually stepping a CPU, or powering a machine by turning a crank. Each output token is kinda like a clock signal, or a full crank turn. There's lots of highly complex stuff happening inside the CPU/machine - circuits switching/gears turning - but there's a limit of how much of it can happen in a single cycle. Say that limit is X. This means if your problem fundamentally requires at least Y compute to be solved, your machine will never give you a reliable answer in less than ceil(Y/N) steps. LLMs are like this - a loop is programmed to step the CPU/turn the crank until the machine emits a magic "stop" token. So in this sense, asking an LLM to be concise means reducing the number of compute it can perform, and if you insist on it too much, it may stop so early as to fundamentally have been unable to solve the problem in computational space allotted. This perspective requires no assumptions about "thinking" or anything human-like happening inside - it follows just from time and energy being finite :). -- [0] - I strongly think the industry is doing a huge disservice avoiding to anthropomorphize LLMs, as treating them as "little people on a chip" is the best high-level model we have for understanding their failure modes and role in larger computing systems - and instead, we just have tons of people wasting their collective efforts trying to fix "lethal trifecta" as if it was a software bug and not fundamental property of what makes LLM interesting. Already wrote more on it in this thread, so I'll stop here.
- taneq 7mo agoMore concise is dumber. Got it.
- Demiurg082 7mo agoCoT token are usually controled via 'extended thinking' or 'adapted thinking'. CoT tokens are usually not affected by the system prompt. There is an effort parameter, though, which states to have an effect on accuracy for over all token consumption. https://platform.claude.com/docs/en/build-with-claude/extended-thinking https://platform.claude.com/docs/en/build-with-claude/extend...
- bitexploder 7mo agoThis helps, but the original prompt is still there. The system prompt is still influencing these thinking blocks. They just don’t end up clogging up your context. The system prompt sits at the very top of the context hierarchy. Even with isolated "thinking" blocks, the reasoning tokens are still autoregressively conditioned on the system instructions. If the system prompt forces "caveman speak" the model's attention mechanisms are immediately biased toward simpler, less coherent latent spaces. You are handicapping the vocabulary and syntax it uses inside its own thinking process, which directly throttles its ability to execute high-level logic. Nothing on that page indicates otherwise.
- Demiurg082 7mo agoI get your point but it seems that extended thinking is based on a hidden system prompt that is not so much affected by the style the user defines. Probably it's a bit in between. https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-extended-thinking.html#claude-messages-extended-thinking-cost https://docs.aws.amazon.com/bedrock/latest/userguide/claude-...
- pxc 7mo agoIf this is true, shouldn't LLMs perform way worse when working in Chinese than in English? Seems like an easy thing to study since there are so many Chinese LLMs that can work in both Cbinese and English. Do LLMs generally perform better in verbose languages than they do in concise ones?
- reedlaw 7mo agoAre you saying Chinese is more concise than English? Chinese poetry is concise, but that can be true in any language. For LLMs, it depends on the tokenizer. Chinese models are of course more Chinese-friendly and so would encode the same sentence with fewer tokens than Western models.
- pxc 7mo ago> Are you saying Chinese is more concise than English? Yeah, definitely. It lacks case and verb conjugations, plus whole classes of filler words, and words themselves are on average substantially shorter. If you listen to or read a hyper-literal transliteration of Chinese speech into English (you can find fun videos of this on Chinese social media), it even resembles "caveman speech" for those reasons. If you look at translated texts and compare the English versions to the Chinese ones, the Chinese versions are substantially shorter. Same if you compare localization strings in your favorite open-source project. It's also part of why Chinese apps are so information-dense, and why localizing to other languages often requires reorganizing the layout itself— languages like English just aren't as information-dense, pixel for pixel. The difference is especially profound for vernacular Chinese, which is why Chinese people often note that text which "has a machine translation flavor" is over-specified and gratuitously prolix. Maybe some of this washes out in LLMs due to tokenization differences. But Chinese texts are typically shorter than English texts and it extends to prose as well as poetry. But yeah this is standard stuff: Chinese is more concise and more contextual/ambiguous. More semantic work is allocated in interpretation than with English, less is allocated in the writing/speaking. Do you speak Chinese and experience the differences between Chinese and English differently? I'm a native English speaker and only a beginner in Chinese but I've formed these views in discussion with Chinese people who know some English as well.
- dTal 7mo agoYeah but not all tokens are created equal. Some tokens are hard to predict and thus encode useful information; some are highly predictable and therefore don't. Spending an entire forward pass through the token-generation machine just to generate a very low-entropy token like "is" is wasteful. The LLM doesn't get to "remember" that thinking, it just gets to see a trivial grammar-filling token that a very dumb LLM could just as easily have made. They aren't stenographically hiding useful computation state in words like "the" and "and".
- Chance-Device 7mo ago> They aren't stenographically hiding useful computation state in words like "the" and "and". Do you know that is true? These aren’t just tokens, they’re tokens with specific position encodings preceded by specific context. The position as a whole is a lot richer than you make it out to be. I think this is probably an unanswered empirical question, unless you’ve read otherwise.
- dTal 7mo agoI am quite certain. The output is "just tokens"; the "position encodings" and "context" are inputs to the LLM function, not outputs. The information that a token can carry is bounded by the entropy of that token. A highly predictable token (given the context) simply can't communicate anything. Again: if a tiny language model or even a basic markov model would also predict the same token, it's a safe bet it doesn't encode any useful thinking when the big model spits it out.
- Chance-Device 7mo agoI just don’t share your certainty. You may or may not be right, but if there isn’t a result showing this, then I’m not going to assume it.
- 8note 7mo agocan you prove this? train an LLM to leave out the filler words, and see it get the same performance at a lower cost? or do it at token selection time?