3 ms·

It's all very clear when you mentally replace "LLM" with "text completion driven by compressed training data". E.g. [Text copletion driven by compressed train

It's all very clear when you mentally replace "LLM" with "text completion driven by compressed training data". E.g. [Text copletion driven by compressed training data] exhibit[s] a puzzling inconsistency: [it] solves complex problems yet frequently fail[s] on seemingly simpler ones. Some problems are better represented by a locus of texts in the training data, allowing more plausible talk to be generated. When the problem is not well represented, it does not help that the problem is simple. If you train it on nothing but Scientology documents, and then ask about the Buddhist perspective on a situation, you will probably get some nonsense about body thetans, even if the situation is simple.

th0ma5 1y ago
Thank you for posting this. I'm struck with how there is a lot of studying of the behavior and isolating it from other assumptions and then these individual capabilities are then described as a new solution or discovered capability that would work with all of those other assumptions. This makes most all of the LLM research feel like whack a mole if the goal was to make accurate and reliable models by understanding these techniques. Instead, it's more like seeing faces in cars and buildings and other artifacts of patterns and pattern groupings and recognition of patterns. Building houses on sand, etc.
lukev 1y ago
Well, that's what a LLM is. The problem is if one's mental model is built on "AI" instead of "LLM." The fact that LLMs can abstract concepts and do any amount of out-of-sample reasoning is impressive and interesting, but the null hypothesis for a LLM being "impressive" in any regard is that the data required to answer the question is present in it's training set.
XenophileJKO 1y ago
This is true, but also misleading. We are learning that the models achieve compression by distilling higher level concepts and deriving generalized human like abilities, for example the recent introspection paper from Anthropic.
- deleted 1y ago
  [deleted]
layoric 1y ago
I have a hard time trying to conceptualize lossy text compression, but I've recently started to think about the "reasoning"/output as just a by product of lossy compression, and weights tending towards an average of the information "around" the main topic of prompt. What I've found easier is thinking about it like lossy image compression, generating more output tokens via "reasoning" is like subdividing nearby pixels and filling in the gaps with values that they've seen there before. Taking the analogy a bit too far, you can also think of the vocabulary as the pixel bit depth. I definitely agree replacing AI or LLMs with "X driven by compressed training data" starts to make a lot more sense, and a useful shortcut.
- suprjami 1y ago
  You're right about "reasoning". It's just trying to steer the conversation in a more relevant direction in vector space, hopefully to generate more relevant output tokens. I find it easier to conceptualize this in three dimensions. 3blue1brown has a good video series which covers the overall concept of LLM vectors in machine learning: https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_... To give a concrete example, say we're generating the next token from the word "queen". Is this the monarch, the bee, the playing card, the drag entertainer? By adding more relevant tokens (honey, worker, hive, beeswax) we steer the token generation to the place in the "word cloud" where our next token is more likely to exist. I don't see LLMs as "lossy compression" of text. To me that implies retrieval, and Transformers are a prediction device, not a retrieval device. If one needs retrieval then use a database.
  - Terr_ 1y ago
    > You're right about "reasoning". It's just trying to steer the conversation in a more relevant direction in vector space, hopefully to generate more relevant output tokens. I like to frame it as a theater-script cycling through the LLM. The "reasoning" difference is just changing the style so that each character has film noir monologues. The underlying process hasn't really changes, and the monologues text isn't fundamentally different from dialogue or stage-direction... but more data still means more guidance for each improv-cycle. > say we're generating the next token from the word "queen". Is this the monarch, the bee, the playing card, the drag entertainer? I'd like to point out that this scheme can result in things that look better to humans in the end... even when the "clarifying" choice is entirely arbitrary and irrational. In other words, we should be alert to the difference between "explaining what you were thinking" versus "picking a firm direction so future improv makes nicer rationalizations."
onraglanroad 1y ago
> Text copletion driven by compressed training data...solves complex problems Sure it does. Obviously. All we ever needed was some text completion. Thanks for your valuable insight.
ToValueFunfetti 1y ago
Why shouldn't you expect a problem's simplicity to correlate tremendously with how well it is represented in training data? Every angle I can think of tilts in that direction. Simpler problems are easier to remember and thus repeat, they come up more often, asd they require less space/time/effort to record (which also means they are less likely to contain errors).
N_Lens 1y ago
This is a popular take on HN yet incomplete in its assessment of LLMs and their capabilities.
keeganpoppen 1y ago
oh man i am pretty tired of the “it’s just autocomplete” armchair warriors… it is an accurate metaphor in only the most pedantic of ways, and has zero explanatory power whatsoever as far as intuition building goes. and i don’t even understand the impulse. “reality is easy, it’s just quantum autocomplete!”
msla 1y ago
> It's all very clear when you mentally replace "LLM" with "text completion driven by compressed training data". So you replace a more useful term with a less useful one? Is that due to political reasons?
msla 1y ago
> It's all very clear when you mentally replace "LLM" with "text completion driven by compressed training data". This isn't what LLMs are, of course, but what some political groups insist they are so they can strengthen copyright law by pointing to LLMs as "theft". It's all very pro-Disney, of course.

th0ma5 1y ago

Thank you for posting this. I'm struck with how there is a lot of studying of the behavior and isolating it from other assumptions and then these individual capabilities are then described as a new solution or discovered capability that would work with all of those other assumptions. This makes most all of the LLM research feel like whack a mole if the goal was to make accurate and reliable models by understanding these techniques. Instead, it's more like seeing faces in cars and buildings and other artifacts of patterns and pattern groupings and recognition of patterns. Building houses on sand, etc.

lukev 1y ago

Well, that's what a LLM is. The problem is if one's mental model is built on "AI" instead of "LLM." The fact that LLMs can abstract concepts and do any amount of out-of-sample reasoning is impressive and interesting, but the null hypothesis for a LLM being "impressive" in any regard is that the data required to answer the question is present in it's training set.

XenophileJKO 1y ago

This is true, but also misleading. We are learning that the models achieve compression by distilling higher level concepts and deriving generalized human like abilities, for example the recent introspection paper from Anthropic.

deleted 1y ago
[deleted]

layoric 1y ago

I have a hard time trying to conceptualize lossy text compression, but I've recently started to think about the "reasoning"/output as just a by product of lossy compression, and weights tending towards an average of the information "around" the main topic of prompt. What I've found easier is thinking about it like lossy image compression, generating more output tokens via "reasoning" is like subdividing nearby pixels and filling in the gaps with values that they've seen there before. Taking the analogy a bit too far, you can also think of the vocabulary as the pixel bit depth. I definitely agree replacing AI or LLMs with "X driven by compressed training data" starts to make a lot more sense, and a useful shortcut.

suprjami 1y ago
You're right about "reasoning". It's just trying to steer the conversation in a more relevant direction in vector space, hopefully to generate more relevant output tokens. I find it easier to conceptualize this in three dimensions. 3blue1brown has a good video series which covers the overall concept of LLM vectors in machine learning: https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi https://youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_... To give a concrete example, say we're generating the next token from the word "queen". Is this the monarch, the bee, the playing card, the drag entertainer? By adding more relevant tokens (honey, worker, hive, beeswax) we steer the token generation to the place in the "word cloud" where our next token is more likely to exist. I don't see LLMs as "lossy compression" of text. To me that implies retrieval, and Transformers are a prediction device, not a retrieval device. If one needs retrieval then use a database.
- Terr_ 1y ago
  > You're right about "reasoning". It's just trying to steer the conversation in a more relevant direction in vector space, hopefully to generate more relevant output tokens. I like to frame it as a theater-script cycling through the LLM. The "reasoning" difference is just changing the style so that each character has film noir monologues. The underlying process hasn't really changes, and the monologues text isn't fundamentally different from dialogue or stage-direction... but more data still means more guidance for each improv-cycle. > say we're generating the next token from the word "queen". Is this the monarch, the bee, the playing card, the drag entertainer? I'd like to point out that this scheme can result in things that look better to humans in the end... even when the "clarifying" choice is entirely arbitrary and irrational. In other words, we should be alert to the difference between "explaining what you were thinking" versus "picking a firm direction so future improv makes nicer rationalizations."

onraglanroad 1y ago

> Text copletion driven by compressed training data...solves complex problems Sure it does. Obviously. All we ever needed was some text completion. Thanks for your valuable insight.

ToValueFunfetti 1y ago

Why shouldn't you expect a problem's simplicity to correlate tremendously with how well it is represented in training data? Every angle I can think of tilts in that direction. Simpler problems are easier to remember and thus repeat, they come up more often, asd they require less space/time/effort to record (which also means they are less likely to contain errors).

N_Lens 1y ago

This is a popular take on HN yet incomplete in its assessment of LLMs and their capabilities.

keeganpoppen 1y ago

oh man i am pretty tired of the “it’s just autocomplete” armchair warriors… it is an accurate metaphor in only the most pedantic of ways, and has zero explanatory power whatsoever as far as intuition building goes. and i don’t even understand the impulse. “reality is easy, it’s just quantum autocomplete!”

msla 1y ago

> It's all very clear when you mentally replace "LLM" with "text completion driven by compressed training data". So you replace a more useful term with a less useful one? Is that due to political reasons?

msla 1y ago

> It's all very clear when you mentally replace "LLM" with "text completion driven by compressed training data". This isn't what LLMs are, of course, but what some political groups insist they are so they can strengthen copyright law by pointing to LLMs as "theft". It's all very pro-Disney, of course.