19 ms·

It would be much more interesting to see PCA (or t-SNE or whatever) on the internal representation within the model itself. As in the activations of a certain n

It would be much more interesting to see PCA (or t-SNE or whatever) on the internal representation within the model itself. As in the activations of a certain number of layers or neurons, as they change from token to token. I don't think the OpenAI embeddings are necessarily an appropriate "map" of the model's internal thoughts. I suppose that raises another questions: Do LLMs "think" in language? Or do they think in a more abstract space, then translate it to language later? My money is on the latter.

higuidebot 2y ago
Text embeddings are underused WRT model understanding IMO. "Interpretability" focuses on more complex tools but perhaps misses some of the basics - shouldn't we have some sort of visual understanding of model thinking?
eightysixfour 2y ago
> I suppose that raises another questions: Do LLMs "think" in language? Or do they think in a more abstract space, then translate it to language later? My money is on the latter. The processing happens in latent space and then is converted to tokens/token space. There is research into reasoning models which can spend extra compute in latent space instead of in token space: https://arxiv.org/abs/2412.06769 https://arxiv.org/abs/2412.06769
- HarHarVeryFunny 2y ago
  I'd have to guess that the "transformations" being made to the embeddings at each layer are basically/mostly just adding (tagging with) incremental levels of additional grammatical/semantic information that has been gleaned by the hierarchical pattern matching that is taking place. At the end of the day our own "thinking" has to be a purely mechanical process, and one also based around pattern recognition and prediction, but "thinking" seems a bit of a loaded term to apply to LLMs given the differences in "cognitive architecture", and smacks a bit of anthromorphism. Reasoning (search-like chained predictions) is more of an algorithmic process, but it seems that the "reactive" pass-thru predictions of the base LLM are more clearly viewed just as pattern recognition and extrapolation/prediction. Prove me wrong!
  - eightysixfour 2y ago
    > Prove me wrong! For future reference, it is hard to parse tone over the internet but this "command" read pretty poorly to me. I would have preferred if you asked a question or something else. However, assuming best intentions... > I'd have to guess that the "transformations" being made to the embeddings at each layer are basically/mostly just adding (tagging with) incremental levels of additional grammatical/semantic information that has been gleaned by the hierarchical pattern matching that is taking place. > Reasoning (search-like chained predictions) is more of an algorithmic process, but it seems that the "reactive" pass-thru predictions of the base LLM are more clearly viewed just as pattern recognition and extrapolation/prediction. I'm having trouble following. Are you saying that: * The "reactive pass-thru predictions" are just pattern matched responses from the training text that come from "incremental levels of additional semantic information" * There is some other algorithmic process which results in "search-like chained predictions" from the pattern matched responses * These two capabilities, combined in a single "thing," are not analogous to thinking ? > At the end of the day our own "thinking" has to be a purely mechanical process, and one also based around pattern recognition and prediction, but "thinking" seems a bit of a loaded term to apply to LLMs given the differences in "cognitive architecture", and smacks a bit of anthromorphism. You can pick whatever term you like. What we seem to have is a system which can, through the embedded patterns of language, create a recursive search through a problem space and try and solve it by exploring plausible answers. If my dog came up with a hypothesis based on patterns it had previously observed, considered that hypothesis, discarded it, and then came up with a new hypothesis, I'd say it was thinking. There are clear gaps between where we are and human capabilities especially as it relates to memory, in-context learning, and maintaining coherence over many iterations (well, some humans), but (to me) one of two things is probably true: 1. Models are doing something analogous to thinking that we don't understand. 2. Thinking is just a predict-act-evaluate loop with pattern matching to generate plausible predictions. I lean towards the second. That's not to ignore the complexity of the human brain, it is just that the core process seems quite clear in the abstract to me via both observation and introspection. What can "thinking" (as you define it) do that is beyond these capabilities?
    - HarHarVeryFunny 2y ago
- sgk284 2y ago
  Another take on a similar idea from FAIR is a Large Concept Model: https://arxiv.org/pdf/2412.08821 https://arxiv.org/pdf/2412.08821
  - eightysixfour 2y ago
    Sort of, yes. I think one of our next unlocks will be some kind of system which predicts at multiple levels in latent space. Something like predicting the paragraph, then the sentences in the paragraph, then the words in the sentences, where the higher level "decisions" are a constraint that guides the lower level generation. In meta's byte-level model they made the tokens variable length based on how predictable the bytes were to a smaller model, allocating compute resources based on entropy.

higuidebot 2y ago

Text embeddings are underused WRT model understanding IMO. "Interpretability" focuses on more complex tools but perhaps misses some of the basics - shouldn't we have some sort of visual understanding of model thinking?

eightysixfour 2y ago

> I suppose that raises another questions: Do LLMs "think" in language? Or do they think in a more abstract space, then translate it to language later? My money is on the latter. The processing happens in latent space and then is converted to tokens/token space. There is research into reasoning models which can spend extra compute in latent space instead of in token space: https://arxiv.org/abs/2412.06769 https://arxiv.org/abs/2412.06769

HarHarVeryFunny 2y ago
I'd have to guess that the "transformations" being made to the embeddings at each layer are basically/mostly just adding (tagging with) incremental levels of additional grammatical/semantic information that has been gleaned by the hierarchical pattern matching that is taking place. At the end of the day our own "thinking" has to be a purely mechanical process, and one also based around pattern recognition and prediction, but "thinking" seems a bit of a loaded term to apply to LLMs given the differences in "cognitive architecture", and smacks a bit of anthromorphism. Reasoning (search-like chained predictions) is more of an algorithmic process, but it seems that the "reactive" pass-thru predictions of the base LLM are more clearly viewed just as pattern recognition and extrapolation/prediction. Prove me wrong!
- eightysixfour 2y ago
  > Prove me wrong! For future reference, it is hard to parse tone over the internet but this "command" read pretty poorly to me. I would have preferred if you asked a question or something else. However, assuming best intentions... > I'd have to guess that the "transformations" being made to the embeddings at each layer are basically/mostly just adding (tagging with) incremental levels of additional grammatical/semantic information that has been gleaned by the hierarchical pattern matching that is taking place. > Reasoning (search-like chained predictions) is more of an algorithmic process, but it seems that the "reactive" pass-thru predictions of the base LLM are more clearly viewed just as pattern recognition and extrapolation/prediction. I'm having trouble following. Are you saying that: * The "reactive pass-thru predictions" are just pattern matched responses from the training text that come from "incremental levels of additional semantic information" * There is some other algorithmic process which results in "search-like chained predictions" from the pattern matched responses * These two capabilities, combined in a single "thing," are not analogous to thinking ? > At the end of the day our own "thinking" has to be a purely mechanical process, and one also based around pattern recognition and prediction, but "thinking" seems a bit of a loaded term to apply to LLMs given the differences in "cognitive architecture", and smacks a bit of anthromorphism. You can pick whatever term you like. What we seem to have is a system which can, through the embedded patterns of language, create a recursive search through a problem space and try and solve it by exploring plausible answers. If my dog came up with a hypothesis based on patterns it had previously observed, considered that hypothesis, discarded it, and then came up with a new hypothesis, I'd say it was thinking. There are clear gaps between where we are and human capabilities especially as it relates to memory, in-context learning, and maintaining coherence over many iterations (well, some humans), but (to me) one of two things is probably true: 1. Models are doing something analogous to thinking that we don't understand. 2. Thinking is just a predict-act-evaluate loop with pattern matching to generate plausible predictions. I lean towards the second. That's not to ignore the complexity of the human brain, it is just that the core process seems quite clear in the abstract to me via both observation and introspection. What can "thinking" (as you define it) do that is beyond these capabilities?
  - HarHarVeryFunny 2y ago
sgk284 2y ago
Another take on a similar idea from FAIR is a Large Concept Model: https://arxiv.org/pdf/2412.08821 https://arxiv.org/pdf/2412.08821
- eightysixfour 2y ago
  Sort of, yes. I think one of our next unlocks will be some kind of system which predicts at multiple levels in latent space. Something like predicting the paragraph, then the sentences in the paragraph, then the words in the sentences, where the higher level "decisions" are a constraint that guides the lower level generation. In meta's byte-level model they made the tokens variable length based on how predictable the bytes were to a smaller model, allocating compute resources based on entropy.