4 ms·

> I would argue it is (obviously) impossible the way the current implementation of models work. > How could a system which produces a single next word based up

by drdeca 2y ago

> I would argue it is (obviously) impossible the way the current implementation of models work. > How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically? Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. Now, whether a probability distribution over strings can efficiently computed in this form, is another question. But, if we are being so theoretical that we don’t care about the computational cost (as long as it is finite), then the “it is next token prediction” can’t preclude anything which “it produces a probability distribution over strings” doesn’t already preclude. As for the temperature, given any probability distribution over a discrete set, we can modify it by adding a temperature parameter. Just take the log of the probabilities according to the original probability distribution, scale them all by a factor (the inverse of the temperature, I think. Either that or the temperature, but I think it is the inverse of the temperature.), then exponentiate each of these, and then normalize to produce a probability distribution. So, the fact that they work by next token prediction, and have a temperature parameter, cannot imply any theoretical limitation that wouldn’t apply to any other way of expressing a probability distribution over strings, as far as discussing probability distributions in the abstract, over strings, rather than talking about computational processes that implement such probability distributions over strings. But also like, going between P(next token is x | initial string so far is y) and P(the string begins with z) , isn’t that computationally costly? Well, in one direction anyway. Because like, P(next token is x|string so far is y) = P(string begins with yx) / P(string begins with y) . Though, one might object to P(string starts with y) over P(string is y) ?

abernard1 2y ago
> Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. And such a probability distribution would not generally understand concepts, efficient or otherwise. The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide. They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system. The academic exercise here is similar to monads: "yes, any computed function f(x) can be expressed as a sufficiently pre-computed large lookup table." With LLMs we're dealing with approximate lookups due to lossy compression, but that's still what these prior probabilities are: lookup tables. Lookup tables are not smart, do not understand concepts, and they have little to no capacity to generate new results not sufficiently represented in the training set.
- drdeca 2y ago
  My main concern here is the theoretical point, and so I’m not addressing the “this is what current (e.g. transformer based) models do” parts. > The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide. Regardless of whether this is true for existing transformer-based models, this is not true for all computable conditional probability distributions over text. Any computable task can be framed as sampling from some conditional probability distribution. (If the task is deterministic, that just means that the conditional probability distribution to sample from is one which has probability 1 for some string, when conditioned on the thing it is to be conditioned on.) Whether transformer based models are lookup tables or not, not all computable probability distributions over text are. (As, of course, not all computable tasks can be expressed as a simple finite lookup table.) I don’t know exactly what you mean by “generally understand concepts”, though I suppose > They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system. is describing that somewhat. And, in that case, if there is any computational process which counts as having “enough conceptual power to generate new facts and know that they are facts consistent with the model”, then, a computable conditional probability distribution over strings conditioned on their prefixes, and therefore also a computable probability distribution over next tokens given all-tokens-so-far , is also (theoretically) capable of that. And so, it would follow that “it only predicts the next token” doesn’t (in principle/theory) preclude it having such an understanding of concepts, unless no computational process ever can.
  - abernard1 2y ago
    > “it only predicts the next token” doesn’t (in principle/theory) preclude it having such an understanding of concepts, unless no computational process ever can. In my opinion, this is highly reductive and academic. Whether these models are transformers or not, lookup likelihood is not indicative of understanding of concepts in any reasonable way. If the response to a algebraic equation was based upon probability of tokens in a corpus... and not an actual deterministic application of the rules of algebra, would that response know concepts? Would it be intelligent? With math, specifically given the unbounded size of the tokens compared to language, it's clear that token prediction is not a useful methodology. Let's say we're just trying to multiply two integers. Even if a model had Rain Man powers of memorization, and it memorized phone book after phone book of multiplication tables, the probabilistic likelihood model would fail for the very obvious reason that we cannot enumerate (and train on) all the possible outcomes of math and calculate their frequencies. We can however understand and use the concepts of math, which is distinct from their symbolic representation.

by drdeca 2y ago

abernard1 2y ago

> Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. And such a probability distribution would not generally understand concepts, efficient or otherwise. The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide. They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system. The academic exercise here is similar to monads: "yes, any computed function f(x) can be expressed as a sufficiently pre-computed large lookup table." With LLMs we're dealing with approximate lookups due to lossy compression, but that's still what these prior probabilities are: lookup tables. Lookup tables are not smart, do not understand concepts, and they have little to no capacity to generate new results not sufficiently represented in the training set.

drdeca 2y ago
My main concern here is the theoretical point, and so I’m not addressing the “this is what current (e.g. transformer based) models do” parts. > The P(next_token) is based upon the syntactical structure built via the model and some basic semantic distance that LLMs provide. Regardless of whether this is true for existing transformer-based models, this is not true for all computable conditional probability distributions over text. Any computable task can be framed as sampling from some conditional probability distribution. (If the task is deterministic, that just means that the conditional probability distribution to sample from is one which has probability 1 for some string, when conditioned on the thing it is to be conditioned on.) Whether transformer based models are lookup tables or not, not all computable probability distributions over text are. (As, of course, not all computable tasks can be expressed as a simple finite lookup table.) I don’t know exactly what you mean by “generally understand concepts”, though I suppose > They don't have enough conceptual power to reliably generate new facts and know that they are facts consistent with the model. That would be an internal representation system. is describing that somewhat. And, in that case, if there is any computational process which counts as having “enough conceptual power to generate new facts and know that they are facts consistent with the model”, then, a computable conditional probability distribution over strings conditioned on their prefixes, and therefore also a computable probability distribution over next tokens given all-tokens-so-far , is also (theoretically) capable of that. And so, it would follow that “it only predicts the next token” doesn’t (in principle/theory) preclude it having such an understanding of concepts, unless no computational process ever can.
- abernard1 2y ago
  > “it only predicts the next token” doesn’t (in principle/theory) preclude it having such an understanding of concepts, unless no computational process ever can. In my opinion, this is highly reductive and academic. Whether these models are transformers or not, lookup likelihood is not indicative of understanding of concepts in any reasonable way. If the response to a algebraic equation was based upon probability of tokens in a corpus... and not an actual deterministic application of the rules of algebra, would that response know concepts? Would it be intelligent? With math, specifically given the unbounded size of the tokens compared to language, it's clear that token prediction is not a useful methodology. Let's say we're just trying to multiply two integers. Even if a model had Rain Man powers of memorization, and it memorized phone book after phone book of multiplication tables, the probabilistic likelihood model would fail for the very obvious reason that we cannot enumerate (and train on) all the possible outcomes of math and calculate their frequencies. We can however understand and use the concepts of math, which is distinct from their symbolic representation.