3 ms·

Please rigorously define what the term "next token predictor" means.

dmitrygr 2y ago
A device or system that, given a set of tokens, predicts the next token purely based on the previous tokens and its own initial state. It is incapable of coming up with a new token type or changing its own workings
- Grimblewald 2y ago
  On top of your definition being a bit circular, it is also like saying humans can't come up with new things, because we can't come up with new characters when talking online. We're limited by the character-set we're given but that doesn't mean we can't conceive of new characters and more importantly, can't represent new characters using existing character-sets using some encoding scheme to represent that new character. To that end, lets check with chatGPT if it can come up with a new and useful concept for a token, using existing characters to encode for it. https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df98931 https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df989... And there you have it, novel constructed from available ones, with a genuine pragmatic and LLM specific use case.
  - suspended_state 2y ago
    [dead]
- Lerc 2y ago
  So if they can come up with their own internal representations is that sufficient? https://arxiv.org/abs/2410.05864 https://arxiv.org/abs/2410.05864 Or are you wanting them to use keys that are not on their keyboard?
YeGoblynQueenne 2y ago
Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally: P(tₖ|t₁,..,tₖ₋₁) For example, the original OpenAI GPT tranformer model was trained to optimise the objective: ∑ log P(tₖ|tₖ₋ₙ,..,tₖ₋₁;Θ) Where n the length of a sliding "context" window and Θ the parameters of a neural net model (here, a Transformer architecture) used to optimise the objectivbe. See page 3 of the OpenAI paper: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised... Also see, for alternative formulations: Hidden Markov Models: https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model N-Gram word models: https://en.wikipedia.org/wiki/Word_n-gram_language_model https://en.wikipedia.org/wiki/Word_n-gram_language_model
- School-Cotton 2y ago
  > Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally: > P(tₖ|t₁,..,tₖ₋₁) Not a meaningful definition unless you say what probability distribution it's maximizing it for! Because for any system that takes any actions (e.g. human beings), there's some probability distribution that that's true for...
  - YeGoblynQueenne 2y ago
    The probability distribution that is being modelled is the conditional distribution P(tₖ|t₁,..,tₖ₋₁).
    - School-Cotton 2y ago
      That’s not a description of a probability distribution.
      - YeGoblynQueenne 2y ago
        It is.
        CyberDildonics 2y ago
        Nope, there are no numbers there, how can you put in a position and get a value (probability) out? Saying "it is" is not evidence or an explanation. Here is the normal distribution probability function. https://en.wikipedia.org/wiki/Normal_distribution https://en.wikipedia.org/wiki/Normal_distribution You can see that there is an actual equation where all the numbers can be filled in except for X, which is the function input.
        deleted 2y ago
        [deleted]
        2y ago
suspended_state 2y ago
[dead]

A device or system that, given a set of tokens, predicts the next token purely based on the previous tokens and its own initial state. It is incapable of coming up with a new token type or changing its own workings

Grimblewald 2y ago
On top of your definition being a bit circular, it is also like saying humans can't come up with new things, because we can't come up with new characters when talking online. We're limited by the character-set we're given but that doesn't mean we can't conceive of new characters and more importantly, can't represent new characters using existing character-sets using some encoding scheme to represent that new character. To that end, lets check with chatGPT if it can come up with a new and useful concept for a token, using existing characters to encode for it. https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df98931 https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df989... And there you have it, novel constructed from available ones, with a genuine pragmatic and LLM specific use case.
- suspended_state 2y ago
  [dead]
Lerc 2y ago
So if they can come up with their own internal representations is that sufficient? https://arxiv.org/abs/2410.05864 https://arxiv.org/abs/2410.05864 Or are you wanting them to use keys that are not on their keyboard?

YeGoblynQueenne 2y ago

Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally: P(tₖ|t₁,..,tₖ₋₁) For example, the original OpenAI GPT tranformer model was trained to optimise the objective: ∑ log P(tₖ|tₖ₋ₙ,..,tₖ₋₁;Θ) Where n the length of a sliding "context" window and Θ the parameters of a neural net model (here, a Transformer architecture) used to optimise the objectivbe. See page 3 of the OpenAI paper: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised... Also see, for alternative formulations: Hidden Markov Models: https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model N-Gram word models: https://en.wikipedia.org/wiki/Word_n-gram_language_model https://en.wikipedia.org/wiki/Word_n-gram_language_model

School-Cotton 2y ago
> Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally: > P(tₖ|t₁,..,tₖ₋₁) Not a meaningful definition unless you say what probability distribution it's maximizing it for! Because for any system that takes any actions (e.g. human beings), there's some probability distribution that that's true for...
- YeGoblynQueenne 2y ago
  The probability distribution that is being modelled is the conditional distribution P(tₖ|t₁,..,tₖ₋₁).
  - School-Cotton 2y ago
    That’s not a description of a probability distribution.
    - YeGoblynQueenne 2y ago
      It is.
      - CyberDildonics 2y ago
        Nope, there are no numbers there, how can you put in a position and get a value (probability) out? Saying "it is" is not evidence or an explanation. Here is the normal distribution probability function. https://en.wikipedia.org/wiki/Normal_distribution https://en.wikipedia.org/wiki/Normal_distribution You can see that there is an actual equation where all the numbers can be filled in except for X, which is the function input.
        deleted 2y ago
        [deleted]
      - 2y ago

suspended_state 2y ago

[dead]