4 ms·
Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More
by YeGoblynQueenne 2y ago
Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally:
P(tₖ|t₁,..,tₖ₋₁)
For example, the original OpenAI GPT tranformer model was trained to optimise the objective:
∑ log P(tₖ|tₖ₋ₙ,..,tₖ₋₁;Θ)
Where n the length of a sliding "context" window and Θ the parameters of a neural net model (here, a Transformer architecture) used to optimise the objectivbe.
See page 3 of the OpenAI paper:
https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised...
Also see, for alternative formulations:
Hidden Markov Models:
https://en.wikipedia.org/wiki/Hidden_Markov_model https://en.wikipedia.org/wiki/Hidden_Markov_model
N-Gram word models:
https://en.wikipedia.org/wiki/Word_n-gram_language_model https://en.wikipedia.org/wiki/Word_n-gram_language_model
- School-Cotton 2y ago> Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally: > P(tₖ|t₁,..,tₖ₋₁) Not a meaningful definition unless you say what probability distribution it's maximizing it for! Because for any system that takes any actions (e.g. human beings), there's some probability distribution that that's true for...
- YeGoblynQueenne 2y agoThe probability distribution that is being modelled is the conditional distribution P(tₖ|t₁,..,tₖ₋₁).
- School-Cotton 2y agoThat’s not a description of a probability distribution.
- YeGoblynQueenne 2y agoIt is.
- CyberDildonics 2y agoNope, there are no numbers there, how can you put in a position and get a value (probability) out? Saying "it is" is not evidence or an explanation. Here is the normal distribution probability function. https://en.wikipedia.org/wiki/Normal_distribution https://en.wikipedia.org/wiki/Normal_distribution You can see that there is an actual equation where all the numbers can be filled in except for X, which is the function input.
- deleted 2y ago[deleted]
- School-Cotton 2y agoNo, the conditional distribution is a construction that transforms one probability distribution to another; you still haven’t said what the underlying distribution was.
- deleted 2y ago[deleted]