4 ms·
> Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More
by School-Cotton 2y ago
> Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally:
> P(tₖ|t₁,..,tₖ₋₁)
Not a meaningful definition unless you say what probability distribution it's maximizing it for!
Because for any system that takes any actions (e.g. human beings), there's some probability distribution that that's true for...
- YeGoblynQueenne 2y agoThe probability distribution that is being modelled is the conditional distribution P(tₖ|t₁,..,tₖ₋₁).
- School-Cotton 2y agoThat’s not a description of a probability distribution.
- YeGoblynQueenne 2y agoIt is.
- CyberDildonics 2y agoNope, there are no numbers there, how can you put in a position and get a value (probability) out? Saying "it is" is not evidence or an explanation. Here is the normal distribution probability function. https://en.wikipedia.org/wiki/Normal_distribution https://en.wikipedia.org/wiki/Normal_distribution You can see that there is an actual equation where all the numbers can be filled in except for X, which is the function input.
- deleted 2y ago[deleted]
- School-Cotton 2y agoNo, the conditional distribution is a construction that transforms one probability distribution to another; you still haven’t said what the underlying distribution was.
- deleted 2y ago[deleted]