4 ms·
i'm as hype for ai as anyone but mathematically it is a markov chain; a trained llm model is a function that gives a probability distribution over token n+1 con
by ftxbro 3y ago
i'm as hype for ai as anyone but mathematically it is a markov chain; a trained llm model is a function that gives a probability distribution over token n+1 conditional on the preceding n tokens.
- GaggiX 3y agoWell, on Wikipedia you can read "Random process independent of past history" and "a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event." honestly I would not consider the entire context window to be part of a single event, since the model generates tokens one at a time, separately.
- ftxbro 3y ago> honestly I would not consider the entire context window to be part of a single event, since the model generates tokens one at a time, separately. OK but traditional small language model Markov chains have context windows too, and they also generate tokens one at a time, separately. Maybe you are arguing that there is a qualitative difference between an eight token context window and an eight thousand token context window. I guess that would be fair enough.