3 ms·
Generative AI often uses Markov chains as its theoretical basis and is modeled in the language of conditional probabilities albeit not stuck to a single ancesto
by treprinum 2y ago
Generative AI often uses Markov chains as its theoretical basis and is modeled in the language of conditional probabilities albeit not stuck to a single ancestor but to a chain of ancestors. One can often view LLMs as scaled out Markov chains (more like "Erlang" chains allowing deeper memory) that encode the probability of the next token based on the past of the size of the context window and then doing some magic when picking next tokens.
- Jensson 2y agoEven old Markov chains has a context window of a few words, single word context Markov chains produce horrible results. People who don't think LLMs are Markov chains are just ignorant, not realizing that Markov chain isn't an algorithm, you can compute the probability in any manner and it is still a Markov chain.
- throwthrowuknow 2y agoIt’s not an n-gram, go learn about transformers
- Jensson 2y agoA context window is an n-gram.
- throwthrowuknow 2y agoIt’s not used as an n-gram