3 ms·
+1 But there's a fundamental difference between Markov chains and transformers that should be noted. Markov chains only learn how likely it is for one token to
by vhantz 2y ago
+1
But there's a fundamental difference between Markov chains and transformers that should be noted. Markov chains only learn how likely it is for one token to follow another. Transformers learn how likely it is for a set of token to be seen together. Transformers add a wider context to msrkov chain. That quantitative change leads to a qualitative improvement: transformers generate text that is semantically plausible.
- wnoise 2y agoYes, but k-token lookup was already a thing with markov chains. Transformers are indeed better, but just because they model language distributions better than mostly-empty arrays of (token-count)^(context).