3 ms·
I think that part of the reasons conclusions about emergence are tenable is due to the opaque nature of transformer architectures. For example if it was possib
by usgroup 3y ago
I think that part of the reasons conclusions about emergence are tenable is due to the opaque nature of transformer architectures.
For example if it was possible to train a Hidden Markov Model with billions of hidden states on a trillion tokens, you could more literally look and see what was going on.
Other than not being able to scale HMMs to this kind of scale, is there any good reason to believe they would not perform equally well but without the magic?
- kolinko 3y agoYou're using billions and trillions loosely here. The hidden state in HMM would be (num_tokens ^ context size), so something like 60000^2000.
- usgroup 3y agoI'm not sure that calculation is correct, but say it is, perhaps a Variable Length Markov Chain then.
- kolinko 3y agoVariable length markov chains would merge some states, sure, but it will still be a similar order of magnitudes. Anything longer than 4 tokens/words of context and you bump into 30k ^ 4-10 -> cross a billion/trillion state boundary and you lose any chance of using markov chains. Also - but here I may be wrong - there is no way to "train" markov chains to do generalisations - that is, if a given sentence didn't appear on the internet, it won't be available as a state for the chain. In this aspect they are more similar to a database than anything else.