6 ms·
You're using billions and trillions loosely here. The hidden state in HMM would be (num_tokens ^ context size), so something like 60000^2000.
by kolinko 4y ago
You're using billions and trillions loosely here.
The hidden state in HMM would be (num_tokens ^ context size), so something like 60000^2000.
- usgroup 4y agoI'm not sure that calculation is correct, but say it is, perhaps a Variable Length Markov Chain then.
- kolinko 4y agoVariable length markov chains would merge some states, sure, but it will still be a similar order of magnitudes. Anything longer than 4 tokens/words of context and you bump into 30k ^ 4-10 -> cross a billion/trillion state boundary and you lose any chance of using markov chains. Also - but here I may be wrong - there is no way to "train" markov chains to do generalisations - that is, if a given sentence didn't appear on the internet, it won't be available as a state for the chain. In this aspect they are more similar to a database than anything else.