4 ms·
> Not that that would necessarily make sense in practice, but in theory Markov chains have enormous expressive power. Right - but this feels like being half-wa
by TeMPOraL 1y ago
> Not that that would necessarily make sense in practice, but in theory Markov chains have enormous expressive power.
Right - but this feels like being half-way between transformers and lookup tables; the latter have enormous expressive power too, as long as you're willing to handle even more state. I'm curious what else could be put on that spectrum, and where. Maybe there's something weaker than transformers, but good enough for practical applications, while being more resource-efficient?
Related thought: understanding and compression seem to be fundamentally the same thing.
- joquarky 1y ago> Maybe there's something weaker than transformers, but good enough for practical applications, while being more resource-efficient? I wish! Gboard is terrible at predicting the correct form of a word. It generally knows the best word, but it doesn't get from context that (e.g.) a gerund form of the verb is the most appropriate next
- joe_the_user 1y agoBut no, as another poster comments, the state space blow up is much worse than transformers for much less [1]. Transformers have more kinds of statefulness while Markov chains seem simpler cause they just multiply the number of states - but that state-multiplication makes them completely impractical. They aren't flexible on a per-state basis, just the opposite. Basically, for most forms of complexity, Markov Chains are a strictly bad model. Perhaps if there was a way to "virtualize" the multiplication of states, you could do something reasonable. Idle speculation though. [1] https://news.ycombinator.com/item?id=45354488 https://news.ycombinator.com/item?id=45354488
- taneq 1y ago> Related thought: understanding and compression seem to be fundamentally the same thing. Both are fundamentally prediction. :)
- impulsivepuppet 1y agoRelated and likely inspired by the related thought: https://www.mattmahoney.net/dc/ https://www.mattmahoney.net/dc/
- refset 1y agoSpecifically 1.4 "Compression is an Artificial Intelligence Problem" https://www.mattmahoney.net/dc/dce.html#Section_14 https://www.mattmahoney.net/dc/dce.html#Section_14
- jgalt212 1y agoMost of human life is a lookup table derived action.
- econ 1y agoI notice that people with poor lookup table hardware need to think a lot more. The public part of the table is mostly populated with their thoughts. To quote a friend of mine: If I can't just trust the status quo I would have to question everything????
- TeMPOraL 1y agoThey're right, of course. You have to pick your battles. There's an art to that - and rules of that art are part of the lookup table too! #ItsSoMetaEvenThisAcronym