3 ms·
the idea that AI and data compression are isomorphic is really interesting and makes me think of Ashby's law of requisite variety, where the number of possible
by beaconstudios 6y ago
the idea that AI and data compression are isomorphic is really interesting and makes me think of Ashby's law of requisite variety, where the number of possible states (this could also be rendered in information-theoretic terms of entropy) of an agent operating in an environment must be at least as high as the number of states of the environment itself. Thus, to navigate a given environment is to be able to represent that environment internally. Reducing the byte-size of that representation then means that you are representing the environment more tersely, and in my experience (there's probably a theoretical explanation for this but I've only seen it experientially) terser-but-just-as-accurate representations are usually better models.
- seesawtron 6y agoThis paper [0] discusses similar ideas from a neuroscience perspective of feature extraction in deeper layers and argues the possible significance of minimizing the representation decision space in one-shot learning paradigm [0] https://www.biorxiv.org/content/10.1101/2021.03.21.436284v1.full.pdf https://www.biorxiv.org/content/10.1101/2021.03.21.436284v1....
- beaconstudios 6y agohaving thought about it a bit, I think the idea of terser accurate models being "better" is the core of the idea of "beauty" in programming (and probably elsewhere).
- chriswarbo 6y agoSchmidhuber thinks a similar thing https://people.idsia.ch/~juergen/beauty.html https://people.idsia.ch/~juergen/beauty.html (he's also worked with Hutter on AI https://people.idsia.ch/~juergen/unilearn.html https://people.idsia.ch/~juergen/unilearn.html )
- beaconstudios 6y agofascinating, thanks for sharing. The idea that a formal theory of beauty could belong within the memeplex of complexity theory is really interesting. I find that complexity theory itself is full of beautiful concepts, so perhaps it's no surprise.
- npsimons 6y agoI've always felt that the best non-CS programmers were physicists, because they are always looking for the simplest model (eg, the same community that rejected a 27 dimension string theory as being too complex, but don't quote me on this, I'm no physicist).