3 ms·
The parent commenter is not disagreeing with information theory (and what you're saying is shown in the article anyway). They're making a practical distinction
by fractionalhare 6y ago
The parent commenter is not disagreeing with information theory (and what you're saying is shown in the article anyway).
They're making a practical distinction that you generally don't have access to the actual thing in an empirical format for which compression will achieve true learning. Instead you have access to training data which represents, let's say, a projection of the actual thing in a smaller space with fewer dimensions.
Like trying to learn from images instead of the 3d world. Humans learn to distinguish between objects in a 3-dimensional space using sight and interaction. This learning generalizably transfers to recognition in 2 dimensions. We don't generally equip models with robotic interfaces to train in 3d before benchmarking them on ImageNet.
- logicchains 6y ago>Like trying to learn from images instead of the 3d world. Humans learn to distinguish between objects in a 3-dimensional space using sight and interaction. This learning generalizably transfers to recognition in 2 dimensions. If we use human "comprehension" as a reference point, then the relevant point of comparison should be the understanding a human can develop given the same inputs.
- fractionalhare 6y agoSure. But again, practically speaking, that isn't the reality of how we learn. The commenter wasn't refuting Kolmogorov complexity. They're just saying it's an extremely limited way of viewing the problem. Useful sure, but insufficient.
- wongarsu 6y agoSure, but how do you measure that? How do we figure out how much understanding a human can develop from only ever seeing 2d pictures, without any movement or interaction with a 3d world? Most ML problems are things humans are quite good at and have a lot of context to draw from.
- reader_mode 6y ago> We don't generally equip models with robotic interfaces to train in 3d before benchmarking them on ImageNet. Don't they train models using 3D rendering and simulations ? We have relatively realistic simulations for various scenarios - having a learned model that could make inferences based on those complex simulations sounds like a win.