4 ms·

>Compression is a necessary but not sufficient condition for 'comprehension'. Actually, algorithmic information theory shows that maximum compression necessari

>Compression is a necessary but not sufficient condition for 'comprehension'. Actually, algorithmic information theory shows that maximum compression necessarily entails maximum comprehension, because the only way to maximally compress something is to exactly understand the process producing it.

mjburgess 6y ago
Relative to a device to perform the understanding. A string X maximally compresses datsets Y iff X is a 'comprehending' of Y. 'OK'... but what produces and evaluates X? ie., comprehension. This is the problem with defining these terms mathematically; you state the problem in basically useless ways. Yes, you can specify what eqn produces the mass of the higgs boson. Thats basically no guide to building the LHC. The production of such understanding is not abstract. Comprehension isnt a relation between two binary strings; it is an action taken in an environment with a goal.
- logicchains 6y ago
  >Relative to a device to perform the understanding. It uses Kolmogorov complexity, which is defined as the length of a shortest computer program in a predetermined programming language that produces the object as output. Note that this measure is relative to a programming language, not a program, and the exact choice of language doesn't matter too much. Compression means creating a smaller program that produces the same output, and to produce the same output with less code necessarily requires more understanding. As a concrete example, imagine the output is [1,2,fizz,4,buzz,fizz,7,8,fizz,buzz,11,fizz,13,14,fizzbuzz..1000). The longest program to output this would just hard-code it in the source code (much as a very inexperienced programmer might solve the problem, or a large neural net). Someone with a better understanding would write a program using iteration and the modulus operator, which would be shorter.
gbrown 6y ago
But what if we drop the word comprehension, and we just go with “functional approximation X -> Y, computed from a finite dataset, which minimizes a predictive risk “? It’s unclear why compression is necessary there, except as a practical benefit.
funkisjazz 6y ago
Yeap, it gets into a philosophical debate of what comprehension. One could argue it's not about compression in bits but compression to primitives that makes sense to the human mind. But then the definition becomes to fuzzy because it naturally invites the question "Who's mind?"
fractionalhare 6y ago
The parent commenter is not disagreeing with information theory (and what you're saying is shown in the article anyway). They're making a practical distinction that you generally don't have access to the actual thing in an empirical format for which compression will achieve true learning. Instead you have access to training data which represents, let's say, a projection of the actual thing in a smaller space with fewer dimensions. Like trying to learn from images instead of the 3d world. Humans learn to distinguish between objects in a 3-dimensional space using sight and interaction. This learning generalizably transfers to recognition in 2 dimensions. We don't generally equip models with robotic interfaces to train in 3d before benchmarking them on ImageNet.
- logicchains 6y ago
  >Like trying to learn from images instead of the 3d world. Humans learn to distinguish between objects in a 3-dimensional space using sight and interaction. This learning generalizably transfers to recognition in 2 dimensions. If we use human "comprehension" as a reference point, then the relevant point of comparison should be the understanding a human can develop given the same inputs.
  - fractionalhare 6y ago
    Sure. But again, practically speaking, that isn't the reality of how we learn. The commenter wasn't refuting Kolmogorov complexity. They're just saying it's an extremely limited way of viewing the problem. Useful sure, but insufficient.
  - wongarsu 6y ago
    Sure, but how do you measure that? How do we figure out how much understanding a human can develop from only ever seeing 2d pictures, without any movement or interaction with a 3d world? Most ML problems are things humans are quite good at and have a lot of context to draw from.
- reader_mode 6y ago
  > We don't generally equip models with robotic interfaces to train in 3d before benchmarking them on ImageNet. Don't they train models using 3D rendering and simulations ? We have relatively realistic simulations for various scenarios - having a learned model that could make inferences based on those complex simulations sounds like a win.
visarga 6y ago
Compression is applied to data collected in the past while comprehension or intelligence also require adapting well to the future. Thus a pure compressor would optimize for a static distribution while an intelligent agent will probably be sub-optimal because it also needs to learn and adapt to the future.

mjburgess 6y ago

Relative to a device to perform the understanding. A string X maximally compresses datsets Y iff X is a 'comprehending' of Y. 'OK'... but what produces and evaluates X? ie., comprehension. This is the problem with defining these terms mathematically; you state the problem in basically useless ways. Yes, you can specify what eqn produces the mass of the higgs boson. Thats basically no guide to building the LHC. The production of such understanding is not abstract. Comprehension isnt a relation between two binary strings; it is an action taken in an environment with a goal.

logicchains 6y ago
>Relative to a device to perform the understanding. It uses Kolmogorov complexity, which is defined as the length of a shortest computer program in a predetermined programming language that produces the object as output. Note that this measure is relative to a programming language, not a program, and the exact choice of language doesn't matter too much. Compression means creating a smaller program that produces the same output, and to produce the same output with less code necessarily requires more understanding. As a concrete example, imagine the output is [1,2,fizz,4,buzz,fizz,7,8,fizz,buzz,11,fizz,13,14,fizzbuzz..1000). The longest program to output this would just hard-code it in the source code (much as a very inexperienced programmer might solve the problem, or a large neural net). Someone with a better understanding would write a program using iteration and the modulus operator, which would be shorter.

gbrown 6y ago

But what if we drop the word comprehension, and we just go with “functional approximation X -> Y, computed from a finite dataset, which minimizes a predictive risk “? It’s unclear why compression is necessary there, except as a practical benefit.

funkisjazz 6y ago

Yeap, it gets into a philosophical debate of what comprehension. One could argue it's not about compression in bits but compression to primitives that makes sense to the human mind. But then the definition becomes to fuzzy because it naturally invites the question "Who's mind?"

fractionalhare 6y ago

The parent commenter is not disagreeing with information theory (and what you're saying is shown in the article anyway). They're making a practical distinction that you generally don't have access to the actual thing in an empirical format for which compression will achieve true learning. Instead you have access to training data which represents, let's say, a projection of the actual thing in a smaller space with fewer dimensions. Like trying to learn from images instead of the 3d world. Humans learn to distinguish between objects in a 3-dimensional space using sight and interaction. This learning generalizably transfers to recognition in 2 dimensions. We don't generally equip models with robotic interfaces to train in 3d before benchmarking them on ImageNet.

logicchains 6y ago
>Like trying to learn from images instead of the 3d world. Humans learn to distinguish between objects in a 3-dimensional space using sight and interaction. This learning generalizably transfers to recognition in 2 dimensions. If we use human "comprehension" as a reference point, then the relevant point of comparison should be the understanding a human can develop given the same inputs.
- fractionalhare 6y ago
  Sure. But again, practically speaking, that isn't the reality of how we learn. The commenter wasn't refuting Kolmogorov complexity. They're just saying it's an extremely limited way of viewing the problem. Useful sure, but insufficient.
- wongarsu 6y ago
  Sure, but how do you measure that? How do we figure out how much understanding a human can develop from only ever seeing 2d pictures, without any movement or interaction with a 3d world? Most ML problems are things humans are quite good at and have a lot of context to draw from.
reader_mode 6y ago
> We don't generally equip models with robotic interfaces to train in 3d before benchmarking them on ImageNet. Don't they train models using 3D rendering and simulations ? We have relatively realistic simulations for various scenarios - having a learned model that could make inferences based on those complex simulations sounds like a win.

visarga 6y ago

Compression is applied to data collected in the past while comprehension or intelligence also require adapting well to the future. Thus a pure compressor would optimize for a static distribution while an intelligent agent will probably be sub-optimal because it also needs to learn and adapt to the future.