4 ms·

It's not an unreasonable definition, if you are aware of Kolmogorov complexity and Solomonoff induction. Intelligence is intimately connected to the ability to

It's not an unreasonable definition, if you are aware of Kolmogorov complexity and Solomonoff induction. Intelligence is intimately connected to the ability to predict and model things, and it turns out that data compression is _also_ connected to the ability of predict and model things.

lIIllIIllIIllII 3y ago
It makes sense intuitively right? If I tell you that this week it's raining on Monday, raining on Tuesday, raining on Wednesday, etc.. Compressing that information into "it's raining every day this week" requires creating an abstraction around it. Finding a pattern. Producing more order from something more chaotic.
- vjerancrnjak 3y ago
  Interestingly, compression increases entropy. So the first description is more "ordered", or less random.
- DaiPlusPlus 3y ago
  > Compressing that information into "it's raining every day this week" requires creating an abstraction around it. Finding a pattern. Producing more order from something more chaotic. Is there any way to determine if that process of abstraction-finding and pattern-finding will always/actually/sometimes/never result in a more compressed output than simpler approaches though?
  - GolDDranks 3y ago
    A hunch of mine is that's undecidable in the general case. In simple cases like this, there's likely to be a way to prove that yes, this is tho most compressed thing out there.
DaiPlusPlus 3y ago
> It's not an unreasonable definition, if you are aware of Kolmogorov complexity and Solomonoff induction I was already familiar with Kolmogorov Complexity, but not _Solomonoff's theory of inductive inference_ - I skimmed the Wikipedia article just now and I think/hope I get the general gist of it - so thank you for that. > Intelligence is intimately connected to the ability to predict and model things I agree with that statement - but I am unsure how this relates to other aspects of intelligence - or even other notions of intelligence entirely. If we're redefining exactly what the word "Intelligence" means aren't we just moving the goalposts? ...so I'm unsure if the word "Intelligence" in your post can/should be read as a layperson would understand it, or if it's a specific, well-defined term in this field? (Like how "complexity" in algorithms does not refer to how convoluted and unmaintainable a program is, which is what a layperson would think of when an SWE person says "this program has high computational complexity" to them). > and it turns out that data compression is _also_ connected to the ability of predict and model things. This is where I have difficulty following your argument: As a preface: my academic understanding of data compression only covers things like algorithms in the LZ family, Entropy coding, and (lossy and lossless) domain-specific compression schemes (like JPEG/DCT, MP3/FLAC, etc). I am aware of very interesting results coming from present-day AI approaches, like using LLMs fed only on compressed data - but these are completely outside my scope of understanding (and LLMs are still very spooky to me). Does it matter if a scheme is lossless or lossy? In a lossless system, surely a data compression system needs to be deterministic and mechanical? If so, what room is there for "intelligence" in a rigid, statically-defined, system? Take entropy coding for example - or even simpler probability-based compression schemes like Morse Code (e.g. "E" has a high probability, so it has a very short symbol length). I just can't see the connection from things like entropy-encoding to "modelling": supposing I have a large file (CSV?) of meterological data for a specific region over time - if it's plaintext then I expect I can compress it better using gzip than by using a system that can (somehow!) identify some structure to the weather sensor data (the "model", right?) and then use that as the basis for a better domain-specific compression - but doing this means having to add additional metadata to the source data to describe the structure that the system identified, and then hope that this approach is better than a comparatively "dumb" approach like DEFLATE - and even then, assuming that employing that "model" really does result in smaller compressed output, how is that an example of the system having a general "intelligence"?

lIIllIIllIIllII 3y ago

It makes sense intuitively right? If I tell you that this week it's raining on Monday, raining on Tuesday, raining on Wednesday, etc.. Compressing that information into "it's raining every day this week" requires creating an abstraction around it. Finding a pattern. Producing more order from something more chaotic.

vjerancrnjak 3y ago
Interestingly, compression increases entropy. So the first description is more "ordered", or less random.
DaiPlusPlus 3y ago
> Compressing that information into "it's raining every day this week" requires creating an abstraction around it. Finding a pattern. Producing more order from something more chaotic. Is there any way to determine if that process of abstraction-finding and pattern-finding will always/actually/sometimes/never result in a more compressed output than simpler approaches though?
- GolDDranks 3y ago
  A hunch of mine is that's undecidable in the general case. In simple cases like this, there's likely to be a way to prove that yes, this is tho most compressed thing out there.

DaiPlusPlus 3y ago

> It's not an unreasonable definition, if you are aware of Kolmogorov complexity and Solomonoff induction I was already familiar with Kolmogorov Complexity, but not _Solomonoff's theory of inductive inference_ - I skimmed the Wikipedia article just now and I think/hope I get the general gist of it - so thank you for that. > Intelligence is intimately connected to the ability to predict and model things I agree with that statement - but I am unsure how this relates to other aspects of intelligence - or even other notions of intelligence entirely. If we're redefining exactly what the word "Intelligence" means aren't we just moving the goalposts? ...so I'm unsure if the word "Intelligence" in your post can/should be read as a layperson would understand it, or if it's a specific, well-defined term in this field? (Like how "complexity" in algorithms does not refer to how convoluted and unmaintainable a program is, which is what a layperson would think of when an SWE person says "this program has high computational complexity" to them). > and it turns out that data compression is _also_ connected to the ability of predict and model things. This is where I have difficulty following your argument: As a preface: my academic understanding of data compression only covers things like algorithms in the LZ family, Entropy coding, and (lossy and lossless) domain-specific compression schemes (like JPEG/DCT, MP3/FLAC, etc). I am aware of very interesting results coming from present-day AI approaches, like using LLMs fed only on compressed data - but these are completely outside my scope of understanding (and LLMs are still very spooky to me). Does it matter if a scheme is lossless or lossy? In a lossless system, surely a data compression system needs to be deterministic and mechanical? If so, what room is there for "intelligence" in a rigid, statically-defined, system? Take entropy coding for example - or even simpler probability-based compression schemes like Morse Code (e.g. "E" has a high probability, so it has a very short symbol length). I just can't see the connection from things like entropy-encoding to "modelling": supposing I have a large file (CSV?) of meterological data for a specific region over time - if it's plaintext then I expect I can compress it better using gzip than by using a system that can (somehow!) identify some structure to the weather sensor data (the "model", right?) and then use that as the basis for a better domain-specific compression - but doing this means having to add additional metadata to the source data to describe the structure that the system identified, and then hope that this approach is better than a comparatively "dumb" approach like DEFLATE - and even then, assuming that employing that "model" really does result in smaller compressed output, how is that an example of the system having a general "intelligence"?