6 ms·

"as a path to AGI" anyone else disagree with this? Creating a highly efficient compressor seems very much not AGI

by asperous 7y ago

dooglius 7y ago
AGI would probably be sufficient, but not necessary.
- jobigoud 7y ago
  Do you mean necessary but not sufficient? I run an non-artificial GI in my brain and can't solve this challenge manually. So AGI doesn't seem sufficient.
  - imtringued 7y ago
    That's a complete misunderstanding of the problem. The compressor should be compared to the machine that runs the AGI, not the AGI itself. Your brain is compressing data all the time but it is happening unconsciously. There are other flaws with the idea of compression being a path to AGI but this is not one of them.
  - dTal 7y ago
    I suspect you're selling yourself a little short. If you studied the text and learned everything in it, you'd be able to generate a new text from scratch, from your internal representation of the information. It wouldn't be word-for-word identical, but that's actually evidence that there is a compressed representation in your head. If you got a good sense of Wikipedia's style out of it, your reconstruction would be even better. Now imagine that we let you take notes - little reminders of unusual turns of phrase, hard to remember dates, small outlines etc. You could probably do quite well indeed, perhaps even well enough that your output could be diffed with the original and result in a comparatively small file. I think it would be fair to count that as a win for you, since the computer program is deterministic and can store the diff along with its output. True, we can't measure the size of the data in your head, but compression is definitely occurring because you would certainly fail miserably if you tried to reconstruct 100 megabytes of random binary data in a similar way. Still, to achieve that last little bit of the way to truly lossless reconstruction without the diff-storing trick will present you with a challenge, I agree. Lossless reproduction inherently requires encoding a lot of noise, to which no pattern can be matched. It's still a valid test because, although there is a hard floor on how much the data can be compressed, a better encoder will get closer and closer to that floor. And there's room for interesting optimizations all the way to the bottom - for example, with the help of a language parser and a thesaurus one could probably encode the difference between your lossy output and the original text rather better than 'diff'...
asynchrony 7y ago
Being able to discern large scale themes and patterns in data, then using those findings distill the original data input to minimally redundant messages, is similar to the goal of an AGI that can comprehend any input data.
herendin2 7y ago
A compressor needs to detect patterns in data. Best if it works with general data. General intelligence is excellent at these tasks.
gwern 7y ago
All something like GPT-2 is, is a text compressor, a model for an arithmetic encoder specifically. It predicts the probability of the next token conditional on a history. You can directly measure the 'bits per character'/'perplexity' as a measure of (compression) performance, and that is typically how these language models are evaluated: eg http://nlpprogress.com/english/language_modeling.html http://nlpprogress.com/english/language_modeling.html https://paperswithcode.com/task/language-modelling https://paperswithcode.com/task/language-modelling The Hutter dataset is included, incidentally, and the top is a Transformer-XL (277M parameters) at 0.94 BPC. Does all of that seem 'very much not AGI'?
- doctorpangloss 7y ago
  Yes, it does not at all seem like AGI! You’ve explained succinctly why the GPT2 result is maybe not on the road to AGI.
  - gwern 7y ago
    Language models trained in a predictive fashion achieve ever closer to human compression rates (humans would only get ~0.7-0.8 BPC on Hutter), and are responsible for not just the creepiest & most realistic text generation to date but also setting SOTAs across all language-related AI benchmarks like SuperGLUE, designed to test language understanding and real-world reasoning, or chatbots. If that has nothing whatsoever to do with progress towards AGI, we'll just have to agree to disagree.
  - dwohnitmok 7y ago
    I found Scott Alexander's take on this fairly convincing. Essentially if you master language you get broadly generalizable insights across domains. https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g... If you think of intelligence as distilling the essence of the universe around you and building on top of that, at least the former starts to sound like really really really sophisticated compression.
- tux3 7y ago
  What's unfortunate is the small size of enwiki8. A decompressor with a high upfront cost (say a pretrained model) and otherwise higher compression ratio would be more interesting than tweaking yet another PAQ8 variant, but I'm not sure this contest's rules make that very viable.
jpalomaki 7y ago
From AGI perspective it should be more relevant to come up with the ideas/concepts described in those Wikipedia articles - not the exact words used by humans to describe them.
- salty_biscuits 7y ago
  Wouldn't that just be an extension to perceptually lossy compression?
throwawaywego 7y ago
Imagine a highly efficient compressor watching Toy Story in 4k definition. If it was really intelligent, then it could abstract the 3D-models, motion, physics, lightning, and textures, and store the movie (or something imperceivably like it) in a highly compressed representation. Or even write down just the script and direct a movie so similar that it can't be distinguished from the original.

"as a path to AGI" anyone else disagree with this? Creating a highly efficient compressor seems very much not AGI

dooglius 7y ago

AGI would probably be sufficient, but not necessary.

jobigoud 7y ago
Do you mean necessary but not sufficient? I run an non-artificial GI in my brain and can't solve this challenge manually. So AGI doesn't seem sufficient.
- imtringued 7y ago
  That's a complete misunderstanding of the problem. The compressor should be compared to the machine that runs the AGI, not the AGI itself. Your brain is compressing data all the time but it is happening unconsciously. There are other flaws with the idea of compression being a path to AGI but this is not one of them.
- dTal 7y ago
  I suspect you're selling yourself a little short. If you studied the text and learned everything in it, you'd be able to generate a new text from scratch, from your internal representation of the information. It wouldn't be word-for-word identical, but that's actually evidence that there is a compressed representation in your head. If you got a good sense of Wikipedia's style out of it, your reconstruction would be even better. Now imagine that we let you take notes - little reminders of unusual turns of phrase, hard to remember dates, small outlines etc. You could probably do quite well indeed, perhaps even well enough that your output could be diffed with the original and result in a comparatively small file. I think it would be fair to count that as a win for you, since the computer program is deterministic and can store the diff along with its output. True, we can't measure the size of the data in your head, but compression is definitely occurring because you would certainly fail miserably if you tried to reconstruct 100 megabytes of random binary data in a similar way. Still, to achieve that last little bit of the way to truly lossless reconstruction without the diff-storing trick will present you with a challenge, I agree. Lossless reproduction inherently requires encoding a lot of noise, to which no pattern can be matched. It's still a valid test because, although there is a hard floor on how much the data can be compressed, a better encoder will get closer and closer to that floor. And there's room for interesting optimizations all the way to the bottom - for example, with the help of a language parser and a thesaurus one could probably encode the difference between your lossy output and the original text rather better than 'diff'...

asynchrony 7y ago

Being able to discern large scale themes and patterns in data, then using those findings distill the original data input to minimally redundant messages, is similar to the goal of an AGI that can comprehend any input data.

herendin2 7y ago

A compressor needs to detect patterns in data. Best if it works with general data. General intelligence is excellent at these tasks.

gwern 7y ago

All something like GPT-2 is, is a text compressor, a model for an arithmetic encoder specifically. It predicts the probability of the next token conditional on a history. You can directly measure the 'bits per character'/'perplexity' as a measure of (compression) performance, and that is typically how these language models are evaluated: eg http://nlpprogress.com/english/language_modeling.html http://nlpprogress.com/english/language_modeling.html https://paperswithcode.com/task/language-modelling https://paperswithcode.com/task/language-modelling The Hutter dataset is included, incidentally, and the top is a Transformer-XL (277M parameters) at 0.94 BPC. Does all of that seem 'very much not AGI'?

doctorpangloss 7y ago
Yes, it does not at all seem like AGI! You’ve explained succinctly why the GPT2 result is maybe not on the road to AGI.
- gwern 7y ago
  Language models trained in a predictive fashion achieve ever closer to human compression rates (humans would only get ~0.7-0.8 BPC on Hutter), and are responsible for not just the creepiest & most realistic text generation to date but also setting SOTAs across all language-related AI benchmarks like SuperGLUE, designed to test language understanding and real-world reasoning, or chatbots. If that has nothing whatsoever to do with progress towards AGI, we'll just have to agree to disagree.
- dwohnitmok 7y ago
  I found Scott Alexander's take on this fairly convincing. Essentially if you master language you get broadly generalizable insights across domains. https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/ https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g... If you think of intelligence as distilling the essence of the universe around you and building on top of that, at least the former starts to sound like really really really sophisticated compression.
tux3 7y ago
What's unfortunate is the small size of enwiki8. A decompressor with a high upfront cost (say a pretrained model) and otherwise higher compression ratio would be more interesting than tweaking yet another PAQ8 variant, but I'm not sure this contest's rules make that very viable.

jpalomaki 7y ago

From AGI perspective it should be more relevant to come up with the ideas/concepts described in those Wikipedia articles - not the exact words used by humans to describe them.

salty_biscuits 7y ago
Wouldn't that just be an extension to perceptually lossy compression?

throwawaywego 7y ago

Imagine a highly efficient compressor watching Toy Story in 4k definition. If it was really intelligent, then it could abstract the 3D-models, motion, physics, lightning, and textures, and store the movie (or something imperceivably like it) in a highly compressed representation. Or even write down just the script and direct a movie so similar that it can't be distinguished from the original.