3 ms·
Prediction and compression as a way to encode data cheaply is a reasonable idea. It's really cheap to encode something that the predictor expected next. This is
by Animats 6y ago
Prediction and compression as a way to encode data cheaply is a reasonable idea. It's really cheap to encode something that the predictor expected next. This is essentially what GPT-2 and GPT-3 are doing.
The author seems to be missing some implications of that. One is that a predictor/compressor has a bias towards what it has compressed before. We know that humans tend to over-generalize, and that that's a survival trait. Something like this might be the mechanism behind that.
- murbard2 6y agoOver-generalizing means that you will compress worse, as does under-generalizing. Can't use the mere fact that "compression is going on", you have to look into utility, costs, etc.
- jacobush 6y agoHow? "All X people behave in Y ways" is an over-generalization and seems to compress very well.