Like I said, rough analogy. The "lossy compression" here is in the implicit world models these networks learn, not necessarily in the generation space (though w

Like I said, rough analogy. The "lossy compression" here is in the implicit world models these networks learn, not necessarily in the generation space (though we certainly see deviations from reality there too). However, the bigger difference between learning from SEO-generated blogspam and human teaching is the lack of feedback mechanisms to correct errors in generated output that then get taken as ground truth.