I think it's more that it fills in gaps without regard to meaning. If it can't find an argument/explanation/etc. that contains the nouns and verbs (i.e. the sub

I think it's more that it fills in gaps without regard to meaning. If it can't find an argument/explanation/etc. that contains the nouns and verbs (i.e. the substance) of what it's being prompted with, it will happily manufacture that content out of completely unrelated sentences that are similar in structure. It isn't that it's imitating liars any more than Stable Diffusion is imitating photographs of people with faces growing out of their crotches or eight arms. Very convincingly 3D horrific imagery in Stable Diffusion is the equivalent of confident lying in ChatGPT.