Simple models will very quickly produce content that doesn't fit. Just stringing a bit of grammar and a dictionary together will produce locally correct sentenc

Simple models will very quickly produce content that doesn't fit. Just stringing a bit of grammar and a dictionary together will produce locally correct sentences, but it will completely fail at generate larger texts with context. GPT in contrast knows what it was talking about five sentences ago and can refer to it in newly formed sentences.