3 ms·
Great points. I suspect a non-degenerative model is orders of magnitude harder, in that good long prose requires much deeper complexity. For example, I was jus
by wpietri 6y ago
Great points. I suspect a non-degenerative model is orders of magnitude harder, in that good long prose requires much deeper complexity.
For example, I was just reading this excellent long piece from Rebecca Traister: https://nymag.com/intelligencer/article/andrew-cuomo-misconduct-allegations.html https://nymag.com/intelligencer/article/andrew-cuomo-miscond...
Even if a language model had all the raw facts needed to write a long piece, there's a ton of structure here. She develops ideas, builds themes, assembles a thesis. She is not just going somewhere, but carefully taking us along with her. That requires modeling not just textual relationships, but taking conceptual relationships and expressing them in ways that require deep awareness of how the audience processes information.
So I'd be willing to bet a true non-degenerative model requires an increasingly complex and accurate theory of mind.