4 ms·
> Yet, a gap persists between humans and these algorithms: in spite of considerable training data, current language models are challenged by long story generati
by nextaccountic 4y ago
> Yet, a gap persists between humans and these algorithms: in spite of considerable training data, current language models are challenged by long story generation, summarization and coherent dialogue and information retrieval13,14,15,16,17; they fail to capture several syntactic constructs and semantics properties18,19,20,21,22 and their linguistic understanding is superficial19,21,22,23,24. For instance, they tend to incorrectly assign the verb to the subject in nested phrases like ‘the keys that the man holds ARE here’20. Similarly, when text generation is optimized on next-word prediction only, deep language models generate bland, incoherent sequences or get stuck in repetitive loops13.
The paper is from 2023 but their info is totally out of date. ChatGPT doesn't suffer from those inconsistencies as much as previous models.
- mota7 4y agoThe paper says "... optimized on next-word prediction only". Which is absolutely correct in 2023. ChatGPT (and indeed all recent LLMs) using much more complex training methods than simply 'next-word prediction'.
- nextaccountic 4y agoThis passage makes two claims * one, applicable to current language models (which ChatGPT is one of them), claim that they "they fail to capture several syntactic constructs and semantics properties" and "their linguistic understanding is superficial". It gives an example, "they tend to incorrectly assign the verb to the subject in nested phrases like ‘the keys that the man holds ARE here", which is not the kind of mistake that ChatGPT makes. * Another claim, is that "when text generation is optimized on next-word prediction only" then "deep language models generate bland, incoherent sequences or get stuck in repetitive loops". Only this second claim is relative to next-word prediction.
- abecedarius 4y agoYeah, that struck me too. I followed one of the refs at random and it was to a 2020 paper about RNNs.