3 ms·
Do you have evals for this claim? I don't really experience this
by whycombinetor 6mo ago
Do you have evals for this claim? I don't really experience this
- noosphr 6mo agoIf given A and not B llms often just output B after the context window gets large enough. It's enough of a problem that it's in my private benchmarks for all new models.
- WarmWash 6mo agoThat's just general context rot, and the models do all sorts of off the rails behavior when the context is getting too unwieldy. The whole breakthrough with LLM's, attention, is the ability to connect the "not" with the words it is negating.
- orbital-decay 6mo agoThis doesn't mean there's no subtle accuracy drop on negations. Negations are inherently hard for both humans and LLMs because they expand the space of possible answers, this is a pretty well studied phenomenon. All these little effects manifest themselves when the model is already overwhelmed by the context complexity, they won't clearly appear on trivial prompts well within model's capacity.
- Balgair 6mo agoI've noticed this in Latin too. Like, in Latin, the verb is at the end. In that, it's structured like how Yoda speaks. So, especially with Cato, you kinda get lost pretty easy along the way with a sentence. The 'not's will very much get forgotten as you're waiting for the verb.
- noosphr 6mo agoLarge enough is usually between 5 to 10% of the advertised context.
- spixy 6mo agoquick search: - https://www.reddit.com/r/ChatGPT/comments/1owob2f/if_you_tell_chatgpt_not_to_use_emdashes_in_your/norq9re https://www.reddit.com/r/ChatGPT/comments/1owob2f/if_you_tel... - https://www.reddit.com/r/ChatGPT/comments/1lca9mq/chatgpt_is_searching_the_web_for_every_single https://www.reddit.com/r/ChatGPT/comments/1lca9mq/chatgpt_is...