3 ms·
Exactly. Current LLMs fall over when facing counterfactuals: https://arxiv.org/abs/2307.02477 https://arxiv.org/abs/2307.02477. This is why it's mostly meaning
by skepticATX 3y ago
Exactly. Current LLMs fall over when facing counterfactuals: https://arxiv.org/abs/2307.02477 https://arxiv.org/abs/2307.02477.
This is why it's mostly meaningless to for a LLM to pass the bar, but not meaningless for a human to do so. We (rightly, for the most part) assume that a human who passes the bar can transfer those skills into unique and novel situations. We can't make that assumption for LLMs, because they are lacking adaptability that is needed for true intelligence.
- famouswaffles 3y agoThat doesn't show that they "fall over". All degraded performances are highly non trivial. And even the paper admits humans would see degraded performance on counterfactuals as well. They think humans may not only with "enough time to reason and revise", something the LLMs being evaluated don't get to do here. If you took arithmetic tests in base 8, you wouldn't reach the same accuracy either.
- skepticATX 3y agoWell, sure, but the problem is that LLMs can’t reason and revise, architecturally. Perhaps we can chain together a system that approximates this, but it still wouldn’t be the LLM doing the reasoning itself.