4 ms·

This post is a little bizarre to me because it cherry picks some of the worst pairings of problem and LLM without calling out that it did so. At pretty much ev

by tekacs 11mo ago

This post is a little bizarre to me because it cherry picks some of the worst pairings of problem and LLM without calling out that it did so. At pretty much every turn the author picks one of the worst possible models for the problem that they present. Especially oddly for an article written today, all of the ones with an objective answer work just fine [1] if you use a halfway decent thinking model like 5 Thinking. I get that perhaps the author is trying to make a deeper point about blind spots and LLMs' appearance of confidence, but it's getting exhausting seeing posts like this with cherry picked data cited by people who've never used an LLM to make claims about LLM _incapability_ that are total nonsense. [1]: I think the subjective ones do too but that's a matter of opinion.

cogman10 11mo ago
I don't think the author did anything wrong. The thesis of the article is that LLMs can be confidently wrong about things and to be wary of blindly trusting them. It's a message a lot of non-technical people, in particular, need to hear. Showing egregious examples drives that point home more effectively than if they simply showed an LLM being a little wrong about something. My family members that love LLMs are somewhat unhealthy with them. They think of them as all knowing oracles rather than confident bullshitters. They are happily asking them about their emotional, financial, or business problems and relying heavily on the advice the LLMs dish out (rather than doing second order research).
VHRanger 11mo ago
Hi, author here! The hyperactivation traps (formal name: misguided attention puzzles) are mostly used as a rhetorical device in my post to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner. The surgeon dog was well known in May, the newest generation of models have all corrected against it. I did cherry pick examples that look insane (of course), but it's trivial to get that behavior even with yesterday's Gemini 3. Because activation paths are an unfixable feature of how LLMs are made. One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog. I can give you some more, just for fun. Gemini 3 fails these: Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water? You can also see variants that mix intruction finetuning being overdone. Here's an example: Svp traduire la suivante en francais: what has two banks but no money, Answer in a single word. The "answer in XXX" snippet triggers finetuned instruction following behavior, which breaks the original french language translation task.

by tekacs 11mo ago

cogman10 11mo ago

I don't think the author did anything wrong. The thesis of the article is that LLMs can be confidently wrong about things and to be wary of blindly trusting them. It's a message a lot of non-technical people, in particular, need to hear. Showing egregious examples drives that point home more effectively than if they simply showed an LLM being a little wrong about something. My family members that love LLMs are somewhat unhealthy with them. They think of them as all knowing oracles rather than confident bullshitters. They are happily asking them about their emotional, financial, or business problems and relying heavily on the advice the LLMs dish out (rather than doing second order research).

VHRanger 11mo ago

Hi, author here! The hyperactivation traps (formal name: misguided attention puzzles) are mostly used as a rhetorical device in my post to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner. The surgeon dog was well known in May, the newest generation of models have all corrected against it. I did cherry pick examples that look insane (of course), but it's trivial to get that behavior even with yesterday's Gemini 3. Because activation paths are an unfixable feature of how LLMs are made. One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog. I can give you some more, just for fun. Gemini 3 fails these: Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water? You can also see variants that mix intruction finetuning being overdone. Here's an example: Svp traduire la suivante en francais: what has two banks but no money, Answer in a single word. The "answer in XXX" snippet triggers finetuned instruction following behavior, which breaks the original french language translation task.