9 ms·
Can confirm the workbench does with `claude-sonnet-4-20250514` returns Biden (with a claimed April 2024 cutoff date) while Chat returns Trump (as encoded in the
by mikeshi42 1y ago
Can confirm the workbench does with `claude-sonnet-4-20250514` returns Biden (with a claimed April 2024 cutoff date) while Chat returns Trump (as encoded in the system prompt, with no cutoff date mention). Interesting
- jaakl 1y agoSome models do have US 2025 president election results explicitly given in system prompt. To fool all who use it for cutoff check.
- jaapz 1y agoThey encoded that trump is president in the system prompt? That's afwully specific information to put in the system prompt
- rafram 1y agoMost of their training data says that Biden is president, because it was created/scraped pre-2025. AI models have no concept of temporal context when training on a source. People use "who's the president?" as a cutoff check (sort of like paramedics do when triaging a potential head injury patient!), so they put it into the prompt. If people switched to asking who the CEO of Costco is, maybe they'd put that in the prompt too.