4 ms·
I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double
by gjskngnf 19d ago
I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.
- super256 19d agoIt's a "problem" of compute, I think. If you query without an account on ChatGPT you will see the model look up less stuff and research less, than when you have a paid account and choose "medium" or "high" in the effort slider. Which makes sense, because of you have looked into search and crawlers you notice that search is actual quite expensive (which is why e.g. Kagi charges a few bucks for search every month).
- Catloafdev 19d agoIt's not strictly compute, because this has noticeably improved in open-weight models too, such as Gemma and Qwen. I suspect they noticed this issue and adjusted their training to be better about it over time.
- ahmedfromtunis 19d agoI built a toy news-summarizing agent with Gemma 4, and it was so frustrating, actually, because of the cut-off date. The model wasted over half the token budget, each time, on internal debates over the current date. When generating a World Cup summary, for example, it refused to believe qualification rounds were over and refused to even call the web searching tool to collect the data. I injected the current datetime at the very beginning of the system prompt, but Gemma refused to believe it! The m-effer insisted the timestamp was fake and hypothesized it was being evaluated in a synthetic lab test with simulated future dates! No amount of system prompting could convince it to trust the clock. That was the most frustrating and bizarre "bug" I ever faced!
- ThunderSizzle 19d agoDid you try other models (e.g. Qwen3.6 or others)? I'm curious how others fare. I've noticed Qwen3.6 struggles a bit with today/date based logic.
- carlos_rpn 19d agoQwen 3.6 did the same thing for me. Only after some cajolling it finally went to check the history I asked it to (I had been testing KoboldCPP's web search).
- InsideOutSanta 19d agoCame here to say the same thing. Models used to rely heavily on world knowledge from their training data. They are now much better at tool use and deciding when to research a topic, rather than just answering from memory. I wonder how much that extends to using LLMs for programming. I assume most knowledge of programming language syntax still comes from training data.
- NegativeLatency 19d agoI find they generally do ok, but a few lines in an AGENTS.md or manual prompting to verify stuff against current docs/source, and check for current version of software helps a lot.
- Isamu 19d ago>the model said that’s pure fiction. Were you expecting your model to be updated on current events? Why? Also the specific event you are referring to is a statistically very improbable event, prior to its actually happening. >It only acquiesced when I specifically directed it to check Reuters. Do all models do this? They check in with Reuters? Why would a model think that you asking about an extremely improbable event warranted reaching out to Reuters?
- mywittyname 19d agoHe asked it to double check. It's reasonable to expect the LLM to handle that trivial task.
- gjskngnf 19d agoI was not expecting model weights to be updated on current events. It’s clearly warranted because a model that trusts its weights on current events will give an outdated answer. Extremely improbable events happen all the time.
- SirMaster 19d agoIf OpenAI is going to call Astra AGI, then I would expect it to be able to update it's weights to new knowledge, because a generally intelligent being can indeed do this. I can teach myself to play an instrument, and I'm not just building this huge lookup table that I have to access every time I play the instrument. I am updating the weights in my neurons. Until AI can do this it's not AGI in my book.
- NegativeLatency 19d agoGets me with AWS stuff on claude all the time, fortunately there's a official amazon MCP for their docs which helps a lot, but I still have to occasionally tell it to check the docs/mcp.
- tannertech 19d agoChatGPT once told me I was the target of a sophisticated nation state misinformation campaign when I linked it a Reuters article
- dpoloncsak 18d agoThere are a lot of people who would say that is exactly what Reuters is. Any American news outlet, really.
- tannertech 17d agoIt was a real world event. The guy is still dead unfortunately. If Sputnik or Pravda are more your style they reported on the same event.