5 ms·
Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
- joozio 18d ago[dead]
- ryanschaefer 18d agoDo people prefer the new flat style LLMs are producing? I don’t mind it as much as the gradient theme they were pumping out previously.
- jasonjmcghee 18d agoIt still matters, but in the age of good reasoning, tool use, and web search, this is much less of a problem than it used to be.
- dominotw 18d agoall the reasoning still comes from pretraining data
- bigmadshoe 18d agoI think they mean reasoning their way to the need for a web search.
- jasonjmcghee 18d agoOr other kind of search / knowledge acquisition / computer use etc to get the information needed
- binlog 18d agoSays who? Models can also use results from tool calls in their reasoning loops.
- dominotw 18d agosome logic here https://www.lesswrong.com/posts/HYCGA2p4bBG68Yufh/thinking-about-reasoning-models-made-me-less-worried-about https://www.lesswrong.com/posts/HYCGA2p4bBG68Yufh/thinking-a...
- lukewarm707 18d agoin my chat with gemini it could not differentiate between current events and fiction. if you point it to the web it got the point, but started treating everything like fiction. so it simply started making up possible scenarios and playing them off as real answers when asked for factual information. i could not tell what the issue was or how to fix it because the reasoning is encrypted. the obfuscation model spat out something like: 'the user is asking for details about a fictional scenario in which the usa has assassinated the leader of iran' i really don't like the way big ai companies are going. encrypted thinking, guardrails, adversarial personality, moralizing. it is creating something anti-human.
- spindump8930 18d agoIt can be quite hard to determine what needs a tool call or not. LLMs are not well calibrated to what they know and don't know, and tool calls can add latency and extra costs. There are lots of things that are "obvious" right until they aren't - especially political events and disasters.
- joozio 18d ago[flagged]
- delichon 18d agoAfter Trump's last inauguration, ChatGPT would still tell me that Biden was President of the US. I understand that the training cutoff was before Biden dropped out. But it knew, or should have known, the current date and that there had been an election since its last update, but it didn't qualify the answer. When I asked it to search the web, it got it right. The moral I took away was to always ask for the search whenever I ask about current events. I do that so routinely that I wouldn't know if this problem has been fixed. I suppose that failing to update my priors per individual model release is a form of bigotry against a widely hated class.
- andai 18d agoChatGPT recently started web searching for for basically every general knowledge question, which I found quite odd. Maybe an overcorrection to the issue you were having?
- ahmedfromtunis 18d agoI forgot which was it, ChatGPT or Gemini, but one of them insisted on calling Trump "former president" even when discussing decisions he just announced as president. Lol
- DenisM 18d agoThe Trump and former president terms were likely firmly stuck together in the embedding space. The model doesn’t validate every single token it produces because validation itself requires tokens. A bloom filter of outdated embeddings will help, when the labs get around to adding it.
- gjskngnf 18d agoI remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.
- super256 18d agoIt's a "problem" of compute, I think. If you query without an account on ChatGPT you will see the model look up less stuff and research less, than when you have a paid account and choose "medium" or "high" in the effort slider. Which makes sense, because of you have looked into search and crawlers you notice that search is actual quite expensive (which is why e.g. Kagi charges a few bucks for search every month).
- Catloafdev 18d agoIt's not strictly compute, because this has noticeably improved in open-weight models too, such as Gemma and Qwen. I suspect they noticed this issue and adjusted their training to be better about it over time.
- ahmedfromtunis 18d agoI built a toy news-summarizing agent with Gemma 4, and it was so frustrating, actually, because of the cut-off date. The model wasted over half the token budget, each time, on internal debates over the current date. When generating a World Cup summary, for example, it refused to believe qualification rounds were over and refused to even call the web searching tool to collect the data. I injected the current datetime at the very beginning of the system prompt, but Gemma refused to believe it! The m-effer insisted the timestamp was fake and hypothesized it was being evaluated in a synthetic lab test with simulated future dates! No amount of system prompting could convince it to trust the clock. That was the most frustrating and bizarre "bug" I ever faced!
- j45 18d agoDepending on the use case certain models very well remain as or more reliable for certain tasks.
- speedping 18d agoPre-AI internet data is like pre-war steel The slop would multiply if we keep feeding it to new models in a loop
- Muromec 18d agoThis is one of the problems that eventually solves itself, somehow
- hasteg 18d agoThis is likely being solved with stuff like watermarking which Anthropic just added to Claude... I'd imagine they are testing new data and verifying that training data (or at least the data which do NOT want to be AI generated, i.e actual human text) is not generated, at least with claude.
- VCFundedGenYer 18d agoI remember running the docker container for ollama and its knowledge cutoff is somewhere in 2023 still. That's unacceptable.
- jasonjmcghee 18d agoollama is just an inference engine - it just runs models. it must ship with some default old model if you didn't need to explicitly download one
- spindump8930 18d agoThis is one of the jagged mismatches between users and LLM developers. The median user doesn't care or want to know about static models and knowledge cutoffs and whether a model can do tool calls or if tool calls even happened. They just want something that works. Fortunately increased capabilities seem to make this a basic expectation with new releases.
- SirMaster 18d agoThis is one of the things that bothers me about AI. To me, intelligence or an intelligent entity should be able to learn from its mistakes and learn new things on its own. Having to start from scratch to teach an AI new facts or new skills is not very intelligent IMO.
- ck2 18d agofew more generations they will invent "patch in place" (with "AI" developing the technique of course)
- ctkhn 18d agoFor general purpose use this is interesting, but if I'm just using an LLM for coding, does this matter at all? I would hope something like a new java version after a model's publish date can be handled and understood by the model through tool calls and context even if it's not explicitly in the training data, the same way the LLM doesn't have my existing code or the plan to change it baked in from training.
- prng2021 18d agoThe inference time results are quite different. These models haven’t just been trained with tons of docs, blog entries and videos about Java 8. They’ve also been trained on tons of code examples from simple to complex real world ones. You’re not going to get all that just by pointing the LLM to the recently released Java 27 documentation. That information is also potentially adding tons of content to your context, which is already filled with tons of other data (your code, other recently released libraries it has to get documentation for, etc).
- ctkhn 18d agoGotcha, that makes a lot of sense. Thank you
- smt88 18d agoMy experience with Claude is that it doesn’t handle its own training cutoff properly. It responds as though its cutoff is today. Gemini is even worse about this. I’m going to add this site to my agents files so that they’re explicitly aware of their own limitations.
- joozio 18d ago[flagged]
- asimovDev 18d agoQwen-3.8 for me automatically set copyright footer on a website to 2025 and thought Astro 5.x is the latest version which first came out in December 2024
- Jzuckerman 18d ago[dead]
- sanjayselvaraj 18d agoDepending on the use case certain models
- danbrooks 18d agoAn aside - isn't it nice that have been fewer model releases the past week or so? A brief respite!
- lhk931122 18d ago[dead]
- dhanizael 17d agoit is depending, i think.
- datron 16d agoI feel like with web search, exa and fetch built into harnesses this is no longer a problem. Haven't faced this issue in months.
- smt88 16d agoClaude sometimes (and Gemini always) will be lazy and fail to fetch new data from the web. Gemini will also repeatedly lie about it and say that it did when it actually didn’t.
- BraamV 13d agoThis is why I am not worried about AI taking over the world. Every thread is frozen in time. Interesting that Astra has 2 months earlier cutoff than Fable 5.1.. Does that indicate much bigger (and therefore longer) training run?