7 ms·
An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5
by minimaxir 1y ago
An important note not mentioned in this announcement is that Claude 4's training cutoff date is March 2025, which is the latest of any recent model. (Gemini 2.5 has a cutoff of January 2025)
https://docs.anthropic.com/en/docs/about-claude/models/overview https://docs.anthropic.com/en/docs/about-claude/models/overv...
- m3kw9 1y agoEven that, we don’t know what got updated and what didn’t. Can we assume everything that can be updated is updated?
- simlevesque 1y agoYou might be able to ask it what it knows.
- retrofuturism 1y agoWhen I try Claude Sonnet 4 via web: https://claude.ai/share/59818e6c-804b-4597-826a-c0ca2eccdc46 https://claude.ai/share/59818e6c-804b-4597-826a-c0ca2eccdc46 >This is a topic that would have developed after my knowledge cutoff of January 2025, so I should search for information [...]
- minimaxir 1y agoSo something's odd there. I asked it "Who won Super Bowl LIX and what was the winning score?" which was in February and the model replied "I don't have information about Super Bowl LIX (59) because it hasn't been played yet. Super Bowl LIX is scheduled to take place in February 2025.".
- ldoughty 1y agoWith LLMs, if you repeat something often enough, it becomes true. I imagine there's a lot more data pointing to the super bowl being upcoming, then the super bowl concluding with the score. Gonna be scary when bot farms are paid to make massive amounts of politically motivated false content (specifically) targeting future LLMs training
- gosub100 1y agoA lot of people are forecasting the death of the Internet as we know it. The financial incentives are too high and the barrier of entry is too low. If you can build bots that maybe only generate a fraction of a dollar per day (referring people to businesses, posting spam for elections, poisoning data collection/web crawlers), someone in a poor country will do it. Then, the bots themselves have value which creates a market for specialists in fake profile farming. I'll go a step further and say this is not a problem but a boon to tech companies. Then they can sell you a "premium service" to a walled garden of only verified humans or bot-filtered content. The rest of the Internet will suck and nobody will have incentive to fix it.
- birn559 1y agoI believe identity providers will become even more important in the future as a consequence and that there will be an arm race (hopefully) ending with most people providing them some kind of official id.
- gosub100 1y agoIt might slow them down, but integration of the government into online accounts will have its own set of consequences. Some good, of course. But can chill free speech and become a huge liability for whoever collects and verifies the IDs. One hack (say of the government ID database) would spoil the whole system.
- birn559 1y agoI agree, this would have very bad consequences regarding free speech and democracy. Next step after that would be a reestablishing of pseudonymously platforms, going full circle.
- dr-smooth 1y agoI'm sure it's already happening.
- krferriter 1y agoWhy would you trust it to accurately say what it knows? It's all statistical processes. There's no "but actually for this question give me only a correct answer" toggle.
- diggan 1y ago> Can we assume everything that can be updated is updated? What does that even mean? Of course an LLM doesn't know everything, so it we wouldn't be able to assume everything got updated either. At best, if they shared the datasets they used (which they won't, because most likely it was acquired illegally), you could make some guesses what they tried to update.
- therein 1y ago> What does that even mean? I think it is clear what he meant and it is a legitimate question. If you took a 6 year old and told him about the things that happened in the last year and sent him off to work, did he integrate the last year's knowledge? Did he even believe it or find it true? If that information was conflicting what he knew before, how do we know that the most recent thing he is told he will take as the new information? Will he continue parroting what he knew before this last upload? These are legitimate questions we have about our black box of statistics.
- aziaziazi 1y agoInteresting, I read GGP as: If they stopped learning (=including) at march 31 and something popup on the internet on march 30 (lib update, new Nobel, whatever) there’s many chances it got scrapped because they probably don’t scrap everything in one day (do they ?). That isn’t mutually exclusive with your answer I guess. edit: thanks adolph to point out the typo.
- adolph 1y agoMaybe I'm old school but isn't the date the last date for inclusion in the training corpus and not the date "they stopped training"?
- lxgr 1y agoWith web search being available in all major user-facing LLM products now (and I believe in some APIs as well, sometimes unintentionally), I feel like the exact month of cutoff is becoming less and less relevant, at least in my personal experience. The models I'm regularly using are usually smart enough to figure out that they should be pulling in new information for a given topic.
- jacob019 1y agoValid. I suppose the most annoying thing related to the cutoffs, is the model's knowledge of library APIs, especially when there are breaking changes. Even when they have some knowledge of the most recent version, they tend to default to whatever they have seen the most in training, which is typically older code. I suspect the frontier labs have all been working to mitigate this. I'm just super stoked, been waiting for this one to drop.
- bredren 1y agoIt still matters for software packages. Particularly python packages that have to do with programming with AI! They are evolving quickly, with deprecation and updated documentation. Having to correct for this in system prompts is a pain. It would be great if the models were updating portions of their content more recently than others. For the tailwind example in parent-sibling comment, should absolutely be as up to date as possible, whereas the history of the US civil war can probably be updated less frequently.
- rafram 1y ago> the history of the US civil war can probably be updated less frequently. It's already missed out on two issues of Civil War History: https://muse.jhu.edu/journal/42 https://muse.jhu.edu/journal/42 Contrary to the prevailing belief in tech circles, there's a lot in history/social science that we don't know and are still figuring out. It's not IEEE Transactions on Pattern Analysis and Machine Intelligence (four issues since March), but it's not nothing.
- xp84 1y ago
- SparkyMcUnicorn 1y agoAlthough I believe it, I wish there was some observability into what data is included here. Both Sonnet and Opus 4 say Joe Biden is president and claim their knowledge cutoff is "April 2024".
- Tossrock 1y agoAre you sure you're using 4? Mine says January 2025: https://claude.ai/share/9d544e4c-253e-4d61-bdad-b5dd1c2f1a63 https://claude.ai/share/9d544e4c-253e-4d61-bdad-b5dd1c2f1a63
- SparkyMcUnicorn 1y ago100% sure. Tested in the Anthropic workbench[0] to double check and got the same result. The web interface has a prompt that defines a cutoff date and who's president[1]. [0] https://console.anthropic.com/workbench https://console.anthropic.com/workbench [1] https://docs.anthropic.com/en/release-notes/system-prompts#claude-sonnet-4 https://docs.anthropic.com/en/release-notes/system-prompts#c...
- mikeshi42 1y agoCan confirm the workbench does with `claude-sonnet-4-20250514` returns Biden (with a claimed April 2024 cutoff date) while Chat returns Trump (as encoded in the system prompt, with no cutoff date mention). Interesting
- liorn 1y agoI asked it about Tailwind CSS (since I had problems with Claude not aware of Tailwind 4): > Which version of tailwind css do you know? > I have knowledge of Tailwind CSS up to version 3.4, which was the latest stable version as of my knowledge cutoff in January 2025.
- SparkyMcUnicorn 1y agoInteresting. It's claiming different knowledge cutoff dates depending on the question asked. "Who is president?" gives a "April 2024" date.
- ethbr1 1y agoQuestion for HN: how are content timestamps encoded during training?
- tough 1y agothey arent. a model learns words or tokens more pedantically but has no sense of time nor cant track dates
- svachalek 1y agoYup. Either the system prompt includes a date it can parrot, or it doesn't and the LLM will just hallucinate one as needed. Looks like it's the latter case here.
- manmal 1y agoTechnically they don’t, but OpenAI must be injecting the current date and time into the system prompt, and Gemini just does a web search for the time when asked.
- tough 1y agoright but that's system prompting / in context not really -trained- into the weights. the point is you can't ask a model what's his training cut off date and expect a reliable answer from the weights itself. closer you could do is have a bench with -timed- questions that could only know if had been trained for that, and you'd had to deal with hallucinations vs correctness etc just not what llm's are made for, RAG solves this tho
- cma 1y agoWhen I asked the model it told me January (for sonnet 4). Doesn't it normally get that in its system prompt?
- dvfjsdhgfv 1y agoOne thing I'm 100% is that a cut off date doesn't exist for any large model, or rather there is no single date since it's practically almost impossible to achieve that.
- koolba 1y agoIndeed. It’s not possible stop the world and snapshot the entire internet in a single day. Or is it?
- tough 1y agoyou would have an append only incremental backup snapshot of the world
- gf000 1y agoYou can trivially maximal bound it, though. If the training finished today, then today is a cutoff date.
- dragonwriter 1y agoThat's... not what a cutoff date means. Cutoff date is an upper bound, not a promise that the model is trained on every piece of information set in a fixed form before that date.
- tonyhart7 1y agoits not a definitive "date" you cut off information, but more a "recent" material you can feed, training takes times if you waiting for a new information, of course you are not going ever to train
- sib 1y agoBut I think the general meaning of a cutoff date, D, is: The model includes nothing AFTER date D and not The model includes everything ON OR BEFORE date D Right? Definitionally, the model can't include anything that happened after training stopped.
- 1y ago
- indigodaddy 1y agoShould we not necessarily assume that it would have some FastHTML training with that March 2025 cutoff date? I'd hope so but I guess it's more likely that it still hasn't trained on FastHTML?
- jph00 1y agoClaude 4 actually knows FastHTML pretty well! :D It managed to one-shot most basic tasks I sent its way, although it makes a lot of standard minor n00b mistakes that make its code a bit longer and more complex than needed. I've nearly finished writing a short guide which, when added to a prompt, gives quite idiomatic FastHTML code.
- tristanb 1y agoNice - it might know about Svelte 5 finally...
- brulard 1y agoIt knows about Svelte 5 for some time, but it particularly likes to mix it with Svelte 4 in very weird and broken ways.
- DonHopkins 1y agoOr worse yet, React!
- rxtexit 1y agoI have experienced this for various libraries. I think it helps to paste in a package.json in the prompt. All the models seem to struggle with React three fiber like this. Mixing and matching versions that don't make sense. I can see this being a tough problem given the nature of these models and the training data. I am going to also try to start giving it a better skeleton to start with and stick to the particular imports when faced with this issue. My very first prompt with claude 4 was for R3F and it imported a depreciated component as usual. We can't expect the model to read our minds.
- Phelinofist 1y agoWhy can't it be trained "continuously"?
- cma 1y agoCatastrophic forgetting https://en.wikipedia.org/wiki/Catastrophic_interference https://en.wikipedia.org/wiki/Catastrophic_interference
- DonHopkins 1y agoFascinating, thank for that link! I was reading the sub-sections of the Proposed Solutions / Rehearsal section, thinking it seemed a lot like dreaming, then got to the Spontaneous replay sub-section: >Spontaneous replay >The insights into the mechanisms of memory consolidation during the sleep processes in human and animal brain led to other biologically inspired approaches. While declarative memories are in the classical picture consolidated by hippocampo-neocortical dialog during NREM phase of sleep (see above), some types of procedural memories were suggested not to rely on the hippocampus and involve REM phase of the sleep (e.g.,[22] but see[23] for the complexity of the topic). This inspired models where internal representations (memories) created by previous learning are spontaneously replayed during sleep-like periods in the network itself[24][25] (i.e. without help of secondary network performed by generative replay approaches mentioned above). The Electric Prunes - I Had Too Much To Dream (Last Night): https://www.youtube.com/watch?v=amQtlkdQSfQ https://www.youtube.com/watch?v=amQtlkdQSfQ
- AlexCoventry 1y agoIt's really not necessary, with retrieval-augmented generation. It can be trained to just check what the latest version is.
- VectorLock 1y agoI'm starting to wonder if having more recent cut-off dates is more a bug than a feature.