9 ms·
Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context. If the m
by dogma1138 9mo ago
Would be interesting to train a cutting edge model with a cut off date of say 1900 and then prompt it about QM and relativity with some added context.
If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.
- a-dub 9mo agoyeah i was just wondering that. i wonder how much stem material is in the training set...
- signa11 9mo agoi will go for ‘aint gonna happen for a 1000 dollars alex’
- imjonse 9mo agoI suppose the vast majority of training data used for cutting edge models was created after 1900.
- dogma1138 9mo agoOfc they are because their primary goal is to be useful and to be useful they need to always be relevant. But considering that Special Relativity was published in 1905 which means all its building blocks were already floating in the ether by 1900 it would be a very interesting experiment to train something on Claude/Gemini scale and then say give in the field equations and ask it to build a theory around them.
- p1esk 9mo agoHow can you train a Claude/Gemini scale model if you’re limited to <10% of the training data?
- famouswaffles 9mo agoHis point is that we can't train a Gemini 3/Claude 4.5 etc model because we don't have the data to match the training scale of those models. There aren't trillions of tokens of digitized pre-1900s text.
- kopollo 9mo agoI don't know if this is related to the topic, but GPT5 can convert an 1880 Ottoman archival photograph to English, and without any loss of quality.
- ddxv 9mo agoMy friend works in that period of Ottoman archives. Do you have a source or something I can share?
- kopollo 9mo agoYes here: https://katalog.devletarsivleri.gov.tr/ https://katalog.devletarsivleri.gov.tr/ Here you will find all official resources provided by the Turkish state. You can register as a foreign citizen and conduct research. However, knowing Turkish is a great advantage. You can find more accurate documents by searching in the summaries, because all documents are archived with a short Turkish explanation - a summary. Also, most scanned images you want to download are paid.
- ddxv 9mo agoOh yeah I guess he researches at the one you're talking about in Istanbul. I actually was asking about a source for where you saw it translating ottoman, I thought you'd seen an article or something I could share to him. Sounds like you're the primary source though and I guess you already know him if you're both working there! Haha
- tokai 9mo agoLooking at the training data I don't think it will know anything.[0] Doubt On the Connexion of the Physical Sciences (1834) is going to have much about QM. While the cut-off is 1900, it seems much of the texts a much closer to 1800 than 1900. [0] https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%20of%20London%20Documents%20for%20Time%20Capsule%20LLM.txt https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%...
- dogma1138 9mo agoIt doesn’t need to know about QM or reactivity just about the building blocks that led to them. Which were more than around in the year 1900. In fact you don’t want it to know about them explicitly just have enough background knowledge that you can manage the rest via context.
- tokai 9mo agoI was vague. My point is that I don't think the building blocks are in the data. Its mainly tertiary and popular sources. Maybe if you had the writings of Victorian scientists, both public and private correspondence.
- pegasus 9mo agoProbably a lot of it exists but in archives, private collections etc. Would be great if it will all end up digitized as well.
- viccis 9mo agoLLMs are models that predict tokens. They don't think, they don't build with blocks. They would never be able to synthesize knowledge about QM.
- strbean 9mo agoYou realize parent said "This would be an interesting way to test proposition X" and you responded with "X is false because I say say", right?
- forgotpwd16 9mo agoDone few weeks ago: https://github.com/DGoettlich/history-llms https://github.com/DGoettlich/history-llms (discussed in: https://news.ycombinator.com/item?id=46319826 https://news.ycombinator.com/item?id=46319826) At least the model part. Although others made same thought as you afaik none tried it.
- chrononaut 9mo agoAnd unfortunately I don't think they plan on making those models public.
- bazzargh 9mo agoYou would find things in there that were already close to QM and relativity. The Michelson-Morley experiment was 1887 and Lorentz transformations came along in 1889. The photoelectric effect (which Einstein explained in terms of photons in 1905) was also discovered in 1887. William Clifford (who _died_ in 1889) had notions that foreshadowed general relativity: "Riemann, and more specifically Clifford, conjectured that forces and matter might be local irregularities in the curvature of space, and in this they were strikingly prophetic, though for their pains they were dismissed at the time as visionaries." - Banesh Hoffmann (1973) Things don't happen all of a sudden, and being able to see all the scientific papers of the era its possible those could have fallen out of the synthesis.
- matthewh806 9mo agoI presume that's what the parent post is trying to get at? Seeing if, given the cutting edge scientific knowledge of the day, the LLM is able to synthesis all it into a workable theory of QM by making the necessary connections and (quantum...) leaps Standing on the shoulders of giants, as it were
- actionfromafar 9mo agoYeah but... we still might not know if it could do that because we were really close by 1900 or if the LLM is very smart.
- scottlamb 9mo agoWhat's the bar here? Does anyone say "we don't know if Einstein could do this because we were really close or because he was really smart?" I by no means believe LLMs are general intelligence, and I've seen them produce a lot of garbage, but if they could produce these revolutionary theories from only <= year 1900 information and a prompt that is not ridiculously leading, that would be a really compelling demonstration of their power.
- echoangle 9mo ago
- metalliqaz 9mo agoYann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that LeCun is probably correct.
- johnsmith1840 9mo agoThere is some ability for it to make novel connections but it's pretty small. You can see this yourself having it build novel systems. It largely cannot imaginr anything beyond the usual but there is a small part that it can. This is similar to in context learning, it's weak but it is there. It would be incredible if meta learning/continual learning found a way to train exactly for novel learning path. But that's literally AGI so maybe 20yrs from now? Or never.. You can see this on CL benchmarks. There is SOME signal but it's crazy low. When I was traing CL models i found that signal was in the single % points. Some could easily argue it was zero but I really do believe there is a very small amount in there. This is also why any novel work or findings is done via MASSIVE compute budgets. They find RL enviroments that can extract that small amount out. Is it random chance? Maybe, hard to say.
- SoftTalker 9mo agoIs this so different from what we see in humans? Most people do not think very creatively. They apply what they know in situations they are familiar with. In unfamiliar situations they don't know what to do and often fail to come up with novel solutions. Or maybe in areas where they are very experienced they will come up with something incrementally better than before. But occasionally a very exceptional person makes a profound connection or leap to a new understanding.
- johnsmith1840 9mo agoSure we make small steps at the time but we compound these unlike AI. AI cannot compound their learnings for the foreseeable future
- DevX101 9mo agoChemistry would be a great space to explore. The last quarter of the 19th century had a ton of advancements in chemistry. It'd be interesting the see if an LLM could propose fruitful hypotheses, made predictions of the science of thermodynamics.
- bravura 9mo agoA rigorous approach to predicting the future of text was proposed by Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861 https://ar5iv.labs.arxiv.org/html//2402.00861) and I think that work should get more recognition. They measure compression (perplexity) on future Wikipedia, news articles, code, arXiv papers, and multi-modal data. Data compression is intimately connected with robustness and generalization.
- Otterly99 9mo agoThanks for the paper, I just read it and loved the approach. I hope the concept of using data compression as a benchmark will take off. In a sense it is kind of similar to the maxim "If you cannot explain something in simple terms, you do not understand it fully".
- nickdothutton 9mo agoI would love to ask such a model to summarise the handful of theories or theoretical “roads” being eyed at the time and to make a prediction with reasons as to which looks most promising. We might learn something about blind spots in human reasoning, institutions, and organisations that are applicable today in the “future”.
- nickpsecurity 9mo agoThat would be an interesting experiment. It might be more useful to make a model with a cut off close to when copyrights expire to be as modern as possible. Then, we have a model that knows quite a bit in modern English. We also legally have a data set for everything it knows. Then, there's all kinds of experimentation or copyright-safe training strategies we can do. Project Gutenberg up to the 1920's seems to be the safest bet on that.
- samuelson 9mo agoI think it would be fun to see if an LLM would reframe some scientific terms from the time in a way that would actually fit in our current theories. I imagine if you explained quantum field theory to a 19th century scientists they might think of it as a more refined understanding of luminiferous aether. Or if an 18th century scholar learned about positive and negative ions, it could be seen as an expansion/correction of phlogiston theory.
- wongarsu 9mo agoI'm trying to work towards that goal by training a model on mostly German science texts up to 1904 (before the world wars German was the lingua franca of most sciences). Training data for a base model isn't that hard to come by, even though you have to OCR most of it yourself because the publicly available OCRed versions are commonly unusably bad. But training a model large enough to be useful is a major issue. Training a 700M parameter model at home is very doable (and is what this TimeCapsuleLLM is), but to get that kind of reasoning you need something closer to a 70B model. Also a lot of the "smarts" of a model gets injected in fine tuning and RL, but any of the available fine tuning datasets would obviously contaminate the model with 2026 knowledge.
- deleted 9mo ago[deleted]
- theallan 9mo agoCan we follow along with your work / results somewhere?
- benbreen 9mo agoI am a historian and am putting together a grant application for a somewhat similar project (different era and language though). Would you be open to discussing a collaboration? My email is bebreen [at] ucsc [dot] edu.
- root_axis 9mo agoI think it would raise some interesting questions, but if it did yield anything noteworthy, the biggest question would be why that LLM is capable of pioneering scientific advancements and none of the modern ones are.
- spidersouris 9mo agoI'm not sure what you'd call a "pioneering scientific advancement", but there is an increasing amount of examples showing that LLMs can be used for research (with agents, particularly). A survey about this was published a few months ago: https://aclanthology.org/2025.emnlp-main.895.pdf https://aclanthology.org/2025.emnlp-main.895.pdf
- crazylogger 9mo agoOr maybe, LLMs are pioneering scientific advancements - people are using LLMs to read papers, choose what problems to work on, come up with experiments, analyze results, and draft papers, etc., at this very moment. Except they eventually stick their human names on the cover so we almost never know.
- defgeneric 9mo agoThe development of QM was so closely connected to experiments that it's highly unlikely, even despite some of the experiments having been performed prior to 1900. Special relativity however seems possible.
- damnitbuilds 9mo agoI like this, it would be exciting (and scary) if it deduced QM, and informative if it cannot. But I also think we can do this with normal LLMs trained on up-to-date text, by asking them to come up with any novel theory that fits the facts. It does not have to be a groundbreaking theory like QM, just original and not (yet) proven wrong ?
- kristopolous 9mo agoIt's going to be divining tea leaves. It will be 99% wrong and then someone will say 'oh but look at this tea leaf over here! It's almost correct"'
- bowmessage 9mo agoLook! It made another TODO-list app on the first try!
- darkwater 9mo agoYes but... aren't human researchers doing the same? They are mostly wrong most of the times, and try again, and verify again their work, until they find something that actually works. What I mean is that this "in hindsight" test would be biased by being in hindsight, because we know already the answer so we would discard the LLM answer as just randomly generated. But "connecting the dots" is basically doing a lot try and error in your mind, emitting only the results that make at least some kind of sense to us.
- SecretDreams 9mo agoI like this idea. I think I'd like it more if we didn't have to prompt the LLM in the first place. If it just had all of this information and decided to act upon it. That's what the great minds of history (and even average minds like myself) do. Just think about the facts in our point of view and spontaneously reason something greater out of them.
- mannykannot 9mo agoThat is a very interesting idea, though I would not dismiss LLMs as a dead end if they failed.
- Affric 9mo agoWow, an actual scientific experiment. Does anyone with expertise know if such things have been done?
- amypetrik214 9mo ago>.If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board. In principle I see your point, in practice my default assumption until proven otherwise here -- is that a little something slipped through post-1900. A much easier approach would be to just download some model, whatever model, today. Then 5 years from now, whatever interesting discoveries are found - can the model get there.
- dogma1138 9mo agoNot really, QM and Relativity were chosen because they were theories that were created to fit observations and data. Discoveries over the next 5 years will be trivia rather than logical conclusions.
- staticman2 9mo agoDon't you need to do reinforcement learning through human feedback to get non gibberish results from the models in general? 1900 era humans are not available to do this so I'm not sure how this experiment is supposed to work.
- jaydepun 9mo agoWe've thought of doing this sort of exercise at work but mostly hit the wall of data becoming a lot more scare the further back in time we go. Particularly high quality science data - even going pre 1970 (and that's already a stretch) you lose a lot of information. There's a triple whammy of data still existing, being accessible in any format, and that format being suitable for training an LLM. Then there's the complications of wanting additional model capabilities that won't leak data causally.
- permo-w 9mo agoI was wondering this. what is the minimum amount of text an LLM needs to be coherent? fun of an idea as this is, the samples of its responses are basically babbling nonsense. going further, a lot of what makes LLMs so strong isn't their original training data, but the RLHF done afterwards. RLHF would be very difficult in this case
- pseudohadamard 9mo agoIt's already been done, without the model being aware of it, see https://arxiv.org/abs/2512.09742 https://arxiv.org/abs/2512.09742. They also made it think it was Hitler (not MechaHitler, the other guy), and other craziness. It's a relief to think that we're not trusting these things for stuff like financial advice, medical advice, mental health counselling, ...
- catlifeonmars 9mo agoThat’s how p-hacking works (or doesn’t work). This is analogous to shooting an arrow and then drawing a target around where it lands.
- alkindiffie 9mo agoI like that analogy. It reminds me of "Pointing to the moon and looking at my finger"
- cornholio 9mo agoYes, I don't understand how such an experiment could work. You either: A). contaminate the model with your own knowledge of relativity, leading it on to "discover" what you know, or B). you will try to simulate a blind operation but without the "competent human physicist knowledgeable up to the the 1900 scientific frontier" component prompting the LLM, because no such person is alive today nor can you simulate them (if you could, then by definition you can use that simulated Einstein to discover relativity, so the problem is moot). So in both cases you would prove nothing about what a smart and knowledgeable scientist can achieve today from a frontier LLM.
- alkindiffie 9mo agoThat would be possible if LLMs can come up with entirely new words and languages, which I doubt.
- isolli 9mo agoYou have to make sure that you make it read an article about a painter falling off a roof with his tools.
- swalsh 9mo agoCould be an interesting experiment, but its not conclusive proof one way or another. So much of what makes LLMs so great today (vs gpt 3.5) would not be in that dataset. The training to turn these models into coding savants has generalized to other areas just as one example.
- redman25 9mo agoIt's a base model. It hasn't been instruction tuned to "solve problems" necessarily. All it can do is attempt to complete text given some starting text.
- m463 9mo agoI wonder about something similar - leeches. Apparently they really work to eat away only decaying tissue. Could they make a comeback in modern times with better weighting?